Oracle Corporation et al v. SAP AG et al
Declaration of Chad Russell in Support of 859 Memorandum in Opposition, to Defendants' Motion to Partially Exclude Testimony of Kevin Mandia and Daniel Levy filed byOracle International Corporation, Oracle USA Inc., Siebel Systems, Inc.. (Attachments: # 1 Exhibit A, # 2 Exhibit B, # 3 Exhibit C, # 4 Exhibit D, # 5 Exhibit E, # 6 Exhibit F, # 7 Exhibit G, # 8 Exhibit H, # 9 Exhibit I, # 10 Exhibit J, # 11 Exhibit K, # 12 Exhibit L, # 13 Exhibit M, # 14 Exhibit N, # 15 Exhibit O, # 16 Exhibit P, # 17 Exhibit Q, # 18 Exhibit R, # 19 Exhibit S, # 20 Exhibit T, # 21 Exhibit U, # 22 Exhibit V)(Related document(s) 859 ) (Russell, Chad) (Filed on 9/9/2010)
Oracle Corporation et al v. SAP AG et al
Doc. 860 Att. 3
Appendices to Report - Review of SAP TN November 16, 2009, supplemented February 12, 2010
Proprietary and Highly Confidential
Appendices A. Methodology
In an effort to determine the extent at which SAP TN downloaded, copied, and distributed Oracle Software, Mandiant relied on commonly used forensic software and well accepted data analysis tools. Mandiant primarily performed the following tasks when identifying and aggregating the data from SAP TN electronic evidence: · · · Reviewed directory structure, file names and file paths, and file content Performed file comparisons using MD5 Hashes Performed file comparisons using the "diff" utility
In reviewing the contents of the electronic evidence, to include directory structures, file names and file paths, Mandiant primarily used Guidance Software's EnCase1 application. EnCase was used by Mandiant to review, identify, search, and record full paths of relevant filenames. A file path is the exact location of a file within a computer's file system. In modern computer systems, two separate files cannot have the exact same file path. This infers that the number of files on a computer system is directly based on the number of file paths.2 To To identify files that were exact matches, Mandiant relied on the Message Digest #5 algorithm MD5 is a common cryptographic hash algorithm with a 128-bit output or thirty-two (MD5). hexadecimal hexadecimal characters.3 In simpler terms, MD5 hashing takes an input value and through a series of In mathematical operations, produces a unique "digital fingerprint" or numerical reference for a file. g Mandiant Mandiant used "md5deep", authored by Jesse Kornblum,4 and EnCase to generate the MD5 hashes and used during analysis. Another type of file comparison Mandiant performed occurred when we compared two files that were not exact matches. In order to do this, we used a standard utility called "diff". The "diff" program program is usually used to show changes between a file and a previous version of the same file.5 The The diff diff program reports differences between two files, expressed as a minimal list of line changes to bring either either file into agreement with the other.6 The output of the "diff" comparison is also called a "diff". The In order to automate some of our processes, Mandiant created shell scripts. Shell scripts are usually written in a specific scripting language that an interpreter must process in order to actually function. For example, a single script could be run to perform an automated sequence of tasks as opposed to manually typing in each command. Script writing is an accepted practice as the scripts reduce the likelihood of human errors when processing multiple sets of data that require the exact same operations to be conducted against that data set. Mandiant also provided the shell scripts created to automate our comparison processes in the appendices referenced in the report. Mandiant also relied on using databases to allow more efficient and accurate analysis of the data. When appropriate, Mandiant provides the Data Definitions and the Queries executed when we
http://www.guidancesoftware.com/computer-forensics-ediscovery-software-digital-evidence.htm. Carrier, B. "File System Forensic Analysis," Addison-Wesley Professional, 2005. 3 Stallings, W., Cryptography and Network Security, 2nd ed., New York: Prentice-Hall, 1997. 4 http://md5deep.sourceforge.net/. 5 Horwitz, Susan, "Identifying the semantic and textual differences between two versions of a program," ACM SIGPLAN Notices 25(6) (June 1989), p. 234-245. 6 Hunt, James W. and McIlroy, M. Douglas. (June 1976) "An Algorithm for Differential File Comparison," Computing Science Technical Report, Bell Laboratories, p. 41.
performed analysis using databases. enormous log files.
We also built and used databases to assist in the review of
Data Sources reviewed by Mandiant
Mandiant reviewed data from many sources, including DCITBU01, Data Warehouse, MAIL03, WEB01, Delivered Updates and Fixes, the SAS database, Pathfinder, BakTrak, CD Client Jukebox, CD Binders, AS/400, deposition testimony, and other interrogatories produced by SAP TN. Each data source is described the section below.
DCITBU01 and Data Warehouse
From July 14, 2008, until February 2, 2009, Mandiant was granted access to 84 EnCase image files representing data from 46 SAP TN systems. During this timeframe, Mandiant was allowed remote access to five different servers maintained by Forensic Consulting Services in order to review EnCase image files representing the 46 SAP TN systems.7 SAP TN stored the vast majority of the materials that it downloaded from Customer Connection and SupportWeb on the G drive of server DCITBU01. See eAppendix "ORCLX-MAN000142." Since the EnCase image files contained only the active files8 for each system, Mandiant was unable to review unallocated space9 on each of these drives. Therefore, Mandiant was unable to search for traces of Oracle materials that had been deleted or removed. (Deposition testimony indicates that there may be significant quantities of Oracle software that existed at one time on SAP TN systems but was deleted.) eAppendix "ORCLX-MAN-000141" provides a detailed list of the systems Mandiant reviewed from the 84 EnCase image files. Mandiant refers to these 46 systems collectively as "Data Warehouse" (see ORCLX-MAN-000142). Mandiant, through Oracle's counsel, identified for production portions of the "Data Warehouse" Encase images it reviewed, and Defendants then initially produced 10,304.5 GB (over 10 TB) and approximately 10,772,535 separate files. The reviews of these systems also led Mandiant to request and obtain from Defendants file system metadata10 for another 2,627 GB (over 2.6 TB) and 2,014,170 separate separate files from these 46 systems which were subsequently produced for analysis after the initial production. production. Mandiant examined the data produced by SAP TN, including the 84 image files representing data data from 46 computer systems. Specifically, Mandiant searched these data sources for: x Oracle Oracle SSMs
7 SAP TN recollected data for many of these systems in 2009, and is continuing to make these recollected systems available as of the time of submission of this report. I reserve the right to supplement this report with respect to Data Warehouse materials either made available for review or produced too close in time to the submission of this report to allow for review and analysis. 8 An active file is a file that has not been marked for deletion and is readily available to the end user. 9 Unallocated space is the area on the hard drive that would contain files or file fragments that had been temporarily created or deleted by the user of the system. Computer forensic examiners are often able to recover deleted documents in their entirety from unallocated space. In this case Mandiant was not granted access to unallocated space. Therefore we were not able to search the 46 systems for trace evidence of Oracle software. 10 File system metadata is administrative information about a file that records a file's characteristics. It can be generally defined as "data about data." The metadata available for a file depends on the file system of the media on which the file resides and the application that was used to create the file. The file metadata obtained by Mandiant include the time a file was created, last modified, and last accessed, as well as the size of the file, its location on disk, and other items that describe the file.
x x x x
Local Local Environments Local Environment Backups Local SAP TN Delivered Updates and Fixes / Modified Oracle Code (SSM and Environment) SAP Documentation on SAP TN's support processes relating to Environments and Fixes Documentation
At the time of Mandiant's review of Data Warehouse, Mandiant's review was limited to identifying the following Oracle Enterprise Software product lines (as defined in my report, above) within the SAP TN data sources: Product Lines Mandiant's Review Identified To Date PeopleSoft, JD Edwards, and Siebel PeopleSoft, JD Edwards, Siebel and Database PeopleSoft HRMS Fixes Oracle
Oracle Materials Oracle SSMs Local Environments and Backups
SAP TN Delivered Updates and Fixes / Modified Oracle Code Table 1: Product Lines within the SAP TN Data Sources
When relevant data was found, Mandiant categorized the file one of two ways, as file production or metadata production. File Production: Mandiant categorized data for File Production by saving EnCase case files noted as "Produce". The "Produce" case files were needed by Mandiant for immediate review to determine how Oracle SSMs were obtained and used by SAP TN. SAP TN produced the files and the file metadata for items Mandiant selected for file production. Metadata Production: Mandiant categorized data for Metadata production by saving EnCase case files noted as "Record." The "Record" case files were not needed by Mandiant for immediate production because many of the files were produced in the production of DCITBU01 in a separate review process. Mandiant ultimately instructed Defendants to produce the case files noted for "Record" after Mandiant received and reviewed the images marked as "Produce." Mandiant received file metadata only for a subset of files reviewed on each SAP TN system, as this metadata was not produced for files not marked by Mandiant as relevant to the case. Therefore, Mandiant was unable to obtain the filenames, file sizes, and total number of all the files reviewed from July 14, 2008 through February 2, 2009. Our specific steps to review the 84 remote images to categorize items for file production or metadata production included the following: Step 1: Mandiant reviewed the directories on the target media looking for the following strings or case-insensitive keywords within the full path.11 x x x x x x
"PS*"12 "Peopletools" "Peoplesoft" "JD Edwards" "Blue" "Documentation"
The full path of a file is a method to refer to the file by its exact location on disk (i.e., "C:\Windows\System32\CMD.EXE" is the Full Path for a file called "CMD.EXE"). 12 The "*" character represents a wild card. Searches of this type identify any folders that begin with the letters "PS."
x x x x x x x x x x x x x x x x
"Environments" "Siebel" "Download" "Fix" "TN" "SAP" "Updates" "Titan" "Information Station" "Informix" "Backup" "Log" "Restore" "Oracle" "IU" "Sales"
Mandiant also reviewed for any directory that contained a SAP TN customer name such as Praxair, Robert Half, Harley, etc.). This search was performed via manual review. When Mandiant forensic examiners saw a directory name that contained a SAP TN customer name or customer prefix represented as a three letter code, Mandiant performed a manual review of that directory. Step 2: Mandiant identified and reviewed all files with the following file extensions (not casesensitive): x x x x x x ".cbl" ".sqr" ".sqc" ".par" ".c" ".h"
When Mandiant identified files with any of these extensions related to Oracle materials, Mandiant marked the entire directory the file was located in for either file production or for metadata production. Step 3: Mandiant also reviewed files with ".DOC", ".XLS", ".PDF", ".EXE", ".ZIP", ".HTM", and ".HTML" extensions for relevant documents. Files were manually reviewed, depending on the working directory the file was located in. For example, when a Mandiant examiner found one of these file extensions in a directory related to the use of Oracle materials, the file was usually manually reviewed for relevance. Step 4: Mandiant used EnCase software to review and "tag" each relevant file or directory.13 Mandiant saved their file selections (or "tags") into two separate case files.14 One of the case files contains all the files Mandiant selected for file production. The other case file contains all the files selected for metadata production. Step 5: For both file production as well as metadata production, Mandiant performed an export function to obtain the file metadata for all selected items. Mandiant recorded the following metadata for each selected file:
EnCase allows a computer forensic examiner to "bookmark" or "select" specific files. An EnCase case file allows the forensic examiner to store information about the case and record operations performed on a forensic image such as bookmarks, keyword searches, and MD5 hash values.
Table 3: Sample Fields from Pathfinder The following table contains a sample of actual data contained in Pathfinder.
Environment Product ID Client Name Name Version 86 Development HR810DEV 8 SP1 88 Development 166 Mutual of Omaha 202 Praxair HR831DMO H881MOHO H801PRXO 8.3 SP1 8.8 SP1 8 SP1 8 SP1 Application Server Machine HOMER HOMER PSDEV01 PSDEV01 PSDEV01 Database Server Machine HOMER HOMER PSDEV01 PSDEV01 PSDEV01 Database Server Plat form SQL Server SQL Server Oracle Oracle Oracle Database Server Release 7 7 9.2.0 8.1.7 8.1.7 Tools Release 8.2 8.2 8.46 8.2 8.19 Tools Patch 0.06 0.06 0.15 0.13 0.13 From CD From CD From CD Build Source
218 Ross Dress for H801ROSO Less, Inc
NT PS Home \\homer\homerrw\hr810dev \\homer\homerrw\hr831dmo \\dcpstemp02\p soft\h881moho \\dcpstemp01\p soft\h801prxo \\dcpstemp01\p soft\h801roso
Table 4: A sample excerpt from PathFinder environment info.xls referenced in full in eAppendix "ORCLX-MAN-000200" Mandiant used the information in Pathfinder to help determine the number of possible locations of Environments and installs of Oracle Database on SAP TN's infrastructure. Additionally, Mandiant used Pathfinder to analyze the releases and versions of PeopleSoft software used by SAP TN to create Local Environments. eAppendix - "ORCLX-MAN-000200"17 refers to the full set of data relied upon.
Defendants produced information from an application known as BakTrak in both native and Excel spreadsheet form.18 Mandiant analyzed the spreadsheets, which Mandiant understands are direct exports of the underlying data in the native application. Similar to SAS, Defendants produced multiple iterations and Mandiant analyzed the most current version. See eAppendix - "ORCLX-MAN-000133" and eAppendix "ORCLX-MAN-000132." BakTrak was used to track various activities related to Local Environments. BakTrak included a function tracking "check-outs" and "check-ins" when SAP TN employees reserved Environments to exclude others from working in them. BakTrak also tracked the creation of Environments, and "backups" and "restores" of Environments. A backup is a copy of all or some portion of an Environment in a compressed format.19 A restore decompresses the backed-up data and copies it to a
Mandiant uses the term "eAppendix" to refer to appendices that are too large or complex to include in a document. These appendices are provided in electronic format for your review. They are usually Excel spreadsheets. 18 See TN Disc 56, TN Disc 79, TN Disc 202. 19 See, e.g., Deposition of George Lester, April 23, 2009 at 43:11-48:20; Deposition of John Baugh, February 6, 2008 at 142:4-145:16; Deposition of John Baugh, February 7, 2008 at 290:24-297:9; Defendants Responses to Plaintiffs' Second Set of Requests for Admission, Nos. 220-222 ("Defendant SAP TN ADMITS that often in the `Backup' entries in BakTrak database where a `Y' is indicated for `NT,' the contents of the PS_Home file corresponding to the name under the column `ENVIRONMENT' would have been backed-up, which could include the use of some form of compressed or zip file.").
designated location on SAP TN's infrastructure for use.20 21 discrete discrete copy of some amount of Oracle software.21
Thus, each backup and/or restore is a Thus,
The following two tables represent sample actual data contained in BakTrak:
BCK_ID 1 MACHINE HOMER APPLIC ENVIRON FILENAME DATE_TIM DESCRIP ATION MENT E TION HRMS HR810DMO HR810DMO_2003 3/25/2003 with tax 0325_1600 15:18 updates through 01G HRMS H831OLNI H831OLNI_2006 1/19/2006 After 0119_0318 3:18 Maintenan ce Packs 3 &4 HRMS H831OLNI H831OLNI_2006 1/20/2006 After 0120_0818 8:18 Maintenan ce Packs 5 &6 HRMS H831OLNI H831OLNI_2006 1/21/2006 After Tax 0121_0347 3:47 Updates 05-C thru 05-F HRMS H801SPGM H801SPGM_2008 1/7/2008 PY08JAN 0107_2348 23:48 Applied & Tested DB Y Y NT UNI PERFORMED REQUESTE FORMAT X BY D BY N chyde NA .ZIP TN_ARCHIVE TNBK0044
Deleted from archive
Deleted from archive
Deleted from archive
H831CCIM_2008 1/7/2008 0107_2354 23:54
PY08JAN Applied & Tested
H881COHM H881COHM_2008 1/7/2008 0107_2351 23:51
PY08JAN Applied & Tested
\\tempstore\PSTEMP BKUP\dcpstemp01\p s_home, \\tempstore\PSTEMP BKUP\dcpsdb01\db backups\mssql \\tempstore\PSTEMP BKUP\dcpstemp01\p s_home, \\tempstore\PSTEMP BKUP\dcpsdb01\db backups\mssql \\tempstore\PSTEMP BKUP\dcpstemp02\p s_home, \\tempstore\PSTEMP BKUP\dcpsdb01\db backups\mssql
Table 5: A sample excerpt from BakTrak_Backups.xls referenced in full in eAppendix "ORCLX-MAN-000133"
RES MAC APPLI TARGET_ SOURCE_ RESTORE BACKUP_FILEN DATA NT_ TOR HIN CATIO ENV ENV _ARCHIVE AME BASE RES E_ID E N _RES TOR TORE E 51 YOG HRMS HG751AN HG75103F TNBK0094- HG75103F_2003 Y Y I C TNBK0091 1024_1800 (mssql7) HG75103F_2003 1023_0235 (from current - but cobol/sqr from backup of pshome) 57 YOG HRMS HS702DEV HS70203G TNBK0107 HS70203G_2003 Y Y I 1201_1541 69 YOG HRMS HR751TST HR75103G TNBK0110 HR75103G_2003 Y Y I 1213_1241 UNI RESTORE_D DESCRIPTION PERFOR REQUESTE X_R ATETIME MED_B DBY EST Y ORE N 11/11/2003 create Muni of Anchorage chyde NA 11:00 environment
93 YOG HRMS I
HG75103E_2003 Y 0825_1325
12/2/2003 create 03G dev chyde 19:59 12/29/2003 create temp env for chyde 9:55 testing of 'off' payrolls, I.e. 53 weeks, 27 cycles, etc. 3/22/2004 created starting chyde 12:29 environment for Cowlitz
20 See, e.g., Deposition of George Lester, April 23, 2009 at 43:11-48:20; Deposition of John Baugh, February 6, 2008 at 142:4-145:16; Deposition of John Baugh, February 7, 2008 at 290:24-297:9; Defendants Responses to Plaintiffs' Second Set of Requests for Admission, Nos. 217-219 ("Defendant SAP TN ADMITS that often in the `Restore' entries of the BakTrak database where a `Y' is indicated for the `NT_RESTORE,' the contents of the PS_Home file identified under the column `SOURCE_ENV' would have been restored to the name identified under the column `TARGET_ENV.'"). 21 This is my understanding based on conversations with Mr. Edward Screven and Mr. Norm Ackermann of This Oracle.
On November 11, 2008, Mandiant and an Oracle employee with extensive knowledge of World software, Greg Story, inspected Defendants' AS/400 in Bryan, Texas. In January 2009, Mandiant and Mr. Story further analyzed a restored version of this same AS/400. After the inspection, SAP TN created and produced complete backups of the ENT01 and WORLD partitions. On January 10, 2009, Mandiant was provided access to restored copies of a subset of SAP TN's libraries on an AS/400 system in Oracle's Denver offices.
Other Data Sources
Mandiant Mandiant relied upon deposition testimony of former SAP TN employees and corporate representatives representatives to further understand and determine the source of SAP TN's environments, the development development and testing processes, and SAP TN's data infrastructure. In addition to testimony, Mandiant also relied upon certain of SAP TN's produced documents and discovery responses. produced Mandiant understands that certain discovery responses are in the process of being amended and/or supplemented by SAP TN and will review those responses for further relevant information as they are made available, and will rely on them as necessary. Mandiant also understands that certain potentially relevant SAP TN and/or third-party witnesses are scheduled to be deposed in the near future, including Carol Geiger, Jeff Buehrle, Nhat Vuong, Wanda Jones, Greg Nelson, John Baugh, Jerry Jin, and others, and Mandiant will review the transcripts of those depositions for further relevant information as they are made available, and will rely on them as necessary.
In order to facilitate efficient and effective code comparisons between the Registered Works and software found on SAP TN's systems, Mandiant created several databases to support its analysis.
File-based Objects from PeopleSoft Registered Works
Mandiant created the "Oracle Registered File-based Objects" database to record data and metadata about the SQR, SQC, and CBL files created by the 25 sets of Install Media that Oracle identified as embodying certain of its Registered Works. These install media are listed in Table 10. Title of Oracle Registered Work PeopleTools PeopleTools PeopleTools PeopleTools PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft 7.5 8.0 8.10 8.4 Software Bates Number ORCL00264040 ORCL00264028 ORCL00264035 ORCL00264024 ORCL00264025 ORCL00264031 ORCL00400498 ORCL00400497 ORCL00264026 ORCL00264019 ORCL00264021 ORCL00604712 ORCL00400499 ORCL00466982 ORCL00264039 ORCL00264038
HRMS 7.0 HRMS 7.5 HRMS 8.0 HRMS 8 SP1 HRMS 8.3 HRMS 8.8 Financials, Distribution and Manufacturing 7 Financials 7.5
PeopleSoft Student Administration Solutions 8 PeopleSoft PeopleSoft Customer Relationship Management 8
Title of Oracle Registered Work PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft PeopleSoft Customer Relationship Management 8.1 Customer Relationship Management 8.8 Financials and Supply Chain Management Enterprise 8 Financials and Supply Chain Management 8 SP1 Rev 1 Financials and Supply Chain Management 8 SP2 Financials and Supply Chain Management 8.4 Enterprise Performance Management 8 SP3 Enterprise Performance Management 8.3 Rev 1 Enterprise Performance Management 8.8 Table 10: Install media embodying the PeopleSoft Application Software Registered Works
Software Bates Number ORCL00604718 ORCL00264027 ORCL00604715 ORCL00604716 ORCL00264022 ORCL00264037 ORCL00604717 ORCL00604719 ORCL00264023 Enterprise
In order to identify the File-based Objects within the Registered Works, Mandiant installed each application provided by Oracle. Specifically, Mandiant installed the 23 products in an operating environment dedicated solely to code-comparison tasks. Each ISO was installed individually with license codes provided by Oracle. After installation, Mandiant copied all the installed files to a common location for the code comparison process. The copy process preserved all the original file system paths. For example, if PeopleTools 7.5 (ORCL00264040) created an installation directory of C:\PT75, the complete PT75 directory was copied to a common storage location under a path "ORCL00264040/PT75." Mandiant created a custom program to identify the File-based Objects contained within all the installed Registered Works and populate the "Oracle Registered File-based Objects" database. For each unique File-based Object identified, Mandiant created a database record that contained the information listed in the table below: File-based Object Record for "Oracle Registered File-based Objects" Database Field Name FileID File Name File Hash File Type File Path ISO Name Release RCS Header24 RCS Release RCS Revision RCS VersionID RCS Date RCS Resolution Number of Comments (c_comments) Number of Lines in File (c_linecount) Presence of Oracle
Description Auto-generated unique index number Name of the COBOL, SQR, or SQC File Processed The unique MD5 hash for the file Whether the file was a CBL, SQR, or SQC file Full directory path of the file being imported into the database Bates Number assigned to an Oracle Registered work by BM Short Version of the Release name after the Registered Work was installed such as PT75 RCS comment for the RCS Header field RCS comment for the Release field RCS comment for the Revision field RCS comment for the Version ID RCS comment for the Date field RCS comment for the Resolution field An integer representing the number of lines that were comments An integer representing the total number of lines in the File-based Object Set to "1" if the strings "Copyright ([Cc]).*PeopleSoft, Inc." or
"RCS" or Revision Control System maintains specific text within code files up to date. RCS data can include the code's current version, update date, and other pertinent information.
Figure 9: Illustration of the File-based Object Comparison Process
Comparison of Downloaded SSMs to Automated Databases
Mandiant performed an MD5 hash comparison of the files identified within the Automated Databases to the downloaded SSMs. As described above, the files of interest from the Automated Databases were compiled by file extension, and 246,629 unique MD5 hashes were used in the comparison. The objective was to determine how much Oracle material from the Automated Databases was present within the Customer Download Folders. These comparisons were performed for each product line JD Edwards, PeopleSoft, and Siebel. See ORCLX-MAN-00045, ORCLX-MAN00046, ORCLX-MAN-000047, and ORCLX-MAN-000142.
Statistics on Automated Databases
Statistics on Automated Databases are listed below in Table 22:
Product Line JD JD Edwards PeopleSoft Siebel Siebel Total:
Number Number of Files Identified 108,264 14,261 28,564 151,089 Table 22: Automated Database File
Size of Identified Files in GB 9.52 10.39 1.33 21.24 Statistics on DCITBU0126
Statistics on Comparisons of Customer Download Files with Oracle Registered Software
Total number of matches found by MD5 Hash 151,089 Number of TN files compared with Automated Databases (DOC, HTM, 2,415,526 HTML, PDF, PPT, XLS, and ZIP) Number of TN files with a matching file name in the Automated Databases 147,678 Number of TN files without a matching file name in the Automated 3,411 Databases Table 23: Comparisons of Customer Download Files on DCITBU01 with Oracle Registered Software Statistics
Procedure for comparison Databases to the SSMs
Mandiant used the previously discussed files from the Automated Databases, specifically DOC, HTM, HTML, PDF, PPT, XLS, and ZIP files. Mandiant conducted the comparison of Automated Databases to the SSMs by utilizing the following process: a. Mandiant calculated the MD5 hash values of all originally provided files as well as the decompressed contents from each ZIP file and each nested ZIP file. Mandiant then searched for the 246,629 unique hash values across SAP TN's server "DCITBU01." Additionally, Mandiant conducted the same search across the entire Data Warehouse. Mandiant compiled the results and determined the total number of files and size of those files for each product line.
Mandiant also conducted a manual review of 2,687 files within the Siebel folder on DCITBU01 that had names similar to the names of html files found within the Siebel Automated Database. Mandiant visually determined that 2,435 files matched. The complete results are located in eAppendix - "ORCLX-MAN-000045", eAppendix "ORCLXMAN-000046", and eAppendix "ORCLX-MAN-000047."
Table 22 reports data concerning automated database files found on DCITBU01; Table 20 of my report reports data about automated database files on all SAP TN systems reviewed by Mandiant.
000354). The following table represents the total number of file name matches per custodian after Mandiant made these file name changes: Number of Total Number of File Additional File Name Name Matches Matches Normalized Cefola 1667 1 1668 Jahrsdoefer 2026 0 2026 Muvvalac 198 3005 3203 Phillips 198 3004 3202 Total 4,089 6010 10,099 Table F: File Name Matches of Siebel Downloads to the Siebel Automated Database In order to determine whether the contents were exact content matches, Mandiant manually reviewed 39 custodian download files: x x Three alerts, three tech notes and three FAQs, selected in Bates number order, that matched by file name to the Siebel automated database without any normalization. 10 alerts, 10 technotes, and 10 FAQ's, selected in Bates number order, that matched by file name to the Siebel automated database after normalization of file names. Custodian Number of File Name Matches Exact
Mandiant determined that 26 of the 30 normalized-name-match files and nine exact-namematch files were exact matches in download-specific content despite having been converted to a text file and stripped of all html code. For each of the four files that did not match exactly, the Automated Database contained an updated version of the download-specific content (See ORCLX-MAN-000363 and ORCLX-MAN-000380). These comparisons support my opinion that approximately 10,099 files present in the analyzed custodial productions are contained within the content of Oracle's automated database copyright registrations (see ORCLX-MAN-000353 and ORCLX-MAN-000354). Further, there is no way for Mandiant to verify that the SSMs in the custodial production were associated with any particular customers that might have had some license to download such files.
Comparison of SAP TN-attributed Fixes to the Registered Works 1. Overview
Mandiant Mandiant performed a file-by-file comparison of the SAP TN PeopleSoft Delivered Updates and Fixes File-based Objects to the Registered Works Registered Works). The objective was to determine how much copyrighted Oracle material was included in the SAP TN Delivered Updates and Fixes. These comparisons did not include JD Edwards or Siebel products, and only considered the File-based Objects represented in Table 21. x x 98.4% of the File-based Objects within the SAP TN Delivered Updates and Fixes contained the Oracle copyright statement. 98% of the File-based Objects within the SAP TN Delivered Updates and Fixes contained the Oracle confidentiality statement.
52.5% of the File-based Objects within the SAP TN Delivered Updates and Fixes contained more than 90% of the best-match of the code in the Registered Works.
Total Number of File-based Objects (unique) 31,084 Number of CBL Files 13,673 Number of SQR Files 10,135 Number of SQC Files 7,276 Number of Files Containing a Copyright Statement 31,080 Number of Files Lacking a Copyright Statement 4 Number of Files Containing a Confidentiality Statement 31,080 Number of Files Lacking a Confidentiality Statement 4 Table 24: Oracle/ PeopleSoft Software Statistics
Statistics on SAP TN Delivered Updates and Fixes Software
Total Number of File-based Objects 6,447 Number of CBL Files 1,538 Number of SQR Files 3,801 Number of SQC files 1,108 Number of Files Containing a Copyright Statement 6,358 Number of Files Lacking a Copyright Statement 89 Number of Files Containing a Confidentiality Statement 6,329 Number of Files Lacking a Confidentiality Statement 118 Table 25: SAP TN Delivered Updates and Fixes Statistics
Procedure for Comparison of the Oracle Registered File-based Objects to the TN Delivered File-based Objects
Mandiant compared pairs of files selected from the two data sources; the File-based Objects from Oracle/PeopleSoft registered software and File-based Objects from SAP TN's Delivered Updates and Fixes repository. Mandiant populated a Code Compare database with the information outlined in the table below: Information Within the Code Compare Database Field Name compareID clientfix_fileID ps_fileID clientfix_lc Description Auto-generated unique index number File ID of a File-based Object from the SAP TN DUF data set File ID of a File-based Object from the Oracle Registered data set Number of lines in the SAP TN file
ps_lc c_pctdup c_duplicate c_TNnew c_PSnew c_change c_leftig c_rightig diff_filename
Number of lines in the Oracle registered file Percent of Oracle Registered file in the TN file Number of duplicate lines Number of new lines in the SAP TN file Number of new lines in the Oracle file Number of lines with minor changes Number of lines in Oracle file with minor changes Number of lines in TN file with minor changes File name of diff output Table 26: Code Compare Database Fields
After the metadata and extracted data from both data sets were loaded into a database, Mandiant iterated through the Delivered Updates and Fixes table. Each file name from this table was compared against the file names in the Oracle/PeopleSoft registered software table. If a match was found, a comparison was performed. No operations were performed on the original files that affected the contents. The flowchart in Figure 11 shows the process used to select and compare every Filebased Object in the "TN Delivered File-based Objects" database. Mandiant created a custom program to automate the following process: a. The custom program accessed the two tables described in the prior sections of this report. Mandiant's custom program loaded a file entry from the "TN Delivered File-based Objects" database for analysis. Note that this step executed on every file referenced in the "TN Delivered File-based Objects" database. This ensured each TN Delivered File-based Object would be compared to all Registered Works that had the same file name. The program searched the "Oracle Registered File-based Objects" database for every file that matched the selected File-based Object from the "TN Delivered File-based Objects" database. If no Oracle Registered File-based Object's name matched the TN Delivered File-based Object's name, this was noted in the Code Compare database with a "placeholder record," described in Step E. Otherwise the process continued on to Step F. When no files matched by file name, a placeholder record was populated with the delivered update and fix file index number (a unique value assigned to the delivered code-based object record by the database) and a value of "0" in the PeopleSoft file ID field in the Code Compare database. The program started the process over by loading a new TN delivered File-based Object entry for comparison. If one or more files in the "Oracle Registered File-based Objects" database had the same file name as the TN delivered File-based Object, the process looped through Steps I-J for each pair of file name matches. For example, if files A, B and C from the Registered Works have the same file name as file #1 from the TN Updates and Fixes, the following three comparisons were performed: x x x File #1 to File A File #1 to File B File #1 to File C
The program searched the Code-Compare database to determine whether the specific comparison had been performed. This comparison was based on the database index for each file record, the DUF file ID, and the PeopleSoft registered software file ID. If the comparison had not been performed, the process continued by comparing the next pair of matched files or the process started over, and the program selected a new TN Delivered File-based Object for comparison. If the comparison had not been performed, the process used a program called "diff" to generate an automated comparison of the two files. The "diff" process generated an output file that reported on several conditions as it performed a line-by-line comparison. The output file from the "diff" process was preserved in a storage location for reference at a later date. All "diff" output files are available for manual review. Automated analysis of the "diff" output file was performed to extract the following results. Number of duplicate lines between the TN Delivered File-based Object and the Oracle Registered file.
m. Number of lines unique to the TN Delivered File-based Object. n. o. Number of lines unique to the Oracle Registered file. Number of lines with minor changes between the TN Delivered file and Oracle Registered file. Using the following equation, the program calculated the percentage of Oracle/PeopleSoft registered code that was present in the SAP TN Delivered Updates and Fixes File-based Object. This figure was labeled as the value "c_pctdup" in the Code-Compare database.
c _ pctdup
# of lines in PeopleSoft Code
# of duplicate lines
Figure 10: Equation used to calculate percentage of Oracle/PeopleSoft registered code that was present in the SAP TN Delivered Updates and Fixes File-based Object The data collected in this iteration of the process was saved to the database. If additional File-based Objects from the Registered Works matched the selected SAP TN Delivered Updates and Fixes file, the program would return to Step F. Otherwise, the process would end for the selected SAP TN file. The next TN file would be selected and the process would begin again at step b. eAppendix - "ORCLX-MAN-000202"
Figure 11: Illustration of the File-based Object Comparison Process
Procedure for Comparison of the Remaining Delivered Updates and Fixes File-based Objects to the Customer Download Folder File-based Objects
Mandiant compared pairs of files selected from the two data sources; the File-based Objects from SAP TN's PeopleSoft Delivered Updates and Fixes that had no file name match in previous
comparisons and File-based Objects from the Customer Download Folder repository. populated a Code Compare database with the information outlined in the table below: Information Within the Code Compare Database Field Name compareID CompareHash DUF_fileID softwaresupport_fileID DUF_lc softwaresupport_lc c_pctdup c_duplicate c_TNnew c_PSnew c_change c_leftig c_rightig diff_filename
Description Auto-generated unique index number Hash that uniquely identifies a particular comparison File ID of a File-based Object from the SAP TN DUF data set File ID of a File-based Object from the Oracle Customer Download Folder data set Number of lines in the SAP TN Delivered Updates and Fixes file Number of lines in the Oracle Customer Download Folder file Percent of Oracle Customer Download Folder file in the TN file Number of duplicate lines Number of new lines in the SAP TN file Number of new lines in the Oracle file Number of lines with minor changes Number of lines in Oracle file with minor changes Number of lines in TN file with minor changes File name of diff output Table 27: Code Compare Database Fields
After the metadata and extracted data from both data sets were loaded into a database, Mandiant iterated through a list of Delivered Updates and Fixes that did not have a file name match in the previous comparison operation (TN Delivered Updates and Fixes vs. Oracle Registered). Each file name from this table was compared against the file names in the SAP TN Customer Download Folder table. If a match was found, a comparison was performed. No operations were performed on the original files that affected the contents. The comparisons run during this operation duplicated the process described in the section titled "Comparing the Oracle Registered File-based Objects to the TN Delivered File-based Objects."
Findings Re Data Comparisons a. Delivered Updates and Fixes Comparisons to Registered Works
Count 28,271 6,447 5,475 972 868
Item Total Number of Comparisons Performed Number of TN Files Compared with Oracle data (SQR, SQC, CBL) Number of TN Files with a Matching a File Name in the Register Oracle Data Number of TN Files without a Matching File Name in the Registered Oracle Data Number of TN Files that did Not Match an Oracle Registered Work Filename (972) with a Matching File Name found in the SSMs Download Server (DCITBU01). Number of TN Files Without a Matching File Name in the Registered Oracle Works or the SSMs on the Download Server (DCITBU01) Table 28: Statistics of Comparisons of Delivered Updates and Fixes
104 to Registered Works
Figure 12: Percentage of Oracle Registered Works Contained within the SAP TN Delivered Updates and Fixes
Figure 13: Distribution of SAP TN Delivered Updates and Fixes Best Match Percentages When Compared to Registered Oracle Works
Delivered Updates and Fixes Comparisons to Downloaded SSMs in the Customer Download Folders
Figure 14: Percentage of SSMs in Customer Download Folders Contained Within SAP TN Delivered Updates and Fixes
Figure 15: Distribution of SAP TN Delivered Updates and Fixes Best Match Percentages When Compared to SSMs in Customer Download Folders eAppendix "ORCLX-MAN-000014"
SAP TN Environments
Mandiant compared a sample set of SAP TN Environments against different installations of Registered Works. Comparisons between the Environments and the associated Registered Works took into account the closest matches in version numbers and application type as determined from the Environment to come to the results provided.
Comparison of SAP TN PS_HOMEs to PeopleSoft Application Software Registered Works a. Methodology
For purposes of the comparison, Mandiant regarded the SAP TN Environments as composed of two separate PeopleSoft applications: the PeopleSoft base application and an associated PeopleTools version. Each Environment comparison was conducted against a registered version of a PeopleSoft base application program and a Registered Work of a PeopleSoft PeopleTools application. Each of the 41 Environments examined was compared to one or more base applications and versions of
Mandiant calculated the percentage of total unique "first deliverable" ".DAT" files that were delivered to customers with a mismatched environment reference. Step 1: Mandiant identified each occurrence of ".DAT" files delivered to a customer where that ".DAT" file was created in another customer's environment or created in a generic environment (DAT Contamination captured in Measure 136). Step 2: Mandiant then divided by the total number of unique "first deliverable" ".DAT" files identified in Measure 142. (Measure 136 ÷ Measure 142) was calculated on a per fix basis.
Detailed analysis of TN Fix 1012062843 for JD Edwards World A7.3 1. Evidence Analyzed
As discussed above, SAP TN after Mandiant's inspection of SAP TN's AS/400 in Bryan, Texas, SAP TN created backups of that server's ENT01 and WORLD partitions. In order to review source code of interest, Oracle restored the libraries of interest from tape backups generated from the SAP TN AS/400 system in Bryan, Texas to an AS/400 system in Oracle's Denver offices. On January 10, 2009, Mandiant was provided access to restored copies of the SAP TN libraries listed in Table 46. These libraries were selected by Mandiant and Oracle. Mandiant reviewed the source code with assistance from Greg Story, an Oracle Senior Database Administrator familiar with installations of J.D. Edwards World on AS/400 servers. List of Libraries Provided by SAP TN BBTNSC81 I807896 KWESECA73 BSI2006 I807916 KWESRC CDF2006 I808319 KWFIX CDF2006_1 I808745 KWJDOB73 DCC2006 I808892 KWSECA73 EDI2006 I810590 KWTNOBJ EGGER JDFOBJCM2 KWTSCM73 EGGER1 KASSEC LPC2006 I807655 KNW2006 LXK2006 I807656 KWEOBJ NCI2006 Table 46: List of Libraries Provided by SAP TN
BBDEVOB81 BBDEVSC81 BBJDOB81 BBJDSC81 BBLWORK BBMISSING BBMODOB81 BBMODSC81 BBSEC81 BBTNOB81
SBCOBJ SBCSRC SPX2006 SSI2006 TNOW TNTSOB73 TNTSSC73 TSB2006 VKA2006
Mandiant focused its analysis on 13 libraries with names ending in "2006." These contained source code objects related to World Year-end 2006 updates. These updates were primarily related to changes in tax laws. Library BSI2006 CDF2006 CDF2006_1 Description Binney Smith 2006 YE Changes CAPTAIN D 2006 YE Changes CAPTAIN D 2006 2nd YE Changes
TN06 I** 394 395 TTTFI TN06 I** 396 410 TTEFLN eAppendix "ORCLX-MAN-000207 provides additional details about similar code in the members found in 12 objects. x x x Items marked "Different" illustrate where the changes made in the member were not similar to the changes in the corresponding member within the KNW2006 object. Items marked "None" denotes when a member was not changed or updated in 2006. The items marked "NA" illustrate where a member did not exist in the corresponding object.
DCC2006 TNSRC CDF2006 JDESRC TSB2006 JDESRC EDI2006 JDESRC55 LPC2006 JDESRC LXK2006 JDESRC NCI2006 JDESRC SPX2006 JDESRC SSI2006 JDESRC VKA2006 JDESRC
J04515 J04515JQ P04512FP P04515 P045151 S045154 V04515
Same Same Same Similar Similar Similar Similar
Same Same Same Similar Similar Similar Similar
Same Same Same Similar Similar Similar Same
Same Same Similar Same Similar Similar Similar
Same Same Similar Similar Similar Similar Similar
Same Same NA Same Same NA Same
Different None NA None None NA None
None None None None None NA None
Same Same NA Similar Similar NA Same
Similar Similar Same Similar Similar Similar Same
Table 50: Comparison of Changes Made in KNW2006/JDESRC to Changes in Other Objects Object Name SSI2006/JDESRC Additional Information 14 additional members (P06735, P06765, P06767, P06767A, P06770, P067703, P06771, P06771L, S06770, S067701, S067702, S067703, S06771, S06771L) had the same changes and comments as those made in KNW; three additional members (P06761, P067701, P067702) had the same changes and similar comments.
BSI2006/JDESRC No members CDF2006_1/JDESRC No changes to members (only two, P04512FP and S045154, are present) Table 51: Additional information on Year End 2006 library objects
JD Edwards System Code Analysis
Mandiant used information provided by Oracle and internally generated documents to match files found within JD Edwards customer specific folders on DCITBU01 to Oracle system codes. Mandiant followed two separate processes depending on whether World or OneWorld downloads were identified within the customer folder. Mandiant performed these analyses to determine if SAP TN had downloaded files that they were not licensed to according to Exhibit 1634.
Mandiant identified all unique customer specific folders with JD Edwards products across SAP TN's central download repository, identified as "DCITBU01." Mandiant then extracted metadata for each identified customer specific folder about every file within that folder using EnCase.
The Object EDI2006/JDESRC contained 28 additional members that were not contained in the KNW2006/JDESRC Object. None of these members were changed or updated in 2006.
Mandiant analyzed the exported metadata for each image to identify which JD Edwards product lines that company was using World, OneWorld, or both. Mandiant conducted a separate analysis depending on what product line(s) were identified in the previous step.
If the identified product line was OneWorld on Exhibit 1634, Mandiant performed the following: a. Mandiant created a subset of files consisting of customer files that both had a matching two letter prefix and were identified in the Reverse Proxy Logs.56 (See ORCLX-MAN009). Mandiant matched the first two-letter prefix of all files to a known list verbally provided by Oracle. See ORCLX-MAN-000015. Mandiant made corresponding matches of identified files to specific version levels of the OneWorld product using material verbally supplied by Oracle. Mandiant identified the number of files each company was licensed to download by matching the determined product version information found in the previous step to the information in Exhibit 1634.
If the identified product line was World on Exhibit 1634, Mandiant performed the following: a. Mandiant compiled information provided by Oracle to generate a filename to system code mapping. See ORCLX-MAN-000013. Mandiant compared the filenames within each identified customer folder to the filenames in the system code mapping. Mandiant identified all files which could be matched by filename and their corresponding system code. Mandiant removed files corresponding to multiple system codes from the analysis. Mandiant analyzed the system codes using the information provided in Exhibit 1634 to determine how many licensed and unlicensed files were within each company folder.
Mandiant identified 13,737 unlicensed file downloads for OneWorld customers and 23,612 unlicensed downloads for World. The complete results of both the OneWorld and World analyses can be found in eAppendix - "ORCLX-MAN-000103," eAppendix "ORCLX-MAN-000104", eAppendix "ORCLX-MAN-116".
Oracle Provided System Codes
Oracle provided Mandiant with lists of system codes for Merck, Metro Machine Corporation, OCE Technologies, SPX, and Yazaki North America. See ORCLX-000002, ORCLX-MAN-000003, ORCLX-MAN-000004, ORCLX-MAN-000005, ORCLX-MAN-000006. Oracle more specifically produced
The list of files identified in the Reverse Proxy Logs is generally present as the "System.Codes" tab in the customer-specific spreadsheets produced at ORCLX-MAN-000220 to ORCLX-MAN-000264, ORCLX-MAN000330, and ORCLX-MAN-000331. The system code information was derived from ORCLX-MAN-000016 and ORCLX-MAN000017.
to Mandiant OneWorld system codes for Merck, OCE Technologies, SPX, and Yazaki North America as well as World system codes for Metro Machine Corporation, OCE Technologies, and Yazaki North America. Mandiant adhered to the methodology outlined above for all World analyses; however, Mandiant performed the process outlined below to analyze the OneWorld system codes. For provided OneWorld system codes, Mandiant utilized information internally generated to match filenames to corresponding system codes: a. Mandiant used the generated information to make comparisons between all filenames in customer folders and the Reverse Proxy Logs to match them to system codes.57 Mandiant removed files corresponding to multiple system codes from the analysis. Mandiant determined the total number and metadata about which files each company was and was not licensed to by comparing the system codes found in the previous step to the Oracle provided system codes.
Company Folder Merck OCE Technologies SPX Cooling SPX Flow SPX Weil-McLain Yazaki Metro Machine Total
Unlicensed Files OneWorld 3,450 1,570 2,674 807 3,597 10,250 0
Unlicensed Files World 0 3,076 0 0 0 1,547 4,363 8,986
22,348 Table 52: Results of Oracle Provided System Code Analysis
From the analysis Mandiant determined that 31,334 files were unlicensed downloads. The total results for both analyses can be found in eAppendix - "ORCLX-MAN-000116." Mandiant similarly analyzed the downloads identified in log files, which results can be found in ORCLX-MAN-000137. See also ORCLX-MAN-000310, ORCLX-MAN-000311, ORCLX-MAN-000314, and ORCLX-MAN-000315.
Removal of files corresponding to multiple system codes
As stated above, files corresponding to multiple system codes were removed for all analyses of World described in Section 1 and for the analyses of OneWorld described in Section 2. For example, the "System.Codes" tab found in ORCLX-MAN-000264 includes the following data:
Excel Row # 23
System Code 43 - Purchase Order Processing
24 JD10222 30 - Product Data Management 48 - Product Data Management Table S: Sample contents of ORCLX-MAN-000264, "System.Codes" tab
See preceding footnote.
Row 24 would be removed from analysis of files on a system-code basis, since the filename corresponded to multiple system codes. Row 23 would be retained, since the filename corresponded to a single system code. See ORCLX-MAN-000383, a listing for the World and OneWorld product lines of each filename tied to a single system code.
Registered ESU's and Other Registered Works
Mandiant was provided with a number of Oracle registered ESU's and other Registered Works. Mandiant searched for these files across 58 SAP TN hard drive images and DCITBU01 using the subsequent procedure. a. Mandiant first extracted all the contents of an Oracle provided .ISO file identified with the ID number, 00264056. Mandiant then compiled the extracted contents with other provided Registered Works. Mandiant calculated the MD5 hash value of all of the extracted contents and other Registered Works. Mandiant used Guidance Software's EnCase to compile these MD5 hash values into a hash set. Mandiant searched for any MD5 matches across the 58 SAP TN hard drive images and DCITBU01 using EnCase. When matches were found, Mandiant exported metadata such as file path information and extension. See ORCLX-MAN-000146. For a small subset of the registered files, Mandiant employed keyword searches within EnCase to identify specific Solution ID's provided by Oracle. See ORCLX-MAN-000332. Mandiant compiled the metadata and extrapolated the number of registered ESU's found as well as the physical location of each registered file.
SAP TN Server DCITBU01 JDWSVR01 DCJDWDEV01 Total:
Number of Registered ESUs 87 26 1 114
Table 53: Location of Matching ESUs The full results of the search can be found in eAppendix "ORCLX-MAN-000146." Mandiant identified six other Registered Works; the full reference is in eAppendix - "ORCLX-MAN-000145" and eAppendix "ORCLX-MAN-000332."
JD Edwards OneWorld Xe Analysis
Mandiant attempted to identify default installations of the Xe product throughout SAP TN systems through the identification of .C and .H files. Mandiant performed a search for all identified files according to the following procedure. a. Mandiant identified all .C and .H files within a default Xe installation.
Disclaimer: Justia Dockets & Filings provides public litigation records from the federal appellate and district courts. These filings and docket sheets should not be considered findings of fact or liability, nor do they necessarily reflect the view of Justia.
Why Is My Information Online?