Oracle Corporation et al v. SAP AG et al

Filing 860

Declaration of Chad Russell in Support of 859 Memorandum in Opposition, to Defendants' Motion to Partially Exclude Testimony of Kevin Mandia and Daniel Levy filed byOracle International Corporation, Oracle USA Inc., Siebel Systems, Inc.. (Attachments: # 1 Exhibit A, # 2 Exhibit B, # 3 Exhibit C, # 4 Exhibit D, # 5 Exhibit E, # 6 Exhibit F, # 7 Exhibit G, # 8 Exhibit H, # 9 Exhibit I, # 10 Exhibit J, # 11 Exhibit K, # 12 Exhibit L, # 13 Exhibit M, # 14 Exhibit N, # 15 Exhibit O, # 16 Exhibit P, # 17 Exhibit Q, # 18 Exhibit R, # 19 Exhibit S, # 20 Exhibit T, # 21 Exhibit U, # 22 Exhibit V)(Related document(s) 859 ) (Russell, Chad) (Filed on 9/9/2010)

Oracle Corporation et al v. SAP AG et al

Doc. 860 Att. 21

EXHIBIT U
Dockets.Justia.com

Reference Manual on Scientific Evidence
Second Edition Federal Judicial Center 2000

This Federal Judicial Center publication was undertaken in furtherance of the Center's statutory mission to develop and conduct education programs for judicial branch employees. The views expressed are those of the authors and not necessarily those of the Federal Judicial Center.

An electronic version of the Reference Manual can be downloaded from the Federal Judicial Center's site on the World Wide Web. Go to http://air.f j c.gov/public/f j c web . n s f/pages/16 For the Center's overall homepage on the Web, go to http://www.f j c.gov

Reference Guide on Statistics
d av i d h. k a y e a n d dav i d a. f r e e d m a n
David H. Kaye, M.A., J.D., is Regents' Professor, Arizona State University College of Law, and Fellow, Center for the Study of Law, Science, and Technology, Tempe, Arizona David A. Freedman, Ph.D., is Professor of Statistics, University of California, Berkeley, California

con t e n t s I. Introduction, 85 A. Admissibility and Weight of Statistical Studies, 86 B. Varieties and Limits of Statistical Expertise, 86 C. Procedures that Enhance Statistical Testimony, 88 1. Maintaining Professional Autonomy, 88 2. Disclosing Other Analyses, 88 3. Disclosing Data and Analytical Methods Before Trial, 89 4. Presenting Expert Statistical Testimony, 89 II. How Have the Data Been Collected? 90 A. Is the Study Properly Designed to Investigate Causation? 90 1. Types of Studies, 90 2. Randomized Controlled Experiments, 93 3. Observational Studies, 94 4. Can the Results Be Generalized? 96 B. Descriptive Surveys and Censuses, 98 1. What Method Is Used to Select the Units? 98 2. Of the Units Selected, Which Are Measured? 101 C. Individual Measurements, 102 1. Is the Measurement Process Reliable? 102 2. Is the Measurement Process Valid? 103 3. Are the Measurements Recorded Correctly? 104 III. How Have the Data Been Presented? 104 A. Are Rates or Percentages Properly Interpreted? 105 1. Have Appropriate Benchmarks Been Provided? 105 2. Have the Data-Collection Procedures Changed? 105 3. Are the Categories Appropriate? 106 4. How Big Is the Base of a Percentage? 107 5. What Comparisons Are Made? 107 B. Is an Appropriate Measure of Association Used? 108 C. Does a Graph Portray Data Fairly? 110 1. How Are Trends Displayed? 110 2. How Are Distributions Displayed? 112 D. Is an Appropriate Measure Used for the Center of a Distribution? 113

83

Reference Manual on Scientific Evidence

E. Is an Appropriate Measure of Variability Used? 114 IV. What Inferences Can Be Drawn from the Data? 115 A. Estimation, 117 1. What Estimator Should Be Used? 117 2. What Is the Standard Error? The Confidence Interval? 117 B. Significance Levels and Hypothesis Tests, 121 1. What Is the p-value? 121 2. Is a Difference Statistically Significant? 123 C. Evaluating Hypothesis Tests, 125 1. What Is the Power of the Test? 125 2. One- or Two-tailed Tests? 126 3. How Many Tests Have Been Performed? 127 4. Tests or Interval Estimates? 128 5. What Are the Rival Hypotheses? 129 D. Posterior Probabilities, 131 V. Correlation and Regression, 133 A. Scatter Diagrams, 134 B. Correlation Coefficients, 135 1. Is the Association Linear? 137 2. Do Outliers Influence the Correlation Coefficient? 137 3. Does a Confounding Variable Influence the Coefficient? 138 C. Regression Lines, 139 1. What Are the Slope and Intercept? 140 2. What Is the Unit of Analysis? 141 D. Statistical Models, 143 1. A Social Science Example, 145 2. Standard Errors, t-statistics, and Statistical Significance, 148 3. Summary, 148 Appendix, 151 A. Probability and Statistical Inference, 151 B. Technical Details on the Standard Error, the Normal Curve, and Significance Levels, 153 Glossary of Terms, 160 References on Statistics, 178

84

Reference Guide on Statistics

I. Introduction
Statistics, broadly defined, is the art and science of gaining information from data. For statistical purposes, data mean observations or measurements, expressed as numbers. A statistic may refer to a particular numerical value, derived from the data. Baseball statistics, for example, is the study of data about the game; a player's batting average is a statistic. The field of statistics includes methods for (1) collecting data, (2) analyzing data, and (3) drawing inferences from data. Statistical assessments are prominent in many kinds of cases, ranging from antitrust to voting rights. Statistical reasoning can be crucial to the interpretation of psychological tests, toxicological and epidemiological studies, disparate treatment of employees, and DNA fingerprinting; this list could easily be extended.1 This reference guide describes the elements of statistical thinking. We hope that the explanations provided will permit judges and lawyers who deal with statistical evidence to understand the terminology, place the evidence in context, appreciate its strengths and weaknesses, and apply legal doctrine governing the use of such evidence. The reference guide is organized as follows: · Section I provides an overview of the field, discusses the admissibility of statistical studies, and offers some suggestions about procedures that encourage the best use of statistical expertise in litigation. · Section II addresses data collection. The design of a study is the most important determinant of its quality. The section reviews controlled experiments, observational studies, and surveys, indicating when these designs are likely to give useful data. · Section III discusses the art of describing and summarizing data. The section considers the mean, median, and standard deviation. These are basic descriptive statistics, and most statistical analyses seen in court use them as building blocks. Section III also discusses trends and associations in data as summarized by graphs, percentages, and tables. · Section IV describes the logic of statistical inference, emphasizing its foundations and limitations. In particular, this section explains statistical estimation, standard errors, confidence intervals, p-values, and hypothesis tests. · Section V shows how relationships between two variables can be described by means of scatter diagrams, correlation coefficients, and regression lines. Statisticians often use regression techniques in an attempt to infer causation
1. See generally Statistics and the Law (Morris H. DeGroot et al. eds., 1986); Panel on Statistical Assessments as Evidence in the Courts, National Research Council, The Evolving Role of Statistical Assessments as Evidence in the Courts (Stephen E. Fienberg ed., 1989) [hereinafter The Evolving Role of Statistical Assessments as Evidence in the Courts]; Michael O. Finkelstein & Bruce Levin, Statistics for Lawyers (1990); 1 & 2 Joseph L. Gastwirth, Statistical Reasoning in Law and Public Policy (1988); Hans Zeisel & David Kaye, Prove It with Figures: Empirical Methods in Law and Litigation (1997).

85

Reference Manual on Scientific Evidence

from association; section V briefly explains the techniques and some of their limitations. · An appendix presents certain technical details, and the glossary defines many statistical terms that might be encountered in litigation.

A. Admissibility and Weight of Statistical Studies
Statistical studies suitably designed to address a material issue generally will be admissible under the Federal Rules of Evidence. The hearsay rule rarely is a serious barrier to the presentation of statistical studies, since such studies may be offered to explain the basis for an expert's opinion or may be admissible under the learned treatise exception to the hearsay rule.2 Likewise, since most statistical methods relied on in court are described in textbooks and journal articles and are capable of producing useful results when carefully and appropriately applied, such methods generally satisfy important aspects of the "scientific knowledge" requirement articulated in Daubert v. Merrell Dow Pharmaceuticals, Inc.3 Of course, a particular study may use a method that is entirely appropriate, but so poorly executed that it should be inadmissible under Federal Rules of Evidence 403 and 702.4 Or, the method may be inappropriate for the problem at hand and thus lacks the "fit" spoken of in Daubert.5 Or, the study may rest on data of the type not reasonably relied on by statisticians or substantive experts, and hence run afoul of Federal Rule of Evidence 703. Often, however, the battle over statistical evidence concerns weight or sufficiency rather than admissibility.

B. Varieties and Limits of Statistical Expertise
For convenience, the field of statistics may be divided into three subfields: probability, theoretical statistics, and applied statistics. Theoretical statistics is the study of the mathematical properties of statistical procedures, such as error rates; probability theory plays a key role in this endeavor. Results may be used by
2. See generally 2 McCormick on Evidence §§ 321, 324.3 (John W. Strong ed., 5th ed. 1999). Studies published by government agencies also may be admissible as public records. Id. § 296. See also United States v. Esquivel, 88 F.3d 722, 727 (9th Cir. 1996) (taking judicial notice of 1990 census data showing the number of Hispanics eligible for jury service). 3. 509 U.S. 579, 589­90 (1993). For a discussion of the implications and scope of Daubert generally, see 1 Modern Scientific Evidence: The Law and Science of Expert Testimony § 1­3.0 (David L. Faigman et al. eds., 1997). 4. See, e.g., Sheehan v. Daily Racing Form, Inc., 104 F.3d 940, 942 (7th Cir. 1997) ("failure to exercise the degree of care that a statistician would use in his scientific work, outside of the context of litigation" renders analysis inadmissible under Daubert). 5. 509 U.S. at 591; cf. People Who Care v. Rockford Bd. of Educ., 111 F.3d 528, 537­38 (7th Cir. 1997) ("a statistical study that fails to correct for salient explanatory variables, or even to make the most elementary comparisons, has no value as causal explanation and is therefore inadmissible in a federal court"); Sheehan, 104 F.3d at 942 (holding that expert's "failure to correct for any potential explanatory variables other than age" made the analyst's finding that "there was a significant correlation between age and retention" inadmissible).

86

Reference Guide on Statistics

applied statisticians who specialize in particular types of data collection, such as survey research, or in particular types of analysis, such as multivariate methods. Statistical expertise is not confined to those with degrees in statistics. Because statistical reasoning underlies all empirical research, researchers in many fields are exposed to statistical ideas. Experts with advanced degrees in the physical, medical, and social sciences--and some of the humanities--may receive formal training in statistics. Such specializations as biostatistics, epidemiology, econometrics, and psychometrics are primarily statistical, with an emphasis on methods and problems most important to the related substantive discipline. Individuals who specialize in using statistical methods--and whose professional careers demonstrate this orientation--are most likely to apply appropriate procedures and correctly interpret the results. On the other hand, forensic scientists and technicians often testify to probabilities or statistics derived from studies or databases compiled by others, even though some of these testifying experts lack the training or knowledge required to understand and apply the information. State v. Garrison6 illustrates the problem. In a murder prosecution involving bite-mark evidence, a dentist was allowed to testify that "the probability factor of two sets of teeth being identical in a case similar to this is, approximately, eight in one million," even though "he was unaware of the formula utilized to arrive at that figure other than that it was `computerized.'"7 At the same time, the choice of which data to examine, or how best to model a particular process, could require subject matter expertise that a statistician might lack. Statisticians often advise experts in substantive fields on the procedures for collecting data and often analyze data collected by others. As a result, cases involving statistical evidence often are (or should be) "two-expert" cases of interlocking testimony.8 A labor economist, for example, may supply a definition of the relevant labor market from which an employer draws its employees, and the statistical expert may contrast the racial makeup of those hired to the racial composition of the labor market. Naturally, the value of the statistical analysis depends on the substantive economic knowledge that informs it.9
6. 585 P.2d 563 (Ariz. 1978). 7. Id. at 566, 568. 8. Sometimes a single witness presents both the substantive underpinnings and the statistical analysis. Ideally, such a witness has extensive expertise in both fields, although less may suffice to qualify the witness under Fed. R. Evid. 702. In deciding whether a witness who clearly is qualified in one field may testify in a related area, courts should recognize that qualifications in one field do not necessarily imply qualifications in the other. 9. In Vuyanich v. Republic National Bank, 505 F. Supp. 224, 319 (N.D. Tex. 1980), vacated, 723 F.2d 1195 (5th Cir. 1984), defendant's statistical expert criticized the plaintiffs' statistical model for an implicit, but restrictive, assumption about male and female salaries. The district court trying the case accepted the model because the plaintiffs' expert had a "very strong guess" about the assumption, and her expertise included labor economics as well as statistics. Id. It is doubtful, however, that economic knowledge sheds much light on the assumption, and it would have been simple to perform a less restrictive analysis. In this case, the court may have been overly impressed with a single expert who

87