Oracle Corporation et al v. SAP AG et al

Filing 835

Memorandum in Opposition to 773 Plaintiff's Motion No. 6 to Exclude Expert Testimony of Bruce Spencer filed by SAP AG, SAP America Inc, Tomorrownow Inc. (Attachments: # 1 Proposed Order)(Froyd, Jane) (Filed on 9/9/2010) Modified on 9/10/2010 (vlk, COURT STAFF).

Oracle Corporation et al v. SAP AG et al

Doc. 835

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Robert A. Mittelstaedt (SBN 060359) Jason McDonell (SBN 115084) Elaine Wallace (SBN 197882) JONES DAY 555 California Street, 26th Floor San Francisco, CA 94104 Telephone: [phone] Facsimile: [phone] [e-mail] [e-mail] [e-mail] Tharan Gregory Lanier (SBN 138784) Jane L. Froyd (SBN 220776) JONES DAY 1755 Embarcadero Road Palo Alto, CA 94303 Telephone: [phone] Facsimile: [phone] [e-mail] [e-mail] Scott W. Cowan (Admitted Pro Hac Vice) Joshua L. Fuchs (Admitted Pro Hac Vice) JONES DAY 717 Texas, Suite 3300 Houston, TX 77002 Telephone: [phone] Facsimile: [phone] [e-mail] [e-mail] Attorneys for Defendants SAP AG, SAP AMERICA, INC., and TOMORROWNOW, INC. UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF CALIFORNIA OAKLAND DIVISION ORACLE USA, INC., et al., Plaintiffs, v. SAP AG, et al., Defendants. Case No. 07-CV-1658 PJH (EDL) DEFENDANTS' OPPOSITION TO PLAINTIFF'S MOTION NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Date: Time: Courtroom: Judge: September 30, 2010 2:30 p.m. 3, 3rd Floor Hon. Phyllis J. Hamilton

HUI-131254v1

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

Dockets.Justia.com

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
HUI-131254v1

TABLE OF CONTENTS Page I. II. III. IV. V. INTRODUCTION .............................................................................................................. 1 SUMMARY OF SPENCER'S REBUTTAL OPINIONS AND REPORT ........................ 2 LEGAL STANDARD......................................................................................................... 3 SPENCER IS QUALIFIED TO REBUT LEVY ................................................................ 4 SPENCER'S REBUTTAL OPINIONS WILL ASSIST THE JURY IN UNDERSTANDING AND WEIGHING LEVY'S OPINIONS ON SAMPLING AND STATISTICS............................................................................................................. 5 A. Applicable Law ....................................................................................................... 6 B. Spencer's Rebuttal Opinions Are Valid and Relevant Statistical Criticisms of Levy's Conclusions and Are Not Legal Conclusions......................................... 6 C. Spencer's Testimony Comports With the Rules and Traditional Role of a Rebuttal Expert ....................................................................................................... 8 D. Taken in Context, Spencer's Statements Are Logical Conclusions........................ 9 SPENCER EMPLOYED RELIABLE METHODS IN EVALUATING AND REBUTTING LEVY'S OPINIONS ON SAMPLING AND STATISTICS ...................... 9 A. Applicable Law ....................................................................................................... 9 B. Precision Range..................................................................................................... 10 C. Sampling Techniques ............................................................................................ 11 1. Sampling with replacement and stratified sampling ................................. 11 2. Student's t-distribution.............................................................................. 12 3. Skewness ................................................................................................... 13 D. Sample Size........................................................................................................... 14 E. Failure to Maintain an Audit Trail ........................................................................ 15 F. Evaluation of Whether the Sample Was Discrepant ............................................. 18 G. Measurement Error ............................................................................................... 19 SPENCER'S OPINIONS SHOULD NOT BE EXCLUDED UNDER RULE 403.......... 19 CONCLUSION ................................................................................................................. 22

VI.

VII. VIII.

-i-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
HUI-131254v1

TABLE OF AUTHORITIES Page(s) Cases Boyd v. City & County of San Francisco, 576 F.3d 938 (9th Cir. 2009)...................................................................................................... 21 Daubert v. Merrell Dow Pharms, Inc., 509 U.S. 579 (1993)......................................................................................................... 3, 4, 6, 8 Hemmings v. Tidyman's Inc., 285 F.3d 1174 (9th Cir. 2002).............................................................................................. 10, 20 Kennedy v. Collagen Corp., 161 F.3d 1226 (9th Cir. 1998)...................................................................................................... 4 Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999)................................................................................................................. 3, 6 Nationwide Transport Fin. v. Cass Info. Sys., Inc., 523 F.3d 1051 (9th Cir. 2008)...................................................................................................... 8 Obrey v. Johnson, 400 F.3d 691 (9th Cir. 2005)...................................................................................................... 10 QR Spex, Inc. v. Motorola, Inc., No. CV-03-6284-JFW, 2004 WL 5642907 (C.D. Cal. Oct. 28, 2004)........................................ 8 Robinson v. G.D. Searle & Co., 286 F. Supp. 2d 1216 (N.D. Cal. 2003) ....................................................................................... 8 Sullivan v. United States Dep't of Navy, 365 F.3d 827 (9th Cir. 2004)...................................................................................................... 10 United Nat'l Maint., Inc. v. San Diego Convention, No. 07-cv-2172, 2010 U.S. Dist. LEXIS 79541 (S.D. Cal. August 3, 2010) .............................. 6 United States v. Dennis, 625 F.2d 782 (8th Cir. 1980)...................................................................................................... 20 White v. Ford Motor Co., 500 F.3d 963 (9th Cir. 2007)...................................................................................................... 21 Other Authorities Mathematical Statistics and Data Analysis, 3rd Edition ............................................................... 18 Rules Fed. R. Civ. P. 26 ........................................................................................................................... 20 Fed. R. Evid. 403 ..................................................................................................... 1, 19, 20, 21, 22 Fed. R. Evid. 702 ..................................................................................................................... 1, 3, 4

- ii -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

I.

INTRODUCTION Plaintiffs seek to wholly exclude the testimony of Defendants' expert statistician, Dr.

Bruce Spencer, who has over 30 years of experience in statistics and is the primary rebuttal expert to Dr. Daniel Levy. Having alleged copyright infringement by TomorrowNow, Plaintiffs hired Levy, an economist claiming to be a statistician, to try to support those allegations by having Levy count certain files located at TomorrowNow and offer purported analyses on particular technical actions taken by TomorrowNow.1 In his report, Levy presents two types of counts: (1) sums where the total value is allegedly known, and (2) estimates based on sampling where the total value is not known. With regard to (2), Levy made certain choices about how he designed and selected the sample, as well as how he calculated his totals from the sample. These choices show a general lack of familiarity with sampling techniques, raise questions as to the sample itself, and impact some of Levy's numerical conclusions. Those criticisms (and others) are the subject of Spencer's 44-page rebuttal expert report and are simply attacks on Levy's credibility and the weight to be given to Levy's estimated counts. Plaintiffs do not, and cannot credibly, dispute that Spencer is well qualified to provide expert testimony rebutting Levy's statistical analysis. Instead, Plaintiffs attempt to exclude Spencer because they disagree with him and argue that he did not fully understand the legal significance of his criticisms of Levy's analysis and that little, if any, weight should be given to his criticisms. Neither of those assertions is grounds for exclusion. The fact that this is really a dispute over credibility and weight is shown by a few examples from Plaintiffs' motion: · Plaintiffs claim that Spencer accused Levy of lying when in fact, Spencer testified, "I've never stated that he was lying." See Declaration of Scott W. Cowan iso Defs.' Opp. to Pls.' Mot. to Exclude Expert Testimony of Bruce Spencer ("Cowan Decl.") ¶ 2, Ex. A (Spencer Tr.) at 123:20. As explained in Defendants' Motion to Partially Exclude Expert Testimony of Kevin Mandia and Daniel Levy (D.I. 780), Levy offers certain opinions that are: (1) beyond his expertise; (2) unsupported, or supported only by unverified conclusions from Plaintiffs' counsel, employees, and other hired experts; and (3) impermissible subjects of expert opinion under Rules 403 and 702. These opinions of Mandia and Levy are not discussed here.
HUI-131254v1 1

-1-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

·

Plaintiffs accuse Spencer of "eyeballing" calculations when in fact he said, "I wouldn't say eyeballing i[t]. An example would be with the correlation tables that we recently turned over, we have the RMS correlation size. And if the RMS correlation size is .6, then the square of that is about .36, which means you'd get an effective reduction in sampling variance of about 35 percent." See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 223:10-16. · Plaintiffs accuse Spencer of testifying that Plaintiffs should be held to a higher burden of proof when his testimony shows he espouses no such opinion. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 95:6-24 (stating he had no information on the burden of proof and would not base an opinion about the burden of proof on his own judgment).

Regardless, three things are clear. Spencer is qualified. Spencer's extensive statistical background, reliance on statistical principles, and rebuttal opinions are relevant to assist the jury in seeing the flaws in Levy's statistical analysis. And Spencer followed reliable methods in performing the analysis that resulted in his rebuttal expert opinions. II. SUMMARY OF SPENCER'S REBUTTAL OPINIONS AND REPORT Defendants engaged Spencer to review and evaluate the expert reports of Levy and Mandia,2 to the extent that Mandia's report related to Levy's work. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 3.1. Levy attempted to design a "statistically valid sample" of data collected and provided to him by Mandia, select samples from this data based on his design specification, and correctly formulate statistics to calculate and report. See Cowan Decl. ¶ 4, Ex. C (Levy Tr.) at 40:25-41:16. Spencer opines that Levy's "application of standard statistical theory was questionable in some cases and simply wrong in others. He made serious mistakes in selection of samples, he did not adequately document how he selected his most important samples, and he made serious mistakes in choosing which `standard statistical formulas' to use." See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 5.4. Mandia is a forensic scientist that Plaintiffs hired in an attempt to support their liability allegations. See D.I. 780 (Defs.' Mot. to Partially Exclude Expert Testimony of Kevin Mandia and Daniel Levy).
HUI-131254v1 2

-2-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Specifically, Spencer points out that Levy: (1) chose a sample design (simple random sampling with replacement) instead of other alternative designs, like simple random sampling without replacement or stratified simple random sampling, which would have yielded more accuracy for the same sample size; (2) failed to adequately document how he selected his samples; (3) selected one sample (his Critical Support sample) that was extreme compared to more than the great majority of possible simple random samples that he could have selected; (4) used the wrong estimator for his sample; (5) used the wrong formula for estimating standard errors for his estimator; (6) offered the wrong interpretation of confidence intervals; (7) failed to check whether the assumptions underlying his use of the normal approximation were valid; (8) used a mathematical distribution to form his confidence intervals that would be wrong even if the assumptions underlying his use of the normal approximation were valid; and (9) incorrectly generated his samples when he computed confidence intervals by "sampling repeatedly from the data to estimate the upper and lower bounds." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 5.4. III. LEGAL STANDARD Rule 702 permits experts qualified by "knowledge, experience, skill, expertise, training, or education" to testify "in the form of an opinion or otherwise" based on "scientific, technical, or other specialized knowledge" if that knowledge will "assist the trier of fact to understand the evidence or to determine a fact in issue." See Fed. R. Evid. 702. The Court serves as the "gatekeeper" in excluding expert testimony that fails to clear the threshold hurdles of relevance and reliability. Daubert v. Merrell Dow Pharms, Inc., 509 U.S. 579, 589 (1993); see also Kumho Tire Co. v. Carmichael, 526 U.S. 137, 147 (1999) (holding that the gate keeping function created by Daubert applies to evaluating technical experts). "This entails a preliminary assessment of whether the reasoning or methodology is scientifically valid and of whether that reasoning or methodology properly can be applied to the facts in issue." Daubert, 509 U.S. at 592-93. To make this determination, the Court must apply a three-part test: (1) Is the proffered expert qualified to testify in the area on which he is opining based on his knowledge, skill, experience, training, or education (qualification requirement)?; (2) Is the proffered expert testimony based on reliable scientific or specialized knowledge that is reliably applied to the facts
HUI-131254v1

-3-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

of this case (reliability requirement)?; and (3) Will the proffered expert testimony assist the trier of fact in understanding the evidence or determining a fact in issue (relevancy requirement)? See Fed. R. Evid. 702; Daubert, 509 U.S. at 592-93. Rule 702 is applied consistent with "the `liberal thrust' of the Federal Rules and their `general approach of relaxing the traditional barriers to `opinion testimony.'" Daubert, 509 U.S. at 588 (citations omitted); see also Fed. R. Evid. 702 advisory committee's notes (2000 Amendments) ¶ 6 (confirming that "rejection of expert testimony is the exception rather than the rule"). Opining on the flaws in another experts' methodology is a common, and admissible, form of expert testimony. See generally, e.g., Kennedy v. Collagen Corp., 161 F.3d 1226, 1230-31 (9th Cir. 1998) ("In arriving at a conclusion, the fact finder may be confronted with opposing experts, additional tests, experiments, and publications, all of which may increase or lessen the value of the expert's testimony. But their presence should not preclude the admission of the expert's testimony-they go to the weight, not the admissibility."). When the threshold for admissibility is met, differences in the experts' opinions simply go to the weight of the testimony and not the admissibility. See id. IV. SPENCER IS QUALIFIED TO REBUT LEVY Spencer is an expert qualified by "knowledge, experience, skill, expertise, training, or education" in the field of statistics and, specifically, sampling. See Fed. R. Evid. 702. Spencer has a Ph.D. in Statistics from Yale University and is currently a Professor at Northwestern University in the Department of Statistics, which he helped found in 1987 and served as the Department Chair for 17 years. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1; see also id. at Ex. B (Spencer Report, Appendix 2) at 1. He is the author of a book on the cost-benefit analysis of statistical data collections (including samples). See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1; see also id. at Appendix 2 at 3. Spencer has served on numerous advisory committees and consultancies to federal government agencies on matters related to statistical sampling, such as the Department of Justice, Department of Education, Census Bureau, Department of Energy, and the Government Accountability Office, as well as on numerous advisory committees and consultancies to high
HUI-131254v1

-4-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

level non-government groups such as the National Academy of Sciences, National Academy of Education, and National Institute of Statistical Sciences. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1; see also id. at Appendix 2 at 1-2, 12. Recently, Spencer assisted the Census Bureau in developing plans for the 2020 census and presented a talk related to a cost-benefit analysis. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1; see also id. at Appendix 2 at 13. He has also been involved in several large-scale public studies that utilized various methods and techniques of statistical sampling, including the National Educational Longitudinal Study of High School Class of 1988, the National Assessment of Educational Progress, and the National Immunization Survey. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1; see also id. at Appendix 2 at 11. And he has carried out private sector work on samples for companies including Amgen and BlueCross BlueShield Association, where he was intimately involved in developing, drawing, testing, and analyzing samples to ensure valid results. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 2.1. Therefore, Spencer is qualified to provide testimony evaluating and rebutting Levy's statistical analysis. V. SPENCER'S REBUTTAL OPINIONS WILL ASSIST THE JURY IN UNDERSTANDING AND WEIGHING LEVY'S OPINIONS ON SAMPLING AND STATISTICS. Plaintiffs argue that Spencer's testimony will not assist the jury because it does not fit the facts of the case. Specifically, Plaintiffs claim that Spencer does not understand the facts of this case or the purposes of the expert reports, including that Levy's report does not concern damages.3 See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 5-11. Plaintiffs string together a series of statements taken out of context to argue that all of Spencer's opinions are fatally flawed because Spencer was not precisely sure how Plaintiffs would use Levy's statistics as a legal matter in this case. This argument fails for at least three reasons: (1) Spencer's criticisms hold regardless of the legal purpose for which Plaintiffs will offer Levy's testimony at trial; (2) Spencer's testimony will assist the trier of fact by explaining well-known statistical points and how Levy did not follow them; and (3) Spencer's statements, when viewed in context of Levy's To the extent Plaintiffs are also asserting a reliability challenge associated with these points, for the reasons discussed in Part VI, Spencer's opinions are reliable.
HUI-131254v1 3

-5-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

report and this case, are logical conclusions. Moreover, Plaintiffs' argument, at least in part, is about the bases and the sources of Spencer's opinions, which goes to the weight, not admissibility, of Spencer's rebuttal opinions. A. Applicable Law.

An expert's testimony need only assist the trier of fact and relate to, or "fit," the underlying facts of the case. Daubert, 509 U.S. at 591. The district court must determine whether the "particular expert ha[s] sufficient specialized knowledge to assist the jurors `in deciding the particular issues in the case.'" Kumho, 526 U.S. at 156. Further, it is well established that objecting to bases and sources of an expert's opinion affect the weight assigned to that opinion, not whether it is admissible. United Nat'l Maint., Inc. v. San Diego Convention, No. 07-cv-2172, 2010 U.S. Dist. LEXIS 79541, at *11-12 (S.D. Cal. Aug. 3, 2010) (noting that as a general rule questions related to the bases and sources of an expert's opinion go to the weight of the opinion and not the admissibility). B. Spencer's Rebuttal Opinions Are Valid and Relevant Statistical Criticisms of Levy's Conclusions and Are Not Legal Conclusions.

Assuming arguendo that Spencer did not know the way in which Plaintiffs planned to use Levy's numbers at trial, then as a matter of law, Plaintiffs' argument still fails. Spencer's critique and evaluation of Levy's opinions are criticisms of Levy's sampling techniques as well as his statistical analysis and calculations, which hold regardless of the legal purpose. In other words, Spencer compares Levy's opinions and work to general statistical principles that apply regardless of how the numbers are used or characterized by Plaintiffs at trial. An example of how Plaintiffs' argument misses the mark is evident in the discussion of Spencer's statements on cost-benefit analysis. Plaintiffs conflate Spencer's cost-benefit analysis, the statistical concept of precision, and the legal concept of "burden of proof" to argue that Spencer seeks to "improperly raise Oracle's burden of proof." D.I. 773 (Pls.' Mot. to Exclude Spencer) at 9. Nowhere in Spencer's report or testimony does he seek to raise Plaintiffs' legal burden of proof. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 95:6-24 (stating he had no information on the burden of proof and would not base an opinion about the burden of proof on
HUI-131254v1

-6-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

his own judgment). Rather, Spencer opines on how Levy does not justify the appropriateness for sampling, as all sampling introduces sampling error--a point Levy admitted at his deposition. See Cowan Decl. ¶ 4, Ex. C (Levy Tr.) at 184:22-185:1 (Q: "Is it fair to say that the percentages derived from the selected samples are estimates based on a sample which has sampling error in your report? A. That's correct."). One way to justify introducing sampling error is by performing a cost-benefit analysis (not done by Levy), which would include looking at the amount of money Plaintiffs seek in this case compared to the cost of collecting all of the data. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 1.4. Separately, Spencer criticizes Levy regarding the statistical concept of precision. Spencer opines that the use of a 50% precision range is "quite a large range of uncertainty to deliberately aim for in a case where Plaintiffs allegations place large sums of money at stake, and as shown in paragraph 6.49 below . . . was not often achieved." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.6. These are two separate criticisms: (1) Levy did not perform a cost-benefit analysis to justify why introducing sampling error is appropriate; and (2) the precision range chosen by Plaintiffs is very large and was not always achieved. Plaintiffs fail to argue how these points, based on Levy's work, are not relevant to evaluating, rebutting, and weighing his opinions. Interestingly, Plaintiffs' entire argument arises out of Plaintiffs' own attempt to equate statistical calculations with burden of proof. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 1011 ("More importantly, Levy's 90% confidence interval (or even somewhat wider confidence interval if one assumes all of Spencer's criticisms could be right) is far above the more-likelythan-not 50.1% required by the `preponderance of the evidence' standard."). This is exactly why Defendants need Spencer's testimony because a confidence interval can be created to make misleading statements about any percentage. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 97:2122 ("I can always come up with a 90 percent confidence interval if I have any data."), 99:13-17 ("A 90 percent confidence interval is a random interval constructed so that with 90 percent probability, or for 90 percent of the possible samples, your interval will include the value that you're interested in."), 116:2-20.4 The burden of proof is not a math equation that can be proved "Q. Well, what does a 51 percent confidence interval mean? A. Well, it means that you've constructed it--it means to me that you've constructed it in order that it have 51 percent
HUI-131254v1 4

-7-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

by statistics--a point that Defendants intend to challenge in the traditional manner through crossexamination of Levy and rebuttal testimony from Spencer. The Court will provide the legal instructions to the jury on burden of proof. See Daubert, 509 U.S. at 596 (explaining that crossexamination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of addressing expert opinions).5 C. Spencer's Testimony Comports with the Rules and Traditional Role of a Rebuttal Expert.

Spencer's report and testimony show that he understands the general principles of statistics and how to conduct proper statistical analysis. By pointing out where Levy fails to follow these general principles, Spencer is assisting the jury in evaluating Levy's testimony and enabling them to properly weigh Levy's opinions. To be clear, this is simply not a case where the rebuttal testimony is unrelated to the direct testimony and so the opinions are not tied to the applicable facts. Spencer reviewed Levy's report and the underlying data associated with this report and directly addressed the basis and methods for Levy's opinions.6 See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶¶ 4.1-4.2, see also id. at Appendix 1. As noted above in Section II, Spencer's opinions show that Levy committed errors in his attempt to arrive at his estimates and that there is reason to doubt Levy's conclusions. Spencer

(continued...)

probability of covering the population value under repeated sampling. That's what it means to me. But that you intend it to be so and that it is so are two different things. Q. So if you properly do the sampling ­ A. Right. Q. ­and analyze it properly ­ A. Right. Right. Q. ­ then you've constructed an analysis that will, on repeated samples, have the value of interest within the bounds, within the bounds of the range 51 percent of the time ­ at least 51 percent of the time. Right? A. No." This case is not like the cases cited by Plaintiffs. For example, in Nationwide, the expert was actually seeking to provide testimony instructing the jury on the legal issues, and, thus, the court of appeals upheld the district court's decision to exclude this testimony. Nationwide Transport Fin. v. Cass Info. Sys., Inc., 523 F.3d 1051, 1055-60 (9th Cir. 2008). 6 This case is unlike the cases cited by Plaintiffs in which the expert did not actually review the underlying data. See, e.g., QR Spex, Inc. v. Motorola, Inc., No. CV-03-6284-JFW, 2004 WL 5642907, at *9 (C.D. Cal. Oct. 28, 2004). Likewise, this case is inapposite to Robinson, in which the plaintiff's expert based his conclusion on a fact that was directly refuted by the plaintiff's admission. See Robinson v. G.D. Searle & Co., 286 F. Supp. 2d 1216, 1221 (N.D. Cal. 2003).
HUI-131254v1 5

-8-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

will also assist the jury to better understand a number of general statistical principles asserted by Levy and to put these opinions in their proper context. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 92:15-20 (noting that in statistics, a 25% error in what was intended to be measured tends to be a large error), 99:13-17 (describing what a confidence interval represents), 121:1-21 (describing when certain sampling methods might be appropriate under sampling theory), 199:2-202:5 (explaining the relationship between confidence intervals and skewness). This is all proper rebuttal testimony that will assist the trier of fact. D. Taken in Context, Spencer's Statements Are Logical Conclusions.

Spencer's statements, taken in the proper context, reveal that he was giving examples and being mindful of how Plaintiffs, not just Levy, would attempt to use these numbers. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 1.4 ("depending on how Plaintiffs intend to use Levy's estimates, a modest sampling error can translate to a very large financial consequence."); see also id. ¶ 6.3 ("The consequences of sampling error could be large, depending on the factual context in which Levy's estimates will be used in this case"). Regardless, for the reasons stated above, Spencer's opinions are not dependent on Levy's testimony being related to damages. VI. SPENCER EMPLOYED RELIABLE METHODS IN EVALUATING AND REBUTTING LEVY'S OPINIONS ON SAMPLING AND STATISTICS Plaintiffs also allege that some of Spencer's opinions are unreliable, but their challenges actually go to the weight of the testimony and not the admissibility. Plaintiffs challenge Spencer's opinions related to: (1) precision ranges; (2) sampling techniques; (3) sample sizes; (4) failing to maintain an audit trial; (5) evaluating whether the sample was discrepant; and (6) potential measurement error. D.I. 773 (Pls.' Mot. to Exclude Spencer) at 11-22. Plaintiffs do not argue that Spencer has invented some new, untested form of statistical analysis; rather, they simply disagree with Spencer's evaluation of Levy's attempted application of statistical theories in an effort to keep the jury from hearing the full story about the ramifications of the choices made by Levy and whether his conclusions are credible. A. Applicable Law.

It is proper for a statistical expert to rely on generally and widely accepted statistical
HUI-131254v1

-9-

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

methodologies. See, e.g., Hemmings v. Tidyman's Inc., 285 F.3d 1174, 1183-89 (9th Cir. 2002). The Ninth Circuit does not require an expert to conduct independent research. Experts are allowed to testify on matters where they relied upon textbooks and other learned treatises. See, e.g., Sullivan v. United States Dep't of Navy, 365 F.3d 827, 833-35 (9th Cir. 2004) (reversing and remanding a case to the district court because the district court did not permit the expert to rely on textbooks stating generally accepted principles that supported his opinion). Moreover, the Ninth Circuit has explained that objections to statistical evidence generally go to the weight and not the admissibility of the opinion. See Obrey v. Johnson, 400 F.3d 691, 695-97 (9th Cir. 2005). B. Precision Range.

Levy defines "precision range" in his report through the use of this example: "In order to design the sample size, she decides to use a 90% confidence level and a 20% precision range; that is she wants to be able to say that if she sampled from this population repeatedly, 90% of the true number of computers would be within plus or minus 10% of her result." See Cowan Decl. ¶ 5, Ex. D (Levy Report) at 12. Levy goes on to discuss the use of a 50% precision range for this case, which means that if sampled from the population repeatedly, the population number would only be within plus or minus 25% of his result. See id. at 16.7 In rebuttal, Spencer points out that in statistics, a small precision range is desirable as compared to a larger precision range because the larger the precision range, the larger the range of uncertainty. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶¶ 6.5-6.6. Additionally, Spencer calculates examples where Levy fails to even achieve a 50% precision range and shows that sometimes the range is as high as 313%. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.49 and Table 5. Plaintiffs' criticism of Spencer is based entirely on Plaintiffs' claim that "Spencer mistakenly thought Levy's statistical analyses were being used for damages calculations . . . ." D.I. 773 (Pls.' Mot. to Exclude Spencer) at 12. Spencer, however, did not opine that, absent a In their motion, Plaintiffs state in a footnote that this precision range was not sought as to every measure. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at n.9. Nowhere else in Levy's report, or his deposition, does he suggest that he aimed for a precision range other than 50% in his estimated counts. If Levy was seeking to achieve a different precision range for different estimated counts, then this should have been clearly articulated as it goes to his methodology. Regardless, Spencer's evaluation of the only given precision range is instructive, based on reliable methods, and attacks the credibility of Levy's numerical counts.
HUI-131254v1 7

- 10 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

large damages claim, the level of uncertainty associated with a 50% precision range would change. The level of uncertainty inherent with a 50% precision range is a fact and nothing Plaintiffs cite or claim shows otherwise. The jury should hear this information to assist them in deciding what weight they should give to Levy's estimates. C. Sampling Techniques.

Levy chose to: (1) use sampling with replacement, (2) use the normal distribution instead of the student's t-distribution, and (3) not consider skewness prior to designing and selecting his sample. Although Plaintiffs now seek to shroud these decisions under a veil of secrecy to which only their expert is allowed to peek beyond, Levy's choices are subject to reasonable disagreement and debate. Moreover, to reliably explain the relevant, technical ramifications associated with these choices requires an expert statistician such as Spencer. 1. Sampling with replacement and stratified sampling.

Plaintiffs argue that Spencer's opinions related to sampling with replacement should be excluded. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 13-14. As explained by Spencer, based upon statistical texts to which Plaintiffs lodge no objection, a sample drawn where the same item can be selected multiple times (with replacement) can result in a higher level of sampling error than other known methods, including sampling without replacement. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.2 n.8; see also Cowan Decl. ¶¶ 7, 8, Ex. F (Cochran) at 30; Ex. G (Sarndal) at 73. In other words, sampling without replacement would increase the level of precision associated with the estimates based on the sample. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.2 n.8. Additionally, Spencer points out that Levy could have used stratified sampling, which would have yielded more accuracy for the same sample size. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶¶ 5.4, 7.3. That is, the estimate would be closer to the population value because the variance would be lower and the precision would be higher. Plaintiffs attempt to challenge this point by arguing that it is based on some type of unreliable, "implicit calculation." D.I. 773 (Pls.' Mot. to Exclude Spencer) at 19-20. Plaintiffs, however, do not challenge the basis for Spencer's opinion, which is a well-known and established statistical text. See Cowan Decl.
HUI-131254v1

- 11 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

¶ 3, Ex. B (Spencer Report) ¶ 7.3 ("`[I]f intelligently used, stratification nearly always results in a smaller variance for the estimated mean or total than is given by a comparable simple random sample.'") (quoting Cochran at 99); see also Cowan Decl. ¶ 8, Ex. F (Cochran) at 99. Moreover, Spencer explained how to calculate the amount by which it will lower the variance at his deposition. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 223:9-224:11. Plaintiffs improperly argue that this is an "implicit calculation." However, as explained by Spencer, it is a simple math calculation derived from materials produced by Spencer. See id. Regardless, it is simply a disagreement that should go to the weight of Spencer's testimony, rather than its admissibility. Spencer's criticism is well-founded, based on known statistical texts and tested methods. This is exactly how experts evaluate other experts' opinions. The fact that Plaintiffs disagree and argue that Levy's method was proper under the circumstances of this case highlights that Plaintiff's motion raises nothing more than the typical situation of dueling expert opinions. 2. Student's t-distribution.

Estimated values based on sampling are generally reported using a confidence interval. Levy chose a 90% confidence interval. There is no dispute that "[i]n plain terms, a 90% confidence interval is a range of numbers calculated from the sample that have the property that the population value being estimated will fall within the range 90% of the time if the sample is repeated independently a very large number of times." See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.26. While Spencer points out several issues with Levy's construction of confidence intervals in this case,8 one of these issues is that Levy uses an incorrect value in his calculations. Namely, Levy uses the value associated with the normal distribution, which is appropriate where the standard error is actually known. Levy does not dispute (nor can he) that the standard error is not actually known here and thus the statistically proper method is to use the value associated with the Student's t-distribution. See Cowan Decl. ¶ 16, Ex. O (Snedecor & Cochran) at 60. Plaintiffs' criticism is that the use of the value associated with the Student's t-distribution does not make a large numerical impact on this case, and, therefore, Spencer should be prohibited from stating it. And Plaintiffs cite to Spencer's report suggesting that Spencer admits this change
8

See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶¶ 6.27-6.35. - 12 DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

HUI-131254v1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

is insignificant. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 20-21. Plaintiffs take this opinion out of context in an effort to make it seem nonsensical. In the section referenced by Plaintiffs, what Spencer does is design a test to evaluate whether Levy's claimed confidence intervals actually reach the 90% threshold Levy was seeking.9 In part, the test results show that "for the Retrofit sample, not a single coverage rate reached 90%." See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.34 and Tables 1-2. Spencer opines that using the "Student's t-distribution, which is the standard approach gives a slight improvement on the method that Dr. Levy employed [as noted by Plaintiffs] . . . However, using the Student's t-distribution does not solve the larger problem, which is that for some measures his confidence intervals can have coverage rates that are much poorer than what Dr. Levy claims." See Cowan Decl. ¶ 3, Ex. B (Spencer Report) at ¶ 6.34. Taken in context, Spencer's criticism related to the Student's t-distribution is a logical building block used to attack the credibility of Levy's numerical conclusions and is based on sound statistical principles. The actual impact of making the change Spencer opines should be made to Levy's methodology goes to the weight, not admissibility, of that opinion. 3. Skewness.

Levy uses the normal distribution in constructing his confidence intervals. Again, based on well-known and accepted statistical texts, Spencer opines that using either the normal distribution or the t-distribution, however, can be a bad assumption if it is not appropriate for the population being sampled. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.32. One way to test the assumption is to look at skewness.10 See id. Spencer examines the skewness associated with the items being measured by Levy. Id. Spencer concludes that, for some measures, the skewness was large and "the skewness coefficients from the sample did not fully reveal the extent of the skewness in the population. . . ." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.32. Spencer expounded on this at his deposition, testifying that: "There's an established mathematical relationship between coverage and skewness." See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 200:16-17. Further, Spencer indicated: "We calculate the confidence interval one measure at a See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.30. 10 "Skewness is the asymmetry in relation to a frequency distribution or a measure of that asymmetry." See Cowan Decl. ¶ 6, Ex. E (Statistics Dictionary) at 376.
HUI-131254v1 9

- 13 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

time. Finding that skewness is a problem for some measures where we can test it gives us insight that it's also a problem for measures where we can't directly test the coverage of the confidence intervals." See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 200:8-13. In response to Spencer's criticism, Levy attempted to perform his own skewness analysis and submitted supplementary calculations.11 Plaintiffs contend that because Levy has a competing view of what the skewness coefficients mean, Spencer's opinions are somehow unreliable. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 21-22. This disagreement, however, is not over the reliability of the calculations, but rather how competing experts interpret them. Likewise, Plaintiffs raise a vague argument regarding Spencer's evaluation of Levy's "underestimation." D.I. 773 (Pls.' Mot. to Exclude Spencer) at 20. Based on the cited testimony, Plaintiffs appear to challenge Spencer's evaluation of the rebuttal skewness information provided by Levy. Spencer testified that Levy's chart may underestimate the skewness in the sampled measures. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 213:24-214:3 ("Well, we can see that for some of the population measures, when you calculate the skewness for the sample, the skewness for the sample considerably underestimates the skewness in the population."). He notes that he could make this determination simply by reviewing the tables presented at his deposition. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 214:19-215:16 (explaining how it was just a matter of looking at a table). Plaintiffs then make the conclusory and unfounded statement that "[i]t is impossible for Oracle to know just how many of Spencer's criticisms and conclusions are based on his `implicit calculations' about `such-and-such' . . . ." D.I. 773 (Pls.' Mot. to Exclude Spencer) at 20. This broad generalization is belied by the detailed and reasoned rebuttal opinions of Spencer contained in, and discussed throughout, his report. D. Sample Size.

Plaintiffs essentially argue that Spencer's rebuttal opinion that a cost-benefit analysis should have been done to determine the proper sample size is inappropriate because Spencer did not perform a cost-benefit analysis and because he references the amount of damages Plaintiffs On April 28, 2010, Plaintiffs produced sur-rebuttal material including skewness calculations. See Cowan Decl. ¶ 9, Ex. H (Sherrod Letter).
HUI-131254v1 11

- 14 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

seek. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 14-15. Spencer is simply pointing out facts: by using sampling, Plaintiffs are introducing sampling error, and "Dr. Levy's own estimates show quite large levels of sampling error, sometimes well exceeding 50% of the estimated total in the population." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.3. Spencer's rebuttal opinion, drawn from these facts and his substantial knowledge and experience regarding sampling, is that a more appropriate method to perform the statistical analysis would be to begin with a cost-benefit analysis to determine whether the high level of sampling error Levy experienced was justified by the need to sample. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.3. Spencer goes on to note that "[f]or example, when compared with the significant damages Plaintiffs seek in this case, the cost savings from sampling may be modest in comparison." Id. This would be one way to have conducted the cost-benefit analysis. The criticism is that Levy did not do one at all. Although Plaintiffs offer statements and arguments about how increasing the sample size would be unreasonable given the facts and circumstances of this case, this again is a factual disagreement that goes to the weight of each expert's respective testimony. Further, Plaintiffs' argument that Spencer should be precluded from offering this criticism because Spencer did not conduct a cost-benefit analysis ignores the statistical point that Plaintiffs are introducing the error by sampling, as well as the legal point (not argued by Spencer, but relevant to this response) that Plaintiffs bear the burden of proof at trial. The failure of Plaintiffs' expert to follow proper procedure is not negated simply because Defendants' expert pointed out this failure and did not go to the next step to correct it. E. Failure to Maintain an Audit Trail.

Plaintiffs argue that Spencer failed to employ reliable principles and methods in his critique of Levy's use of Microsoft Excel to generate numbers as part of his sampling procedure. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 16-17. There is no dispute that Levy used Microsoft Excel's functionality as part of his sampling procedure, and that, at a high level, the way the process works is that when the equation is entered into Microsoft Excel, a number is generated based on the program's internal calculations. What the program does not allow is for
HUI-131254v1

- 15 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

someone else to re-create the exact same numerical sequence to check or audit the process. See Cowan Decl. ¶ 10, Ex. I (McCullogh) at 4590. There are other programs that have this functionality and they use what is known as a "seed" value. Levy in fact used such a program for some of his other calculations. Specifically, Levy used the SAS program for his bootstrap analysis. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.38 n.63. Because Levy used such a program for these other calculations, Spencer was able to audit Levy's process and find several flaws in Levy's process. For example, Spencer determined that Levy's 10,000 bootstrap samples were not independently selected. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.38. Further, Spencer determined that Levy's samples often have too few selections. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.43. The importance of using the seed value, therefore, is already apparent from this case. Based on Levy's choice to use Excel with its known issues, Spencer opines: "In my more than 25 years of statistical consulting experience in drawing samples, I have consistently observed the generally accepted practice of fully documenting the sample selection process. That documentation includes provision of sufficient information such that the sample selection process can be duplicated exactly by anyone else." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.9. Spencer further states: "When a sample is selected by a qualified statistician for use in litigation or in other important situations, it is standard practice for the statistician to select the sample in such a way that the sample selection can be reproduced." Id. Spencer makes these statements based on the application of his experience, knowledge, and training to the facts of this case. Further, Spencer's opinions are consistent with statistical literature regarding the use of Microsoft Excel as a random number generator. See Cowan Decl. ¶¶ 10-12, Ex. I (McCullogh) at 4590 ("As Microsoft does not permit a seed to be set for the Excel [random number generator], there can be no audit trail for Excel [random numbers]"); Ex. J (McCullogh & Heiser) at 4571 ("The Excel random number generator does not fulfill the basic requirements for a random number generator to be used for scientific purposes"); Ex. K (McCullogh & Wilson) at 1249-50 (noting Excel's RAND generator failed empirical tests). Plaintiffs now offer three sur-rebuttal points to Spencer's criticism: (1) No seed value was
HUI-131254v1

- 16 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

used in the Vietnam era random selection for military service; (2) Spencer did not know the details of how Vandaele drew some of the samples that are referenced in Spencer's report; and (3) Spencer accuses Levy of lying. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 16-17. At a minimum, the fact that Plaintiffs used their brief to supply new sur-rebuttal points by itself shows how this is simply a weight, not admissibility argument. Nevertheless, Defendants also have substantive responses to these challenges. With regard to Levy's entirely new opinion and reliance on the sampling methods used in the Vietnam era,12 the literature actually supports Spencer's position as the Vietnam draft process showed the flaws of sampling without using a seed. See Cowan Decl. ¶¶ 13, 14, Ex. L (Fienberg) at 259-60 (providing an extensive discussion on the lack of randomness in the 1970 Vietnam draft lottery procedure, the public criticism of that procedure, and the resulting modification of the procedures for the 1971 draft lottery); Ex. M (Rosenblatt) at 306-308 (describing the detailed process for modifying the procedures and creating a random draft lottery and explaining that it was more than selecting random dates from a bowl). With regard to the reference to Vandaele, he is another of Defendants' experts who has a specialized expertise in using the SAS program. Vandaele performed a series of tests and actions at Spencer's direction, and his report and opinions are not being challenged by Plaintiffs. Given this, the mere fact that Spencer did not recall how Vandaele chose the SAS seed is unpersuasive in establishing a claim about the reliability of Spencer's opinions. Moreover, Vandaele issued his own expert report, and Plaintiffs deposed him on that report. See Cowan Decl. ¶ 17. Finally, contrary to Plaintiffs' assertions, Spencer never suggested that Levy lied. In fact Plaintiffs' counsel questioned Spencer about Levy's testimony relating to using Excel to generate random numbers. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 123:16-20 (Q: "And do you have any evidence that he was lying when he so testified?" A: "I have no evidence that he was lying, and I've never stated that he was lying."). Spencer's critique of Levy's opinions is a very clinical, scientific evaluation of Levy's work and not a personal attack. Defendants have separately objected to Levy's new opinions. See Defs.' Objs. to the Declarations of Daniel Levy Filed in Support of Pls.' Mots. to Exclude.
HUI-131254v1 12

- 17 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

F.

Evaluation of Whether the Sample Was Discrepant.

Plaintiffs claim that one of the methods Spencer uses to evaluate Levy's sample was not reliable. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 18-19. To evaluate Levy's sample, Spencer examined how likely it was that "a sample such as Dr. Levy's would be selected with the procedure he claims to have used." See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.10. To perform this analysis, Spencer used standard statistical calculations, the z statistic, his extensive background in analyzing statistical methods, a relevant statistical text, and common computer software. See id. ¶¶ 6.12-6.14. He explains this method in detail: "In order to put the differences between the sample average with the population average on the same scale for each measure, I divided each difference by the standard error for the sample average. The resulting statistics are sometimes called standard scores, z-score, or z statistics. My calculation is a standard practice in the field of statistics. The z statistic is interpreted as a measure of the actual discrepancy between a given sample average and the population average." Id. ¶ 6.12. Spencer explained that if Levy's sample was selected as Levy claimed, "the z-statistic for a single sample would have a 90% chance of falling between -1.644853627 and +1.644853627," and he provided support for this method by citing to the statistical text Mathematical Statistics and Data Analysis, 3rd Edition. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.13 n.24. If one of the z statistics fell out of that range it would be "discrepant." Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.13. To execute this test, Spencer had a simple random sample selected based on the method described by Levy and had a simulation run on the sample using common statistical analysis software. See id. ¶ 6.14. He found that for one of the samples (the Critical Support sample), the chance of getting as many discrepant z-statistics as Levy did was less than 10%. See id. There is nothing ad hoc about this test, and all of this material was produced to Plaintiffs. Moreover, while Plaintiffs point to certain excerpts of Spencer's testimony related to this issue, the quoted testimony shows that this test is but one part of Spencer's extensive evaluation of Levy's estimated counts. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 19. Additionally, Plaintiffs now come forward with new opinions from Levy proposing a

HUI-131254v1

- 18 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

previously undisclosed "hypothesis test" as sur-rebuttal to Spencer's opinions.13 Spencer specifically testified that he did not conduct a hypothesis test. See Cowan Decl. ¶ 2, Ex. A (Spencer Tr.) at 152:18-153:9 (testifying that he did not set up a formal or informal hypothesis test). It is unclear, therefore, how this is an attack on the admissibility of Spencer's opinions because it does not show that the test Spencer actually performed is unreliable. G. Measurement Error.

Plaintiffs argue that Spencer's statement that Levy assumes no measurement error in the data received from Mandia is somehow entirely speculative. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 22. Spencer's report, however, explains that conclusion in great detail and he relies upon established statistical texts. See Cowan Decl. ¶¶ 3, 15, Ex. B (Spencer Report) ¶ 6.21 nn. 34-35; Ex. N (Levy & Lemeshow) at 35 ("If we assume that there is no measurement error in the survey, then the reliability of an estimator can be stated in terms of its sampling variance or, equivalently, its standard error.") and ("Again, if we assume that there is no measurement error, the validity of an estimator can be evaluated by examining the bias of the estimator."); see also Cowan Decl. ¶ 7, Ex. F (Cochran) at 12 ("Even with estimators that are unbiased in probability sampling, errors of measurement and nonresponse may produce biases in the numbers that we are able to compute from the data."). The fact that Spencer does not know the actual measurement error here is irrelevant. Spencer's relevant, reliable, and valid critique, based on statistics texts, is that Levy took no steps to find out what the measurement error was and failed to acknowledge that the assumption of no measurement error is a key assumption underlying his numerical counts. The jury may conclude that he should have considered this error in his counts and afford less weight to his estimates. VII. SPENCER'S OPINIONS SHOULD NOT BE EXCLUDED UNDER RULE 403 As a catchall, Plaintiffs throw in several Rule 403 challenges throughout their motion. Plaintiffs summarily add Rule 403 to their arguments regarding: (1) burden of proof; (2) precision ranges; (3) sample sizes; (4) use of Excel; and (5) measurement error. See D.I. 773 (Pls.' Mot. to Defendants have separately objected to Levy's new opinions. See Defs.' Objs. to the Declarations of Daniel Levy Filed in Support of Pls.' Mots. to Exclude.
HUI-131254v1 13

- 19 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Exclude Spencer) at 7, 9, 13-15, 17-18, 22. Plaintiffs also make two additional Rule 403 challenges at the end of the motion, arguing that: (1) Spencer should not be allowed to state that Levy used an incorrect formula in his initial and first supplemental reports, as well as making other statistical errors in all of his reports, and (2) Spencer should not be able to opine on some issues related to variance. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 22-23.14 In applying Rule 403, district courts enjoy wide latitude. Hemmings, 285 F.3d at 1184. In Hemmings, the Ninth Circuit affirmed a district court's decision to admit expert statistical testimony over a 403 challenge. See id. at 1187. The court noted that the expert's analysis was based on a consistent application of statistical analysis principles. See id. at 1186. Moreover, "[i]n weighing the probative value of evidence against the considerations enumerated in Rule 403, the general rule is that the balance should be struck in favor of admission." United States v. Dennis, 625 F.2d 782, 797 (8th Cir. 1980). As in Hemmings, all of Spencer's opinions, including those on precision ranges, sample sizes, and measurement error are probative evidence in this case and are based on reliable methods. Further, Spencer does not opine on the burden of proof in this case or make any legal arguments regarding what Plaintiffs must prove at trial. There is nothing unfairly prejudicial, or misleading, in allowing Spencer to opine on his evaluation of Levy's counts, including the choices Levy made and the consequences of those choices in arriving at these counts. Plaintiffs failed to provide any specific grounds that warrant shifting the balance to exclusion. Finally, as noted above, Spencer does not cast any unwarranted aspersions on Levy; he simply provides a professional statistical evaluation and rebuttal to Levy's work. Additionally, Levy did use the wrong formula in his initial report and in the first supplemental report that he issued. See Cowan Decl. ¶¶ 3, 8, Ex. B (Spencer Report) ¶ 6.24 and nn. 36, 38; Ex. G (Sarndal) at 73 (using equation 3.3.24). This is a mistake that was not corrected until February 12, 2010, almost three months after his initial report. See Cowan Decl. ¶ 3, Ex. B Although Plaintiffs refer to Federal Rule of Civil Procedure 26(E) in the heading to this section, it is unclear to which rule they refer. There is no Rule 26(E) and Plaintiffs make no reference to what rule they intend to discuss in the substantive section of their motion. Therefore, Defendants reserve the right to address any argument later raised by Plaintiffs on this point.
HUI-131254v1 14

- 20 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

(Spencer Report) ¶ 6.24 and n. 38. The fact that Levy issued a supplemental report where he corrected this mistake, however, does not preclude Spencer from commenting on this and the other errors in Levy's report. Likewise, Plaintiffs seek to exclude any reference to the fact that Levy admittedly made an error in designing a sampling method that never selected one of the items, referred to in the motion as "corrected some computer code." See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 23; see also Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶ 6.43. Regardless of whether Plaintiffs and Levy want to call this and the other errors in his report "corrections," "mistakes," or "errors," they go directly to Levy's credibility, and it requires an expert such as Spencer to know the correct formulas, where they are located in the literature, and how to apply that information to the analysis of the validity and accuracy of Levy's opinions and conclusions. Even though these criticisms are prejudicial, the requirement is that there must be something "unfair" about the proffered evidence--making a mistake and having the other side's expert comment on the mistake is fair. See Boyd v. City & County of San Francisco, 576 F.3d 938, 948 (9th Cir. 2009) (pointing out that "[p]roof that evidence was prejudicial to one party is insufficient to establish that the prejudice was unfair"). Unfair prejudice means that the evidence not only has a significant impact on the case, but that its admission results in unfairness to one side because of a non-probative aspect. See White v. Ford Motor Co., 500 F.3d 963, 977 (9th Cir. 2007) (citations omitted). Because Levy's statistical errors and mistakes are probative to his credibility and the weight that should be given to his opinions, Plaintiffs cannot meet their Rule 403 burden. Finally, Plaintiffs' argument on variance is relatively confusing. See D.I. 773 (Pls.' Mot. to Exclude Spencer) at 23. The portion of Spencer's opinion that Plaintiffs appear to be discussing is Spencer's criticism that for Levy's estimated counts derived in some way from sampling, Levy should have accounted for sampling error and potential measurement error. See Cowan Decl. ¶ 3, Ex. B (Spencer Report) ¶¶ 8.1-8.2. In other words, "[t]he fact that his [Levy's] percentages are based on a sample, which is subject to sampling error, implies that he is almost surely incorrect to some extent in his claims." Id. ¶ 8.1. Spencer is also noting the potential for measurement error in Levy's counts of the entire population. See id. ¶ 8.1. To the extent Levy
HUI-131254v1

- 21 -

DEFS' OPP. TO PLS.' MOT. NO. 6 TO EXCLUDE EXPERT TESTIMONY OF BRUCE SPENCER Case No. 07-CV-1658 PJH (EDL)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

offers estimates and values as unconditional truths without accounting for such error, Spencer is making a relevant and reliable challenge based on well-known statistical properties, as discussed above. Plaintiffs appear to suggest that Spencer is also criticizing Levy's counts for sampling error where the full value is known (i.e., not sampled). Spencer is not making such a criticism; therefore, there is no basis to limit Spencer's testimony under Rule 403 related to this issue because Spencer's actual opinions (as opposed to Plaintiffs' misperception of them) are based on well-known and accepted statistical analysis principles. VIII. CONCLUSION Spencer has over 30 years of experience in the field of statistics. He has a Ph.D. from Yale University, is a tenured professor in Northwestern University's Department of Statistics, and has extensive experience in both the public and privat