Direct Marketing Association, The v. Huber
MOTION to Exclude the Testimony of Plaintiff's Expert Witnesses F. Curtis Barry, Thomas Adler, and Kevin Lane Keller by Defendant Roxy Huber. (Attachments: # 1 Exhibit A- Barry Dep., # 2 Exhibit B- Barry Report, # 3 Exhibit C- Barry Dep. Exhs., # 4 Exhibit D- Adler Report and Decl., # 5 Exhibit E-1of5- Adler Dep. Exhs, # 6 Exhibit E-2of5- Adler Dep. Exhs, # 7 Exhibit E-3of5- Adler Dep. Exhs, # 8 Exhibit E-4of5- Adler Dep. Exhs, # 9 Exhibit E-5of5- Adler Dep. Exhs, # 10 Exhibit F- Adler Dep., # 11 Exhibit G- Lichtenstein Decl. and Report, # 12 Exhibit H- Keller Dep., # 13 Exhibit I- Keller Report and Decl., # 14 Exhibit J-1of3- Keller Dep. Exhs., # 15 Exhibit J-2of3- Keller Dep. Exhs., # 16 Exhibit J-3of3- Keller Dep. Exhs.)(Snyder, Melanie)
IN THE UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF COLORADO
Civil Action No. 10-cv-01546-REB-CBS
The Direct Marketing Association,
Roxy Huber, in her capacity as Executive Director, Colorado Department of Revenue,
DEFENDANT'S MOTION TO EXCLUDE THE TESTIMONY OF PLAINTIFF'S EXPERT
WITNESSES F. CURTIS BARRY, THOMAS ADLER, AND KEVIN LANE KELLER
Defendant, Roxy Huber in her capacity as Executive Director, Colorado
Department of Revenue ("Department"), moves to exclude all testimony whether by
declaration, deposition excerpt, or live testimony of Plaintiff's expert witnesses F. Curtis
Barry, Thomas Adler, and Kevin Lane Keller pursuant to Federal Rule of Evidence 702.
In this action, Plaintiff, the Direct Marketing Association (“DMA”) raises
constitutional challenges to Colorado H.B. 1193 and implementing regulations (“the
Law”), which impose reporting requirements on retailers who do not collect sales tax.
See Defendant’s Response to Plaintiff’s Motion for Preliminary Injunction, filed
November 19, 2010 [Dkt #50, pp.1-4], for a description of the Law’s requirements. DMA
sought a preliminary injunction on the grounds that the Law violates the Commerce
Clause of the United States Constitution. In support of its motion, DMA offered the
testimony of three experts - F. Curtis Barry on the subject of the alleged costs DMA
members will incur to comply with the Law, Thomas Adler on a survey he conducted of
Colorado consumers regarding their reactions to the Law, and Kevin Lane Keller
interpreting that survey. Each of these witnesses' testimony should be excluded
pursuant to Rule 702 for failure to employ reliable methodology in forming their
RULE 702 DOES NOT PERMIT EXPERT OPINIONS WHICH ARE NOT
BASED ON RELIABLE METHODOLOGY.
Rule 702 governs the admissibility of expert witness testimony:
If scientific, technical, or other specialized knowledge will assist the trier of
fact to understand the evidence or to determine a fact in issue, a witness
qualified as an expert by knowledge, skill, experience, training, or
education, may testify thereto in the form of an opinion or otherwise, if (1)
the testimony is based upon sufficient facts or data, (2) the testimony is
the product of reliable principles and methods, and (3) the witness has
applied the principles and methods reliably to the facts of the case.
Fed.R.Evid. 702. In assessing a Rule 702 motion, the Court must determine: 1)
whether the expert is qualified to render the opinion; and 2) whether the opinion is
reliable. 103 Investors I, L.P. v. Square D. Co., 470 F.3d 985, 990 (10th Cir. 2006).
Exercising its “gatekeeper” role, the Court should exclude the opinions of all three
Plaintiff experts because they are not reliable. See Kumho Tire Co, Ltd. v. Carmichael,
526 U.S. 137, 147 (1999).
As the proponent of the expert testimony, DMA bears the burden of proving the
admissibility of the testimony under Rule 702 by a preponderance of the evidence.
United States v. Crabbe, 556 F. Supp. 2d 1217, 1220 (D. Colo. 2008). DMA, therefore,
must prove that each expert has sufficient expertise to choose and apply a
methodology, that the methodology applied was reliable, that sufficient facts and data
as required by the methodology were used, and that the methodology was otherwise
reliably applied. Id. at 1221; see also Toni's Alpacas, Inc. v. Evans, 2010 WL 3730382,
at *1 (D. Colo. Sept. 16, 2010) (the proponent of the testimony must show that “the
expert has employed a method that is scientifically sound and that the opinion is based
on facts which enable [him] to express a reasonably accurate conclusion as opposed to
conjecture or speculation.”) (internal citations omitted). An opinion is reliable if it is
based on scientific knowledge, grounded in the methods and procedures of science. Id.
(citing Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 58-90 (1993)).
Such knowledge must be more than subjective belief or unsupported speculation. Id.
The critical determination is “whether the evidence is genuinely scientific, as distinct
from being unscientific speculation offered by a genuine scientist.” Goebel v. Denver &
Rio Grande W. R.R. Co., 346 F.3d 987, 992 (10th Cir. 2003).
Under Daubert, “any step that renders the analysis unreliable … renders the
expert’s testimony inadmissible. Mitchell v. Gencorp, Inc., 165 F.3d 778, 782 (10th Cir.
1999). In the reliability assessment, the inquiry is whether the witness employed “the
same level of intellectual rigor that characterizes the practice of an expert in the relevant
field." Toni's Alpacas, 2010 WL 3730382, at *2 (quoting Goebel, 346 F.3d at 992).
Applying these principles, each of DMA's experts' testimony fails the reliability
test and his testimony should be excluded.
F. CURTIS BARRY’S OPINIONS ARE NOT RELIABLE BECAUSE HE
DID NOT PERFORM ANY STUDIES OR FOLLOW ANY ESTABLISHED
METHODOLOGY, AND HIS OPINIONS SET FORTH ONLY HIS
SUBJECTIVE BELIEFS ABOUT COMPLIANCE COSTS.
DMA offers the testimony of Mr. F. Curtis Barry to establish the costs for DMA
members to comply with each of the three regulatory requirements of the Law. Mr.
Barry, however, did not perform any studies of his own (see, e.g., Exh. A., Barry Dep.
53:25-55:8; 126:15-21; 152:18-20), 1 he did not refer to any published literature in his
field (see, e.g., id. at 55:8-11; 126:12-14; 152:21-22), and he did not even refer to any
documents relating to his own experience with clients to establish his opinions. See,
e.g., id. at 52:19-53:14; 128:25-129:4; 164:24-165:7. Instead, he basically eyeballed
the requirements using his “best judgment” and information “in his head” to come up
with dollar figures as to what it will cost DMA members to comply with the Law. See,
e.g., id. at 33:4-9; 119:17-120:11; 131:25-132:6.
An expert’s testimony must be based on facts that enable the expert to express
reasonably accurate conclusions. Cartel Asset Mgm't v. Ocwen Fin. Corp., 2010 WL
3613819, at *2 (D. Colo. Sept. 7, 2010). Mr. Barry’s opinions are not based on
ascertainable facts. For example:
He did not rely on any labor studies or published documents to determine the per
hour cost for computer programming or management. Rather, he used numbers
“in his head.” Exh. A., Barry Dep. 32:9-36:10.
He did not “do any math” to come up with the $5,000-$10,000 estimate for
companies to comply with the transactional notice. Id. at 100:19-101:2. He also
did not attempt to break up the estimate into its components other than to “kind of
go through that mentally.” Id. at 101:13-20.
The one exception is that Mr. Barry’s assistant contacted two printers for quotes on the cost of mailing
the annual customer notice. Id. at 176:12-177:18.
He did not base his estimate on the cost to modify an invoice on anything other
than “my experience in going through what I think would have to be considered
and the time to develop an estimate...” He did not put anything in writing, only
calculated it in his head. Id. at 114:18-117:6. Although Barry claimed to have
done some mathematical calculations, he did not have any notes or documents
reflecting his calculations, and his alleged calculations were based on
assumptions of hourly rates drawn from his experience. Id. at 115:7-119:16. 2
In describing how he arrived at an estimate that it would cost $2,000-$3,000 to
develop call center procedures, he estimated that it would be “pretty easy to
chew up a bunch of time.” He then calculated his estimate by “just kind of the
number of hours and then multiplying by, you know, $40 to $60 an hour” to arrive
at “it would take probably a week or so.” Id. at 123:25-127:5. 3
He could not identify particular data used in estimating that 50 percent of
customers will call a call center to inquire about the transactional notice. The
estimate is based on his “experience with issues that are not similar in terms of
legislation, but for example, when we make changes and we don’t fully think
them through and customers have their first reactions…” Id. at 130:8-131:3.
Ultimately, his estimate was based on his “best judgment.” Id. at 131:25-132:6. 4
His estimate that companies will need $3,000-$5,000 of professional assistance
from lawyers and others to comply with the transactional notice was not based on
any methodology other than “[j]ust from working through it, best judgment,
experience.” Id. at 132:15-22.
When asked what data he considered to arrive at an estimate of ongoing
compliance costs for the transactional notice, he said “just because this is a
sensitive customer service area” and companies would need a whole day of
professional time each year to review their compliance. Id. at 147:18-148:24.
In estimating the amount for companies to voluntarily calculate sales tax (a
discretionary cost), he initially thought the number would be $20,000-$25,000,
Although Mr. Barry claims to have come up with an hourly rate for internal programming by averaging the
range he earlier provided based on his experience (Id. at 116:4-15), when estimating external
programming costs, he plucked the number of “150, 160, somewhere in there” out of his initially estimated
range of $140-$225 per hour. Id. at 118:17-119:16; see also Exh. B, Barry Report, p.4.
His call center cost estimate is based on his assumption that a call would last three to four minutes.
Although there may be published literature in the field as to the average call length, he said “I’m working
with call centers all the time so I have a pretty good idea of these numbers.” Id. at 127:6-128:2. He
estimated that a call to a call center costs $1.50-$2.00; however, he did not refer to any literature in using
this figure, but stated “I just know that it’s an accurate conservative number.” Id. at 128:3-129:4.
This was reached after his email to Plaintiff’s counsel stating, “How about if we figure 50 percent?” Id. at
but then revised the number to $5,000-10,000. He arrived at the final number by
estimating that it would take companies anywhere from “a couple days work” to
modify existing software at a cost of $3,000-$4,000 to $30,000-$40,000 to buy
new software, and then picking a number in the middle that he “thought  was on
the low end of the range, and I think it’s reasonable.” Id. at 149:17-19; 151:1152:25; 154:2-157:4.
He estimated the amount of time to modify computer systems for the annual
customer report either internally or by using an external vendor without
employing any methodology other than his experience. Id. at 162:1-163:4;
As to the facts underlying his conclusions on the cost of the annual customer
report, although he is sure there are published studies about the amount of an
average retail order, he did not refer to them, relying instead on his experience.
Id. at 166:3-21; see also 166:22-167:13 (same for the average number of
customer purchases); 168:13-169:4 (same for the number of repeat buyers);
180:17-182:7 (same as to the cost of mailing the annual customer report) . 5
He did not follow any particular methodology in determining that less than 20
percent of customers (or as few as 10 percent) would buy more than $500 from a
particular retailer. Id. at 169:11-171:17.
His estimated cost of $1,000-$3,000 to comply with the Department’s
specifications for submitting the annual revenue report is “some kind of a
placeholder” in case companies have to buy software. This is again based on no
methodology other than his experience. Id. at 190:7-191:17.
His estimate of the number of management hours it would take to create the
annual report to the Department of Revenue is based on “a conservative
estimate for the number of people and the salaries that we see people paying.”
Id. at 192:3-21. He couldn’t remember, however, what hourly rate he used for
management personnel, and he did not refer to any existing literature or perform
any studies to come up with a rate. Id. at 192:22-193:25.
He estimated that “somewhere between 25 and 50 percent” of customers will
abandon their purchases based on the Law, but he admitted that “I can’t
mathematically” quantify how much retailers would lose. Id. at 195:3-197:20.
Similarly, he did not contact any particular marketing service bureaus to determine what they charge for
mailing processing. Id. at 175:12-176:5.
In this case, actual facts exist to demonstrate DMA members’ costs to comply
with the Law. Retailers are currently complying with the Law, and Mr. Barry could have
researched actual compliance costs. While Mr. Barry contacted a total of 17 retailers,
software vendors, and marketing service bureaus, he failed to ask any of them about
their current compliance costs. Id. at 57:8-58:13; 63:19-64:4; 64:18-21; 79:4-14. 6
Rather, he asked them to validate what the process for compliance would be. Id. at
63:19-64:4. Even as to this level of effort, Mr. Barry failed to employ any reliable
methodology. He did not come up with standard questions to ask the retailers, did not
track their responses, and did not keep notes. Id. at 58:14-19 (admitting “I wasn’t trying
to do something methodically.”); 65:16-25. Remarkably, Mr. Barry could not remember
the names of any of the retailers with whom he spoke, and he could only remember the
names of a few of the software vendors. Id. at 60:2-61:22; 67:19-68:12; 79:15-18. Mr.
Barry admitted, however, that actual costs of compliance could have been relevant and
helpful in forming his opinions. Id. at 80:24-81:23.
For an opinion to be based on scientific knowledge, it must be more than
“subjective belief or unsupported speculation.” Mitchell, 165 F.3d at 780. Experts must
“express a reasonably accurate conclusion as opposed to conjecture or speculation.”
Goebel, 346 F.3d at 987 (internal citations omitted); see also James River Ins. Co. v.
Rapid Funding, LLC, 2009 WL 481688, at *7 (D. Colo. Feb. 24, 2009) (an expert cannot
“jump to an opinion without factual support”) (citing Hathaway v. Bazany, 507 F.3d 312,
Mr. Barry proposed to Plaintiff’s counsel that he obtain quotes from several software vendors as to what
it would take to comply with the three regulatory requirements of the Law, but never did so. Id. at 71:922; 73:4-8; 80:12-17.
318 (5th Cir. 2007) (expert testimony should be excluded when the “proposed expert
offered little more than personal assurances based on experience.”)).
Mr. Barry’s speculation is evident from the progression of his opinions. He went
through several drafts, each of which dramatically reduced his estimated compliance
costs. Some of these reductions were based on encouragement from DMA’s counsel,
who wanted the numbers to be conservative. Id. at 75:25-76:17 (regarding statement to
DMA’s counsel that “I took your advice and cut the costs down considerably to avoid
issues.”); 134:20-135:15 (explaining revision of numbers that DMA’s counsel “felt 
might be too high.”). In making these revisions, Mr. Barry did not perform any additional
studies or do additional work to come up with the new numbers. See, e.g., id. at
105:11-106:25; 107:23-108:21; 109:2-21; 122:7-22. Examples of the progression of his
analysis are as follows:
(Exh. A. Barry
(Id. at 134:15135:15)
(Id. at 107:722)
Final Opinion (Exh. B,
Exhibits A1; C1 to
(Id. at 107:23-108:21)
C, Barry Dep.
Exhs. 5, 11)
As Mr. Barry explained:
So you had initial estimates which were based on
your own experience of 50- or 30- to 50-?
And then you had estimates of 15- to 25- based
on your experience?
Based on thinking about it more.
And then you had your final estimate of 5- to
10- also based on your experience?
Yes, and a limited outside validation.
Did anything that the 17 companies you
interviewed say to you factor into your decision to
revise these numbers downwards?
I would say no. …
Exh. A, Barry Dep. 109:5-17.
In James River, the Court excluded expert testimony when a real estate
appraiser’s opinions as to the cost to rehabilitate a property and the cost to replace it
after a fire were based only on his feelings about how much it would cost. 2009 WL
481688, at *8, 10 (noting that “[f]eelings, however, are not scientifically testable.”).
Likewise, in Milne v. USA Cycling Inc., 575 F.3d 1120, 1134 (10th Cir. 2009), the Tenth
Circuit upheld the exclusion of an expert’s testimony, which was based only on his
experience with safety precautions, when the expert did not employ any studies, or refer
to any other empirical of quantitative studies and the expert’s experience was with a
different type of race.
Mr. Barry’s opinions should be excluded for the same reasons. His failure to
perform any studies, rely on any published literature, or conduct any sort of systematic
methodology in estimating costs, renders his opinions unreliable, and they should be
excluded pursuant to Rule 702.
DR. THOMAS J. ADLER’S OPINIONS ARE NOT RELIABLE BECAUSE
HIS SURVEY METHODOLOGY WAS FATALLY FLAWED AND
YIELDED UNRELIABLE DATA ON WHICH HE BASED HIS OPINIONS.
Dr. Adler is President of Resource Systems Group, Inc. ("RSG"), which
conducted a survey of Colorado consumers’ reaction to the Law (“the survey”). Exh. D,
Adler Report and Decl. ¶¶1, 3. Dr. Adler offers testimony as to the propriety of the
survey - that its methodology was appropriate, that it was conducted properly, and that
the results were gathered properly and accurately reported. Id. at ¶¶4, 6-10. Dr. Adler
also offers testimony summarizing the results of the survey. Id. at ¶ 5. Because the
survey was not designed and implemented with the "same level of intellectual rigor" that
would be used by an expert in the field, Dr. Adler's testimony regarding the survey and
its results should be excluded. See Toni's Alpacas, 2010 WL 3730382 at *2.
The creation of the survey was heavily influenced by DMA counsel. DMA
counsel drafted the initial survey questions and sent two versions of them to Dr. Adler
and RSG before the survey expert prepared any questions. Exh. F, Adler Dep. 16:2 18:16; Exh. E, Adler Dep. Exhs. 42, 43, 44. Dr Adler used those drafts for the survey's
"core questionnaire." Exh. E, Adler Dep. Exh. 47, p.1. 7 Thereafter, he sent his draft
questionnaire to a colleague, Nelson Whipple, for review. Exh. F, Adler Dep. 23:14 25:9; Exh. E, Adler Dep. Exhs. 48, 49. 8 Dr. Adler then sent a draft, known as Version 3,
to DMA counsel. Exh. F, Adler Dep. 30:6 - 32:10. The survey questionnaire
Dr. Adler stated in his deposition that he was referring to the Law as well as the draft questionnaire, Exh.
F, Adler Dep. 21:15 – 22, but his use of the word questionnaire in his email response to counsel calls into
question this explanation. See also Exh. E, Adler Dep. Exh. 28.
Comments in the various redline versions reflect the initials of the commenter as follows: MS= Matthew
Schaefer, TJA=Thomas J. Adler, and NJW=Nelson J. Whipple.
subsequently underwent additional changes suggested by DMA counsel, 9 who finally
gave Dr. Adler the go-ahead to conduct the survey using the final Version 10. Exh. F,
Adler Dep. 51:11 - 52:5; Exh. E, Adler Dep. Exhs. 67, 68.
Despite all the revisions to the questionnaire, the survey suffers from four fatal
flaws that render the methodology entirely unreliable because it: 1) contained a fatal
confound, 2) contained false information, 3) suggested a response, and 4) did not follow
First, the survey does not permit an inference of causality because it contains a
significant confound. Exh. G, Lichtenstein Decl. and Report, pp.12-17. The survey
conflates consumer privacy concerns with consumer resistance to paying a higher price,
due to tax owed.
Id. Dr. Adler recognized that the survey needed to be designed to
determine whether a seller's reporting of information to the Department would cause
consumers not to make future purchase or whether this result would be caused by a
resistance to paying taxes and thus higher prices on purchases. As he explained:
Okay. Next is May 19th. Can you tell me what
that note is?
So this looks like kind of a repeat of the
specific objectives, what we wanted to find out.
Version 3 was further revised after another phone conversation with DMA counsel, resulting in Version
4. Exh. E, Adler Dep. Exhs. 52, 53, 54; Exh. F, Adler Dep. 33:6 - 35:11. DMA counsel then made
additional changes to the survey questionnaire and sent Version 5 to Dr. Adler. Exh. E, Adler Dep. Exhs.
53, 55; Exh. F, Adler Dep. 35:13 - 36:10. Dr. Adler revised the questionnaire again and sent Version 6 to
counsel, who had yet more comments and changes. Exh. E, Adler Dep. Exhs. 56, 57; Exh. F, Adler Dep.
36:13 - 37:16. Counsel sent Version 7 to Dr. Adler with more comments and revisions and Adler
incorporated these in Version 8. Exh. F, Adler Dep. 43:1 - 44:1; 44:16 -23; Exh. E, Adler Dep. Exhs. 58,
60. The next day, Dr. Adler sent a "final review version" of the questionnaire, Version 9, to DMA counsel
incorporating changes discussed between them. Exh. F, Adler Dep. 45:5 – 21; Exh. E, Adler Dep. Exh.
62. Version 9a was created to address word processing issues. Exh. F, Adler Dep. 46:4 -23; Exh. E,
Adler Dep. Exhs. 61, 63. Still unsatisfied with the questionnaire, DMA counsel requested a substantive
change to the wording of Question 7 in Version 10. Exh. F, Adler Dep. 50:10 - 52:10; Exh. E, Adler Dep.
Exhs. 65, 66.
So you wanted to find out whether turning over the
information will cause people to not purchase?
Right, versus not paying taxes. We wanted to
avoid getting into the issue of people not liking
to pay taxes because we know nobody does.
Exh. F, Adler Dep. 137:15 - 23; Exh. E, Adler Dep. Exh. 92, p.4. Nevertheless, Dr.
Adler made no effort to factor tax avoidance or higher prices out of his survey
methodology. The survey did not follow established methodologies for factoring out this
confound. Exh. G, Lichtenstein Decl. and Report, p.15. With price/tax being a potential
cause for consumer behavior, it should have been measured and “covaried” out. Id.
Because the privacy concern cannot be disentangled from a price/tax concern, no
conclusions may be drawn from the survey results. Id. at 13. Due to this serious
confound alone, the survey methodology is “fatally flawed” and no inference of causality
can be drawn from the data. Id. at 13.
Second, the survey contained false information. One of the early survey
questions asked respondents on an agree-disagree scale whether they minded “the
State of Colorado knowing the kinds of products I buy, from whom I buy them, where
I have them shipped, and how much I spend." Exh. D, Adler Report and Decl., Exh. B
thereto, Final Survey Results, p.5 (emphasis supplied). Because the Law does not
allow the reporting of products or even categories of products purchased, this question
is misleading. Exh. G, Lichtenstein Decl. and Report, pp.17-18. Attempting to defend
the question, Dr. Adler posited that since a retailer's name is disclosed, in some cases
an inference might be drawn about the kind of product purchased. Exh. F, Adler Dep.
105:10-17. When pressed, however, Dr. Adler conceded that the question could have
been phrased in a way that was more consistent with the true requirements of the Law.
Id. at 104:7 - 107:7.
Third, the survey questions themselves suggest a response by prompting
concerns about invasion of privacy. Exh. G, Lichtenstein Decl. and Report, pp.21-22.
This type of “reactivity bias” in the methodology of the survey can result in a response
that never would have been in the consumer’s head had it not been suggested by the
question. Id. Further, suggestive questions in the survey likely influenced answers to
later questions, rendering the entire survey unreliable. Id. at 18-22.
Fourth, the survey failed to follow accepted survey standards because it did not
provide sufficient information and did not provide a “don’t know” or “no opinion”
response option. Id. at 24-27. For example, the survey asks “If you were to make a
similar purchase in the future, but with this new disclosure requirement in place, what
would you most likely do?” Exh. E, Adler Report and Decl., Exh. B, Final Survey
Results, p.19. Based on this question, Dr. Adler reports that 63% of respondents would
not make a subsequent purchase from an affected out-of state retailer. Id. at Decl. ¶5; 10
DMA's counsel’s hand is present again in Dr. Adler's declaration. It was initially drafted by counsel and
later revised by counsel. Exh. F, Adler Dep. 73:10 - 75:6; Exh. E, Adler Dep. Exh. 82. In the final version
of Dr. Adler's declaration, he describes the objectives of the survey as: 1) to determine whether Colorado
consumers consider the Law's requirement that out-of-state retailers must report their purchasing
information to the Department to be an invasion of privacy, and 2) to determine whether the reporting
requirement will affect in any way Colorado consumer's future purchases from out-of-state retailers who
are required to report such information to the Department. Exh. D, Adler Report and Decl. ¶ 3. This
differs substantially from the stated survey objective. See Exh. E, Adler Report and Decl., Exh. B, Final
Survey Results, p.3 (“Objective- Understand how a new Colorado law, and requirements imposed by [the
Department] under the law, will impact Colorado consumers' decisions about purchasing from Internet
and catalog retailers that do not collect Colorado sales tax.”).
Exh. B, Final Survey Results, p. 6. 11 However, the question does not provide sufficient
contextual information that can affect a consumer’s purchasing decisions. Exh. G,
Lichtenstein Decl. and Report, pp.23-24. Further, the absence of a “don’t know” or “no
opinion” response option fails to reduce non-meaningful responses. Id. at pp.24-26.
Due to the flawed survey methodology and the overriding influence of counsel
at all stages of the expert process, Dr. Adler’s survey is unreliable and should be
excluded. See Goebel, 346 F.3d at 992 (“Any step that renders the analysis
unreliable... renders the expert’s testimony inadmissible. This is true whether the step
completely changes a reliable methodology or merely misapplies that methodology.”).
PROFESSOR KEVIN LANE KELLER’S OPINIONS ARE NOT
RELIABLE BECAUSE HE DID NOT USE A RELIABLE
METHODOLOGY AND HIS OPINIONS ARE BASED UPON
As discussed supra Part III, Dr. Adler’s survey, upon which Prof. Keller relied for
his opinions, was fatally flawed. Because Prof. Keller relied upon this survey and
performed no independent research of his own, his opinion is unreliable and should be
excluded. Prof. Keller provides testimony that "out-of -state retailers affected by the
Law will likely suffer lost sales and permanent injury to their relationships with
customers as a result of complying with the requirement that they disclose their
After the survey was completed, the results were sent from RSG to DMA counsel asking if counsel
needed any revisions. Exh. F, Adler Dep. 64:21- 65:4; Exh. E, Adler Dep. Exh. 74. A process similar to
the survey questionnaire drafting process ensued. DMA counsel tailored and streamlined the report from
the survey expert who admitted he "typically accepts comments and suggestions from [his] clients to do it
to meet their purposes." Exh. F, Adler Dep. 68:6 - 69:2; Exh. E, Adler Dep. Exh. 76. The report was
modified and a new version of the report sent to DMA counsel. Exh. F, Adler Dep. 70:15 - 71:2; Exh. E,
Adler Dep. Exh. 79, p.2. Another draft of the report was created incorporating DMA's counsels'
modifications. Exh. E, Adler Dep. Exh. 81. When the survey report was finalized, it contained all the
changes suggested by DMA counsel. Exh. F, Adler Dep. 71:4 - 73:8; Exh. E, Adler Dep. Exhs. 21, 80,
customers' purchasing information to the Department." Exh. I, Keller Report and Decl.,
Decl. ¶1. Rule 702, however, requires that the proponent demonstrate that the
testimony is the product of reliable principles and methods and that the expert applied
those principles and methods reliably to the case. Cartel Asset Mgm't, 2010 WL
3613819, at *2. Prof. Keller’s opinions do not meet this standard.
Prof. Keller's opinions are not grounded in scientifically sound methods. See
Toni’s Alpacas, 2010 WL 3730382, at *1. Rather, his methodology consisted of no
more than discussing the case with DMA counsel 12 and reviewing the Complaint, Law,
and survey. Exh. H, Keller Dep. 25:5 -14. He spent very little time formulating the
substance of his opinions in this case 13 and did not review any literature, treatises,
documents on survey methodology, nor any documents or writings on consumer
behavior related to stated intentions. Id. at 25:14 - 29:23. Prof. Keller relied solely on
the survey results and his past work done in preparing a revision of his textbook. When
asked, however, to identify any material from his textbook on which he relied, he could
Prof. Keller’s only notes are from conversations with DMA counsel which demonstrate that he was
simply given DMA's theory of the case. See Exh. J, Keller Dep. Exh. 17. (notes with terms such as
"threatened irreparable harm, "nexus," and "physical presence - employees/facilities, agents").
Prof. Keller worked a total of eight hours on this matter. Exh. H, Keller Dep. 24:17 - 25: l. Of those
eight hours, some 40 minutes were spent reviewing DMA's complaint when he first received it, Exh. H,
Keller Dep. 82:23 - 83:7, and after that, he read the complaint several additional times. Id. at 83:8-17.
Prof. Keller also spent additional time reviewing version 9a of the survey questionnaire, id. at 67:4 72:23; Exh. J, Keller Dep. Exh. 17, and reviewing the declaration of Dr. Adler to be sure it aligned with his
own declaration, Exh. H, Keller Dep. 100: 2 -14; Exh. J, Keller Dep. Exh. 29. He spent a significant
amount of time on the telephone discussing the case with DMA's counsel. Exh. H, Keller Dep. 73:2 80:8, Exh. J, Keller Dep. Exh. 17. Further Prof. Keller "easily spent an hour or two" discussing his
declaration, which DMA's counsel drafted, Exh. H, Keller Dep. 84:1 - 86:20, Exh. J, Keller Dep. Exhs. 2426, 16, and additional time corresponding with counsel via e-mail concerning the declaration, Exh. H,
Keller Dep. 85:12 - 86:5; 96:13 - 100:14; Exh. J, Keller Dep. Exhs. 25, 29-31. He spent additional time
conferring with DMA counsel via telephone and e-mail regarding drafts of his report prepared by DMA
counsel. Exh. H, Keller Dep. 100:12 - 104:20, 107:12 - 109:17; Exh. J, Keller Dep. Exhs. 22, 32, 34, 3637. Given all this, Prof. Keller had a negligible period of time in which to formulate a reliable methodology
or reliably apply that methodology to the data.
think of none. Id. Prof. Keller did not undertake any research to support any
conclusion, or make any independent ascertainment that the data gathered by Dr. Adler
was accurate or that the survey methodology was appropriate. Id. at 25:14 - 29:23. As
discussed supra Part III, the survey questions were the product of DMA's counsel and
Prof. Keller, who claims an expertise in survey methodology, provided little or no input.
Exh. H, Keller Dep. 11:19 – 22.
Further, Prof. Keller was unfamiliar with the methodology employed by
Knowledge Networks, the company that conducted the survey for RSG, and before his
deposition never saw the documents describing that methodology, including the
calibration and weighting process applied to the results. Id. at 11:23 - 14:9; Exh. J,
Keller Dep. Exhs. 18, 19. He simply took the survey report at face value, without
inquiring about the survey process or conducting any research or investigation to test
the results. Because he accepted the survey results at face value, he concluded that
the perceived loss of privacy would change consumer purchasing behavior by the
percentages announced in the survey results. Exh. H, Keller Dep. 11:23 - 14:9; 44:1 –
8; Exh. I, Keller Report and Decl., Decl. ¶¶5-9,13.
Prof. Keller’s wholesale acceptance of the survey cannot form a reliable basis for
his opinions due to the survey’s many flaws. See Crabbe, 556 F.Supp.2d at 1224
(when an expert relies on other opinions, “the component opinion must also be fullysupported under Rule 702 as well”). Professor Keller was unaware of the significant
confound and improperly relied on the survey for a causational analysis. He testified
that "we know two things and we know that [survey respondents] consider the act to
have negative effects on their privacy and we also know that they would subsequently
be less likely to purchase from retailers and from that you can draw the conclusion of
the relationship between the privacy and the ultimate, and its importance in their
decisions to purchase from a retailer outside the state." Exh. H, Keller Dep. 53:13 - 21.
At most, Dr. Adler’s survey established a correlation between an event (the Law) and
consumer behavior (decreased purchases from affected retailers). The survey,
however, simply was not appropriately designed to draw a causal connection between
retailers’ reporting obligation and consumers' future purchasing behavior. Exh. G,
Lichtenstein Decl. and Report, pp.13-17. As a result, no inference of causality can be
drawn. Id. Dr. Keller thus lacked sufficient facts and data to support his opinion
requiring exclusion of his expert testimony. See James River Ins. Co., 2009 WL
481688 at *7 (an expert cannot jump to an opinion without factual support).
Dr. Keller's superficial involvement is further illustrated by his reliance on DMA
counsels' statement of the objectives of the survey. His declaration prepared by
counsel sets forth those objectives. Exh. J, Keller Dep. Exh. 17. Those stated
objectives, however, differ from the announced objectives in the survey. Id. at Exh. 21.
Further, Prof. Keller was unaware of the Department’s statutory obligation to keep
information gathered in connection with tax collection confidential. Exh. H, Keller Dep.
42:7 - 22.
Moreover, Dr. Keller did not rule out alternative causes for the consumer
purchasing behavior measured by the survey. While Prof. Keller acknowledges that lost
sales depend upon a number of factors, Exh. I, Keller Report and Decl.¶ 7, he never
referenced or analyzed those factors or what influence they may have on consumer
behavior. See also Exh. H, Keller Dep. 42:23-44:8 (acknowledging the many factors
that can affect consumer purchasing behavior). Neither Dr. Adler nor Prof. Keller
investigated or explained why potential causes other than concern for privacy would not
account for the predicted consumer behavior change. As a result, Prof. Keller’s
opinions are not reliable. See Crabbe, 556 F.Supp.2d at 1223-4 (discussing one factor
in the reliability analysis as whether the expert adequately accounted for obvious
Given the limited amount of time he spent analyzing the survey methodology and
data, his failure to support or challenge his conclusions with any literature in the field,
his failure to consider and rule out alternative explanations for predicted consumer
behavior, and his wholesale reliance on a survey that is fundamentally flawed, Prof.
Keller did not employ in his expert duties "the same level of intellectual rigor that
characterizes the practice of an expert in his field." Toni's Alpacas, 2010 WL 3730382,
at *2 quoting Goebel v. Denver and Rio Grande Western RR Co. at 992. Consequently,
there is simply “too great an analytical gap” between the survey data and opinions
proffered by Prof. Keller. See Crabbe, 556 F. Supp. 2d at 1223-4. While generally, the
focus of a 702 inquiry is on an expert’s methodology rather than conclusions, nothing in
Daubert or the Federal Rules of Evidence requires a district court to admit opinion
evidence “connected to existing data only by the ipse dixit of the expert.” Goebel, 346
F.3d at 992 (citing General Electric Co. v. Joiner, 522 U.S., 136, 142 (1997)).
Accordingly, the Court should exercise its gatekeeper function and exclude Prof. Keller's
WHEREFORE, Defendant respectfully requests this Court enter an Order
excluding Plaintiff’s expert witnesses F. Curtis Barry, Thomas Adler, and Kevin Lane
Keller pursuant to Federal Rule of Evidence 702.
Respectfully submitted this 16th day of December, 2010.
JOHN W. SUTHERS
s/ Melanie J. Snyder
MELANIE J. SNYDER, 35835*
Assistant Attorney General
JACK M. WESOKY, 6001*
Senior Assistant Attorney General
Business & Licensing Section
1525 Sherman Street, 7th Floor
Denver, Colorado 80203
Telephone: (303) 866-5273 (Snyder)
Telephone: (303) 866-5512 (Wesoky)
FAX: (303) 866-5395
STEPHANIE LINDQUIST SCOVILLE, 31182*
Senior Assistant Attorney General
Civil Litigation and Employment Law Section
*Counsel of Record
Attorneys for Defendant
CERTIFICATE OF SERVICE
I hereby certify that on December 16, 2010, I electronically filed the foregoing
DEFENDANT’S MOTION TO EXCLUDE THE TESTIMONY OF PLAINTIFF'S EXPERT
WITNESSES F. CURTIS BARRY, THOMAS ADLER, AND KEVIN LANE KELLER with
the Clerk of the Court using the CM/ECF system which will send notification of such
filing to the following e-addresses:
Attorney for Plaintiff
s/ Melanie J. Snyder