Campbell et al v. Facebook Inc.

Filing 109

MOTION for Extension of Time to File Plaintiffs' Motion for Extension of Class Certification and Summary Judgment Deadlines filed by Matthew Campbell, Michael Hurley. (Attachments: # 1 Proposed Order, # 2 Declaration of David Rudolph, # 3 Exhibit 1, # 4 Exhibit 2, # 5 Exhibit 3, # 6 Exhibit 4, # 7 Exhibit 5, # 8 Exhibit 6, # 9 Exhibit 7, # 10 Exhibit 8, # 11 Exhibit 9, # 12 Exhibit 10, # 13 Exhibit 11, # 14 Exhibit 12, # 15 Exhibit 13, # 16 Exhibit 14, # 17 Exhibit 15, # 18 Exhibit 16, # 19 Exhibit 17, # 20 Exhibit 18, # 21 Exhibit 19, # 22 Exhibit 20, # 23 Exhibit 21)(Sobol, Michael) (Filed on 9/16/2015)

EXHIBIT 13

Lieff Cabraser Heimann & Bernstein, LLP
275 Battery Street, 29th Floor
San Francisco, CA 94111-3339
t 415.956.1000
f 415.956.1008

July 23, 2015

VIA E-MAIL
Jeana Bisnar Maute, Esq.
[e-mail]
Priyanka Rajagopalan, Esq.
[e-mail]
Gibson, Dunn & Crutcher LLP
1881 Page Mill Road
Palo Alto, California 94304
RE:

Campbell v. Facebook, Inc., N.D. Cal. Case No. 13-cv-05996-PJH

Dear Jeana and Priyanka:
I write to follow up on our July 17, 2015 call, during which we discussed Facebook’s
proposal regarding the use of predictive coding by Facebook, as set forth in Chris Chorba’s
June 19, 2015 to Hank Bates (“June 19 Letter”), to process and cull the documents returned by
Facebook’s application of the search terms listed in the Appendix to Josh Jessen’s May 13, 2015
letter to Mr. Bates.
The June 19 Letter states that the application of the search terms “resulted in a potential
review population of approximately 600,000 unique documents” (the “Filtered By Search Term
Documents”). The June 19 Letter represents Facebook intends to iterate the predictive coding
model until it “achieve[s] a recall rate that returns a statistically significant and industryaccepted percentage of relevant documents, when applied to a subset that was manually
reviewed for relevance.” We understand that the “subset that was manually reviewed for
relevance” was the set of documents which had scores below the cutoff point established by the
model, i.e., the so-called null set.
During our call, Facebook’s document review consultants also represented that:
1.

The predictive coding technology being used was Equivio Relevance;

2.

All steps of the coding, including classification of the control set, have already
been performed by Facebook using the initial 600,000 unique documents;

3.

The control set contained 1,591 documents;

San Francisco

New York

Nashville

www.lieffcabraser.com

Jeana Bisnar Maute, Esq.
Priyanka Rajagopalan, Esq.
July 23, 2015
Page 2
4.

The control set had a point estimate of richness of 0.06%, calculated with a 95%
confidence level and a 2.5% margin of error;

5.

Apart from the analysis of the null “subset that was manually reviewed for
relevance” set forth in the June 19 Letter, the control set was assessed, i.e.,
classified, using the model and the resulting recall rate, calculated with a 95%
confidence level and a 2.5% margin of error, was 80%;

6.

The predictive model produced a cut-off score dividing Relevant and Not
Relevant documents;

7.

Facebook intends to apply the predictive coding model to all of the Filtered by
Search Terms Documents;

8.

Facebook does not intend to further review or produce any of the Filtered By
Search Term Documents that fell beneath the cut-off score established during the
predictive coding;

9.

No further training has been performed, although the model produced by the
training has been used to classify an unspecified number of additional Filtered By
Search Term Documents;

10.

Facebook has not determined whether or not the initial 600,000 Filtered By
Search Term Documents used in the training was representative of any
documents collected subsequently;

11.

No further assessments, i.e., testing for recall against a random sample drawn
from the entire set of Filtered By Search Terms Documents, have been
performed.

Further, you agreed to provide us with:
1.

The number of documents against which the search terms were run to produce
the initial 600,000 unique documents used to create the predictive coding
model;

2.

The names of the custodians whose documents were included in the set of
documents to which the search terms were applied resulting in the 600,000
unique documents used to create the predictive coding model;

3.

The number of true positives, true negatives, false positives and false negatives
that resulted from application of the predictive coding model against the control
set.

In order to assess the validity of the predictive coding process and the resulting model,
Plaintiffs request the following additional information and documents:

Jeana Bisnar Maute, Esq.
Priyanka Rajagopalan, Esq.
July 23, 2015
Page 3
1.

An explanation of how the initial 500 documents in the control set were selected;

2.

Whether seeding was used, and if so, when and how;

3.

The number of true positives, true negatives, false positives and false negatives in
the subset of documents below the cutoff scores that was manually reviewed for
relevance;

4.

The functional areas in which the custodians whose documents were included in
the 600,000 unique Filtered By Search Terms documents worked;

5.

The names and functional areas of the other identified custodians whose
documents were not included in the 600,000 documents used to create the
predictive coding model;

6.

The number of documents classified as Relevant or Not Relevant in the control
set;

7.

The number of documents classified as Relevant or Not Relevant, or which were
Skipped, in the training set;

8.

The names and qualifications of the persons who classified control set documents
as Relevant or Not Relevant;

9.

If more than one person participated in the classification of the control set,
whether they reached consensus on the classification of each document in the
control set; if they did not, how the classifications were performed;

10.

The names and qualifications of the experts used to classify the documents in the
training set and which issue(s) each expert classified;

11.

Whether the persons who classified the training set documents reached
consensus on the classification of each document in the training set; if they did
not, how the classifications were performed;

12.

If any additional manual classifications of random samples have been performed,
then, with respect to each such classification: the number of documents, true
positives, true negatives, false positives and false negatives in the random
sample, the population from which the random sample was drawn, the names of
the expert(s) who made the classifications, the issue(s) with respect to which each
expert made classifications, and, if there was more than one expert, whether
consensus was reached with respect to each document classified;

13.

The documents included in the control set, the training set, the “subset manually
reviewed for relevance,” and in any other manually classified random sample,
and the classification made with respect to each document in each set or random
sample.

Jeana Bisnar Maute, Esq.
Priyanka Rajagopalan, Esq.
July 23, 2015
Page 4
Please provide this information at your earliest convenience. If you have any questions
about or would like to discuss the foregoing, please let us know.
Sincerely,

David T. Rudolph
DTR/wp
1266913.2