Edwards et al v. National Milk Producers Federation et al

Filing 155

ORDER GRANTING 154 STIPULATION RE THE USE OF PREDICTIVE CODING TECHNOLOGY. Signed by Judge Jeffrey S. White on April 17, 2013. (jswlc3, COURT STAFF) (Filed on 4/17/2013)

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page1 of 9

1

7

EIMER STAHL LLP
NATHAN P. EIMER (pro hac vice)
Email: [e-mail]
VANESSA G. JACOBSEN (pro hac vice)
Email: [e-mail]
DANIEL D. BIRK (pro hac vice)
Email: [e-mail]
SARAH E. MALKERSON (pro hac vice)
Email: [e-mail]
224 South Michigan Avenue, Suite 1100
Chicago, IL 60604-2516
Telephone: [phone]
Fax: [phone]

8

Attorneys for Defendant Land O’Lakes, Inc.

9

[Additional Counsel Appear on Signature Page]

2
3
4
5
6

10
11

UNITED STATES DISTRICT COURT

12

NORTHERN DISTRICT OF CALIFORNIA

13

SAN FRANCISCO DIVISION

14
15
16
17
18
19
20
21
22
23

MATTHEW EDWARDS, GEORGIA
BROWNE, and TORAH MONTESSORI
SCHOOL, individually and on behalf of all
others similarly situated,
Plaintiffs,
v.
NATIONAL MILK PRODUCERS
FEDERATION aka COOPERATIVES
WORKING TOGETHER; DAIRY FARMERS
OF AMERICA, INC.; LAND O'LAKES,
INC.; DAIRYLEA COOPERATIVE INC.; and
AGRI-MARK, INC.,

CASE NO.: 3:11-CV-04766-JSW
[consolidated with 11-cv-04791-JSW
and 11-cv-05253-JSW]
CLASS ACTION
JOINT STIPULATION AND
[PROPOSED] ORDER RE THE USE
OF PREDICTIVE CODING
TECHNOLOGY
ACTION FILED: September 26, 2011

Defendants.

24
25
26
27
28

JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page2 of 9

1
2
3

Plaintiffs and Defendant Land O’Lakes, Inc. (collectively “the Parties” and each, individually,
a Party) hereby stipulate, agree, and propose as follows:
I.

DEFINITIONS
A.

4

“Document review corpus” refers to the body of documents remaining after

5

the exclusion of known system files, incompatible file types (e.g., database files), duplicates,

6

and documents that fall outside the agreed date range.
B.

7
8

“Confidence level” means the statistical reliability of a result and in this

instance refers to the likelihood that a measurement reached through sampling is accurate.

9

C.

“Estimation interval” refers to the statistical error rate of a measured

10

confidence level.

11

II.

12

The Parties have discussed the methodologies or protocols for the search and review of

PROTOCOL

13

documents collected from Land O’Lakes and have agreed that Land O' Lakes will utilize

14

Recommind's Axcelerate software for its review workflow and production in this case. Axcelerate

15

incorporates predictive coding (also known as technology-assisted review) functionality, which Land

16

O'Lakes will leverage for a more cost efficient and higher quality review. The Parties agree that the

17

following protocol will be followed to identify potentially responsive documents for review:

18

1.

Step One – Document Collection: Land O’Lakes will collect documents in the

19

possession, custody, or control of each custodian agreed upon by the Parties and in additional

20

locations under the possession, custody, or control of Land O’Lakes (such as shared drives or

21

departmental files) immediately upon entry of this order and isolate the document review corpus.

22

2.

Step Two – Control Set: After loading the document review corpus into Axcelerate,

23

Land O’Lakes will generate an initial control set of documents. The control set will be a random,

24

statistically valid sampling of documents to estimate the number of responsive documents in the

25

corpus. The control set sample shall be determined using a 95% confidence level and 2% estimation

26

interval. The control set sample will then be reviewed for responsiveness and privilege, and Land

27

O’Lakes will provide Plaintiffs with a tracking report of the results of the control set generation and

28

review. At any point during the predictive coding process, Land O’Lakes’ e-discovery liaison will
1
JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page3 of 9

1

meet and confer at Plaintiffs’ request to discuss the results and method of the process. Land O’Lakes

2

agrees to confer in good faith regarding the incorporation of any input from Plaintiffs on the

3

application process.
3.

4

Step Three – Seed Set and Initial Training: All responsive documents identified

5

during review of the control set will be included in an initial seed set, which will be used to train

6

Axcelerate to identify other potentially responsive documents in the document review corpus. The

7

seed set also will include responsive documents identified by Land O’Lakes through the use of search

8

and analytical tools, which will include the following:

9

a)

Approximately 400 documents already identified as responsive by Land O’Lakes;

10

b)

The 500 most highly ranked1 responsive documents hit upon by an application of the

11

Boolean search terms selected by Land O’Lakes (listed in Appendix A), excluding

12

documents already contained in (a), above; and
c)

13

The 500 most highly ranked responsive documents hit upon by application of Boolean

14

search terms selected by Plaintiffs (listed in Appendix A), excluding documents

15

already contained in (a) and (b), above.

16

Land O’Lakes will provide Plaintiffs with a search hit count of all searches conducted and will

17

produce all non-privileged documents included in the seed set.
4.

18

Step Four – Iterative Review and Further Training: Once Axcelerate has

19

identified and prioritized potentially responsive documents based on the initial training, Land

20

O’Lakes will review a sample set of those documents for responsiveness. The sample set will consist

21

of at least the 500 documents ranked most highly by Axcelerate in order of relevance. After this

22

second round of review, the system will be trained again based on reviewer feedback to identify and

23

prioritize more potentially responsive documents. Land O’Lakes will provide Plaintiffs with tracking

24

data for this round of review.
5.

25
26

Step Five – Review: Land O’Lakes will then conduct a manual review of all

documents identified by Axcelerate as potentially responsive. During this review, the iterative

27
28

1

Axcelerate ranks returned documents in order of responsiveness to search terms.
2

JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page4 of 9

1

process will continue; as more responsive documents are reviewed and coded, they will be used to

2

further hone the system’s ability to identify and prioritize other potentially responsive documents

3

among the remaining, unreviewed portion of the document review corpus. Newly identified

4

documents also will be subject to manual review. Review and iterative training will proceed until all

5

identified documents have been reviewed and the system indicates that the remainder of the

6

document review corpus is not likely to be responsive. Land O’Lakes will continue to provide

7

tracking data to Plaintiffs, and all responsive, non-privileged documents will be produced.

8
9

6.

Step Six – Validation: After Land O’Lakes has conducted enough rounds of

iterative review and training to generate a conclusion from Axcelerate that the remaining documents

10

in the document review corpus are not likely to be responsive, Land O’Lakes will perform a

11

validation test by reviewing a statistically valid and random sampling of unreviewed documents to

12

confirm that the number of potentially responsive documents in the unreviewed corpus is statistically

13

insignificant. Using a 99% confidence level and a 1% estimation interval, a random sampling of the

14

unreviewed documents will be reviewed for responsiveness. If the number of responsive documents

15

in the validation sampling results in higher than a 1% responsiveness rate, the review process will

16

continue until the validation test achieves a 1% or less responsiveness rate. All tracking data

17

regarding the validation process will be provided to the plaintiffs, and any responsive, non-privileged

18

documents identified during the process will be produced.

19

THE PARTIES HEREBY STIPULATE AND AGREE TO THE ABOVE TERMS.

20
21
22
23
24
25
26
27
28

DATED: April 16, 2013

Respectfully submitted,
By:/s/ Matthew S. Kahn
George A. Nicoud (SBN 106111)
Matthew S. Kahn (SBN 261679)
GIBSON, DUNN & CRUTCHER LLP
555 Mission Street, Suite 3000
San Francisco, CA 94105-2933
Telephone: [phone]
Fax: [phone]
[e-mail]
[e-mail]
By:/s/ Nathan P. Eimer
Nathan P. Eimer (pro hac vice)
Vanessa G. Jacobsen (pro hac vice)
3

JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page5 of 9

6

Daniel D. Birk (pro hac vice)
Sarah E. Malkerson (pro hac vice)
EIMER STAHL LLP
224 South Michigan Avenue, Suite 1100
Chicago, Illinois 60604
Phone: [phone]
Fax: [phone]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

7

Attorneys for Defendant Land O’ Lakes, Inc.

8

By: /s/ Elaine T. Byszewski
Elaine T. Byszewski
HAGENS BERMAN SOBOL SHAPIRO LLP
301 North Lake Avenue, Suite 203
Pasadena, CA 91101
Telephone [phone]
Facsimile [phone]
[e-mail]

1
2
3
4
5

9
10
11
12

Steve W. Berman
George W. Sampson
Craig R. Spiegel
HAGENS BERMAN SOBOL SHAPIRO LLP
1918 Eighth Avenue, Suite 3300
Seattle, WA 98101
Telephone [phone]
Facsimile [phone]
[e-mail]
[e-mail]
[e-mail]

13
14
15
16
17
18

Attorneys for the Plaintiffs

19
20

ATTESTATION: I, Matthew S. Kahn, attest that concurrence in the filing of this document has been

21

obtained from each of the other signatories.

22
23
24

[PROPOSED] ORDER
PURSUANT TO STIPULATION, IT IS SO ORDERED.

25
26
27

April 17, 2013
Dated: _____________________

___________________________
HONORABLE JEFFREY S. WHITE
UNITED STATES DISTRICT JUDGE

28
4
JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW

Case3:11-cv-04766-JSW Document154 Filed04/16/13 Page6 of 9

Appendix A: Search Terms

1
2

Land O’Lakes’ Search Terms

3

19

“cooperatives working together”
nmpf OR “national milk producers federation”
herd /10 retir*
cow /10 retir*
cwt AND retir*
“self help”
“herd buyout”
“milk diversion program”
cwt AND beef
cwt /5 program
export /10 assistance
cwt /5 ambassador
cwt AND (independent /5 producer)
cwt AND (independent /5 farmer)
cwt AND (target /10 remov*)
cwt AND cull*
cwt AND slaughter*
cwt AND “field staff”
cwt AND assessment
cwt AND delegate
cwt AND (milk /5 suppl*)
cwt AND (price /10 increas*)
cwt AND contribut*
cooperative /10 individual
cwt AND (cow /5 numbers)
cwt AND “farm id”
cents /5 hundredweight
amount /10 checks
cwt AND bid*
cwt AND “leadership council”
cwt /10 member*
cwt /10 summary
cwt AND presentation

20

Plaintiffs’ Search Terms

21

buyout
reentry
re-entry
reenter
re-enter
rejoin
re-join
join /10 again
price /10 increase
price /10 rise
price /10 raise
price /10 hike
price /10 stabiliz*
price /10 maintain
price /10 maintenance

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

22
23
24
25
26
27
28

5
JOINT STIPULATION AND PROPOSED ORDER RE THE USE OF PREDICTIVE CODING TECHNOLOGY
CASE NO.: 3:11-CV-04766-JSW