Garner et al v. Amazon.com Inc et al

Filing 172

ORDER denying Plaintiffs' #143 Motion to Compel. Defendants shall, if they have not already done so, immediately produce the 2,564 documents tagged as responsive through the human review process. Signed by Judge Robert S. Lasnik. (LH)

1
2
3
4
5

UNITED STATES DISTRICT COURT
FOR THE WESTERN DISTRICT OF WASHINGTON
AT SEATTLE

6
7
8

KAELI GARNER, et al.,
CASE NO. 2:21-cv-00750-RSL

9

Plaintiffs,
v.

10
11

AMAZON.COM, INC., et al.,

12

Defendant.

ORDER REGARDING PLAINTIFFS’
MOTION TO PROHIBIT USE OF
TECHNOLOGY-ASSISTED REVIEW
PROCEDURES

13
14
15
16
17

This matter comes before the Court on “Plaintiffs’ Motion to Compel Defendants to
Comply with the Court’s Prior Discovery Orders.” Dkt. # 143. In October 2022, the Court
granted plaintiffs’ motion to compel searches of the electronically-stored information

18

(“ESI”) of 36 Amazon employees using 38 search strings, as proposed by plaintiffs. Dkt.

19

# 136. Using the approved search terms, defendants identified approximately 2 million

20
21
22
23

potentially responsive documents. 1 Within two weeks of the Court’s order, defendants
notified plaintiffs that Amazon intended to “use technology-assisted review [“TAR”] tools
to prepare its production,” promising that they would “discuss these tools with Plaintiffs

24
25
26

1

In the context of the motion to compel, defendants had estimated that the search terms would return between 3.1
and 4.4 million documents.
ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 1

1
2

before beginning review.” Dkt. # 144-4 at 2. 2 Two days later, plaintiffs objected, asserting
that “[a]pplying TAR at this stage – after the parties have agreed to search terms as a basis

3
4

for culling documents, and after the Court issued an order on search terms and custodians,

5

is improper and impermissible.” Dkt. # 144-2 at 3. Plaintiffs argued that it was too late to

6

alter the protocol for the production of ESI in this case. 3 Defendants disagreed, insisting

7

that the large number of retrieved documents necessitated the use of TAR to identify

8
9

responsive, non-privileged documents. Dkt. # 144-1. Plaintiff again objected to any effort

10

“to use TAR to further review/filter the documents retrieved by the Court-ordered search

11

terms.” Dkt. # 144-3 at 3.

12
13
14

Having reviewed the memoranda, declarations, and exhibits submitted by the
parties and having heard the arguments of counsel, the Court finds as follows:
Plaintiffs make clear in reply that they are not trying to prevent Amazon from

15
16
17
18
19

conducting a responsiveness and privilege review of the universe of documents the search
terms retrieved. Rather, they seek to prevent the unilateral application of TAR, apparently
preferring that Amazon conduct its review by hand. In this district, it is presumed that the

20

use of technology-assisted review is a reasonable option for locating or filtering ESI.

21

https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fwww.wawd.uscourt

22
23
24
25
26

2

TAR is also known as computer-assisted review or predictive coding. Reviewers typically code a set of documents
as responsive or unresponsive to “train” the TAR software, which then uses the coded documents to generate
algorithms for reviewing all other documents in the selected universe for responsiveness. Progressive Cas. Ins. Co. v.
Delaney, No. 2:11-cv-00678-LRH, 2014 WL 3563467, at *10 (D. Nev. July 18, 2014);
https://legal.thomsonreuters.com/en/insights/articles/myths-and-facts-about-technology-assisted-review.
3

At approximately the same time, the parties negotiated an extension of the discovery deadline to August 16, 2023.

ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 2

1
2

s.gov%2Fsites%2Fwawd%2Ffiles%2FModelESIAgreement_CLEAN_2.1.23.docx&wdOri
gin=BROWSELINK at 3. Litigants are instructed to confer and attempt to reach agreement

3
4

on “appropriate computer- or technology-aided methodologies[] before any such effort is

5

undertaken.” Id. at 3-4. That requirement is incorporated into the April 2022 ESI

6

Agreement and Order entered in this matter, which also recognized “that searching for

7

relevant ESI is an iterative process.” Dkt. # 89 at 2-4. Case law discussing the use of

8
9

computer-assisted methodologies to review documents for production emphasizes the need

10

for cooperation and transparency. In Youngevity Int’l, Corp. v. Smith, No. No. 16-cv-

11

00704-BTM (JLB), 2019 WL 1542300, at *12 (S.D. Cal. Apr. 9, 2019), for example, the

12
13
14
15
16
17
18
19
20
21

court noted that:
Technology-assisted review of ESI does require an ‘unprecedented degree of
transparency and cooperation among counsel’ in the review and production
of ESI responsive to discovery requests. Progressive Cas. Ins. Co., 2014 WL
3563467, at *10. In this regard, courts typically ‘have required the producing
party to provide the requesting party with full disclosure about the
technology used, the process, and the methodology, including the documents
used to ‘train’ the computer.” Id.
While defendants’ discovery conduct has not been beyond criticism, the nub of this
particular discovery dispute is that when Amazon raised the issue of using TAR, plaintiffs

22

refused to discuss it. This was not a valid option given the circumstances. The parties had

23

already agreed that an extension of the discovery deadline was needed: there was therefore

24
25
26

time to exchange proposals regarding the use of predictive coding to improve the accuracy,
speed, and efficiency of the document review Amazon was then undertaking. Courts

ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 3

1
2

generally agree that the producing party is best situated to evaluate the various options for
reviewing and producing its own ESI and places the burden on that party to make an

3
4

initial, detailed proposal about the technology and methodologies it intends to use. Id. at

5

11; Hyles v. New York City, No. 10-cv-3119(AT)(AJP), 2016 WL 4077114, at *2

6

(S.D.N.Y. Aug. 1, 2016); Progressive Cas. Ins. Co., 2014 WL 3563467, at *10. The

7

requesting party then has an opportunity to consult with its own ESI discovery experts so

8
9

that it can intelligently respond and the parties can move toward an agreed protocol for the

10

use of TAR. By refusing to discuss the use of TAR at all, plaintiffs improperly short-

11

circuited this process.4

12
13
14

Plaintiffs suggest that the use of TAR on a pool of documents already retrieved
through search terms is somehow improper because it “will only reduce the document pool

15

further and will certainly not reveal documents that the application of search terms has

16

precluded.” Dkt. # 143 at 10 (quoting In re Allergan Biocell Textured Breast Implant Prod.

17
18
19
20

Liab. Litig., No. 2:19-md-2921(BRM)(ESK), 2022 WL 16630821, at *3 (D.N.J. Oct. 25,
2022)). But, as discussed above, the Model ESI Agreement in this district clearly
contemplates using TAR to filter, not just locate, documents, and the ESI Agreement

21
4

22
23
24
25
26

The Court acknowledges the similarities between the facts of this case and those of Progressive Cas. Ins. Co. v.
Delaney. In both cases, search terms had been used to identify potentially responsive ESI before the producing party
indicated that TAR should be used to further refine the production. In contrast with this case, however, the stipulated
ESI protocol in Progressive gave the producing party two options once the universe of potentially responsive
documents was identified: to produce all of the “hit” documents or to manually review the retrieved documents and
produce only those that were, in fact, responsive. 2014 WL 3563467, at *11. Progressive chose a third, unauthorized,
path by conducting a TAR review. It also withheld documents that had already been found to be responsive through
an aborted manual review process and it refused to provide details regarding its TAR proposal to the requesting party.
In those circumstances, the court declined to adopt Progressive’s predictive coding proposal because it would only
result in more discovery disputes and delay. The court instead ordered the production of all non-privileged documents
retrieved using the search terms.
ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 4

1
2

entered in April 2022 simply directs the parties “to confer to attempt to reach agreement on
. . . appropriate computer- or technology-aided methodologies[] before any such effort is

3
4
5

undertaken.” The Court finds that the use of search terms is not, standing alone, a bar to
using technology to further refine the production.

6
7

It is worth repeating at this juncture that plaintiffs do not dispute that Amazon has
the right to conduct a responsiveness and privilege review of the documents retrieved

8
9

through application of the search terms to the custodial records, they just want the review

10

done by hand rather than through predictive coding because (a) November 2022 was “too

11

late” to raise the issue and (b) the TAR process has identified only 2,564 responsive

12
13
14

documents out of an initial universe of 2,036,172. For the reasons discussed above, the
Court finds that raising the possibly of using TAR once the search terms were run and the

15

universe of hits was known was not unreasonable and did not excuse plaintiffs from

16

engaging in a discussion regarding the methodology and protocols to be used. Plaintiffs’

17

timeliness objection is overruled.

18

Plaintiffs’ concern regarding the incredibly low production rate (less than 0.13% of

19
20

the documents returned using the search terms) may be valid, but it does not appear to be

21

related to Amazon’s use of TAR. Amazon’s initial proposal regarding how their TAR

22

would work utilized an unacceptably low recall target of 75%. 5 But as the process

23
24
25
26

5

Mr. Johnson states that recall requirements between 70-80% are standard because “[a]ttempting to achieve higher
recall rates, for example 85%, can result in dramatically higher costs for the review, and likely eliminate any potential
costs savings and efficiencies that parties would hope to realize using TAR.” Dkt. # 149 at ¶ 17. The goal of discovery
is the production of all responsive, relevant, non-privileged documents. If TAR is able to assist in reaching that goal
ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 5

1
2

unfolded, Amazon continued its review after that rate was achieved, ultimately conducting
a human review of over 1.8 million documents. When a sample of the remaining pool of

3
4

unreviewed documents was analyzed, the human reviewers found no responsive

5

documents. Thus, humans reviewed over 89% of the documents identified using the

6

agreed-upon search terms and found that only 2,564 of them were actually relevant,

7

responsive, and non-privileged. The 224,924 unreviewed documents are, according to the

8
9

TAR methodology, the least likely to be responsive and, in fact, a human review of 1,527

10

of them revealed no responsive materials. Because humans reviewed the vast majority of

11

the universe of documents and the statistical estimate of responsive documents remaining

12
13
14
15

in the unreviewed documents is 0%, the estimated recall rate approaches 100% There is no
reason to suspect that the low percentage of production is, as plaintiffs argue, the result of
Amazon’s use of TAR versus human review.

16
17
18
19
20

For all of the foregoing reasons, plaintiffs’ motion to compel is DENIED.
Defendants shall, if they have not already done so, immediately produce the 2,564
documents tagged as responsive through the human review process.

21
22
23
24

Dated this 19th day of May, 2023.
Robert S. Lasnik
United States District Judge

25
26

while saving money, it is a useful tool. But if TAR is cost-effective only if 25% of responsive, relevant, nonprivileged documents remain hidden, it may costs less, but it is no longer effective.
ORDER REGARDING PLAINTIFFS’ MOTION TO
PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW
PROCEDURES - 6