Johnson et al v. Ford Motor Company
Filing
543
MEMORANDUM AND OPINION AND ORDER The Court Orders that Ford must provide the ETC source code in read-only format and need not provide Plaintiffs with write access to the ETC source code; parties are Ordered to meet and confer regarding the remaining terms of the special protective order and shall tender an Agreed Order Governing Disclosure of Ford's Source Code to the undersigned no later than 06/22/2015 as outlined within this Order. Signed by Magistrate Judge Cheryl A. Eifert on 6/12/2015. (cc: attys; any unrepresented party) (skm)
IN THE UNITED STATES DISTRICT COURT
FOR THE SOUTHERN DISTRICT OF WEST VIRGINIA
HUNTINGTON DIVISION
CHARLES JOHNSON, et al.,
Plaintiffs,
v.
Case No.: 3:13-cv-06529
FORD MOTOR COMPANY,
Defendant.
MEMORANDUM OPINION and ORDER
On April 3, 2015, the Court denied Ford Motor Company’s (“Ford”) motion for a
protective order that would have prohibited Plaintiffs from discovering Ford’s Electronic
Throttle Control (“ETC”) system source code for various makes and models of Ford
vehicles.1 (ECF No. 421 at 1). However, the Court granted “Ford’s Motion to the extent
that it request[ed] a protective order, separate from the universal protective order
already entered in this case, (ECF No. 169), which is tailored to fit Ford’s particular
concerns related to sharing its highly proprietary commercial information with
Plaintiffs.” (Id. at 2). Accordingly, the Court ordered the parties to meet and confer on
the subject of a special protective order concerning the production of Ford’s ETC source
code. (Id. at 1-2). The parties were required to submit an Agreed Order Governing
“Computer programs are made up of lines of text written in a computer language, called the ‘source code’
of that program.” SAS Institute Inc. v. World Programming Ltd., --- F.Supp.3d ----, 2014 WL 6978300, at
*2 (E.D.N.C. Sept. 29, 2014). Although this opinion references a singular “ETC source code,” Ford has
represented to the Court that different versions of the ETC source code exist for each specific model line
and possibly each specific type of vehicle within a model line (e.g. different engine types). (ECF No. 531 at
10). Ford possesses these different versions, including past versions, of the ETC source code. (Id. at 12).
1
1
Disclosure of Ford’s Source Code to the Court by April 13, or in the event that no
agreement could be reached, to submit their respective versions to the Court by the
same date. (Id.)
Since the Court’s April 3, 2015 order, the parties have been unable to agree on a
protective order governing the production of Ford’s ETC source code, and instead, have
submitted differing proposed protective orders. During a telephonic discovery
conference on April 29, 2015, the parties informed the Court that they principally
disagreed over the format in which Ford was to produce the ETC source code. Plaintiffs
requested the code in “write-access”2 format, while Ford argued that a “read-only”
format was more appropriate. The undersigned permitted the parties to submit briefing
on the issue, (ECF Nos. 502 & 504), and held a hearing related to ETC source code
discovery on May 28, 2015, (ECF No. 531). After the hearing, Plaintiffs supplied the
undersigned with a list of computer software programs (or as Plaintiffs refer to them,
“tools”) that they assert are necessary to effectively and efficiently analyze the ETC
source code, and Plaintiffs designated on that list which format the source code would
have to be produced in for the specific tool to work. Ford responded to Plaintiffs’ list by
providing the undersigned with a document either agreeing or disagreeing with the use
of each tool and asserting its own opinion as to which format the ETC source code must
be in to utilize each tool.3
2 Throughout this opinion, “write access” means that the source code is in a format that provides the
recipient of the code with the ability to change or edit the source code, including adding lines of code to,
the source code.
3 Subsequent to Ford’s submission, Plaintiffs filed a Motion for Leave to Reply to Ford Motor Company’s
June 3, 2015 Source Code Letter, (ECF No. 533), to which Plaintiffs attached a copy of their proposed
reply brief. The Court GRANTS Plaintiffs’ Motion for Leave to Reply and will consider Plaintiffs’ reply
brief.
2
The Court has thoroughly considered the arguments of the parties and ORDERS
that Ford must produce the ETC source code in read-only format and need not provide
Plaintiffs with write access to the ETC source code. Consequently, and keeping in mind
the guidance in this opinion as to the security protocols necessary for source code
review, the parties are ORDERED to meet and confer regarding the remaining terms of
the special protective order and shall tender an Agreed Order Governing Disclosure of
Ford’s Source Code to the undersigned no later than June 22, 2015. If the parties are
unable to agree on the terms not resolved by this opinion, they shall submit their
respective versions to the undersigned by electronic mail no later than 12:00 noon on
June 22, 2015, and the disputed issues shall be addressed at the regularly scheduled
telephone conference on June 24, 2015.
I.
Relevant Facts
These cases involve alleged events of sudden unintended acceleration in certain
Ford vehicles manufactured between 2002 and 2010. In particular, Plaintiffs claim that
their vehicles were equipped with defective ETC systems which were not fault tolerant,
resulting in open throttle events during which the drivers of the vehicles lacked the
ability to control the throttles. Plaintiffs assert that the mechanisms causing the
throttles to open unexpectedly were numerous, including electromagnetic interference,
resistive shorts, and other voltage and resistance fluctuations, and that these issues were
known to Ford. Despite having knowledge of the potential for sudden unexpected
acceleration, Ford nonetheless failed to properly design the ETC system to correct the
events when they occurred, and further neglected to install fail-safes, such as a Brake
Over Accelerator system, that would allow the drivers to physically prevent or mitigate
sudden acceleration.
3
In the course of discovery, Plaintiffs requested that Ford produce for inspection
and review the source code for the ETC system. Ford objected, and the parties were
unable to resolve the issue in informal discussions. Ford then filed a motion for
protective order, asking the Court to prohibit disclosure of the source code in its
entirety. The Court denied Ford’s motion and ordered the parties to meet and confer
regarding the terms for production of the ETC source code under a special protective
order. The parties were not able to agree on the terms of a special protective order, and
instead submitted proposed protective orders and briefs outlining their respective
positions. After discussing the issue no less than twice during telephonic discovery
conferences, the Court held a hearing on the issue where testimony was provided by two
expert witnesses—Nigel Jones for Plaintiffs and Dr. John Kelly on behalf of Ford.
Mr. Jones earned a Bachelor of Science degree in Electrical and Mechanical
Engineering from Brunel University in London, and he is currently employed at the Barr
Group as Chief Engineer and at R.M.B. Consulting, Inc., as its president. (ECF No. 496-1
at 1-3). In these positions, Mr. Jones has designed both hardware and firmware for
various devices.4 (Id. at 1-2). Mr. Jones has also performed “expert witness work,”
including reverse engineering of hardware and software, as well as code review. (Id.) At
the hearing, Mr. Jones performed a demonstration of the typical steps taken in
reviewing source code. He utilized integrated development environment software,5
4 Firmware is defined as “computer programs contained permanently in a hardware device (as a read-only
memory).” Merriam-Webster Online Dictionary, http://www.merriam-webster.com/dictionary/firmware
(last visited June 10, 2015).
An integrated development environment is “a programming environment that has been packaged as an
application program, typically consisting of a code editor, a compiler, a debugger, and a graphical user
interface
(GUI)
builder.”
Search
Software
Equality,
http://searchsoftwarequality.techtarget.com/definition/integrated-development-environment
(last
visited June 10, 2015).
5
4
which permitted him to easily search, compile,6 and debug7 source code. (ECF No. 531 at
47-51). In relation to reading and searching the code, Mr. Jones stressed that some type
of integrated development environment must be used; otherwise, attempting to read the
code would be extremely time consuming given that (1) a skilled engineer can only read
and understand 100 lines of code each day, and (2) Ford’s ETC source code presumably
contains around one million lines of code. (Id. at 53). With regard to the use of a
compiler, Mr. Jones displayed how a compiler can detect errors in the code during the
compiling process and provide warnings at the completion of compiling. (Id. at 56).
Because many different types of compilers exist, Mr. Jones testified that he would need
to obtain and use the same compiler that Ford uses when analyzing source code. (Id. at
60). Mr. Jones also emphasized the importance of using a debugger to test “timing
relationships” within the code, “race conditions,” and “stack usage.”8 (Id. at 50-51).
After explaining the usefulness of a debugger, Mr. Jones demonstrated how code
may be analyzed using a software simulator and the process of fault injection.9 (Id. at
A compiler is “[s]oftware that translates a program written in a high-level programming language ... into
machine language.” PC Magazine, http://www.pcmag.com/encyclopedia/term/40105/compiler (last
visited
June
10,
2015);
see
also
American
Heritage
Dictionary
Online,
https://ahdictionary.com/word/search.html?q=compiler (last visited June 10, 2015) (defining compiler
as “[a] program that translates another program written in a high-level language into machine language
so that it can be executed.”); (ECF No. 531 at 48). In other words, a compiler translates source code into
binary code or object code that can be understood by a computer.
6
“Debugging software means locating the errors in the source code.”
http://www.pcmag.com/encyclopedia/term/41022/debug (last visited June 10, 2015).
7
PC
Magazine,
A stack is “[a] set of hardware registers or a reserved amount of memory used for arithmetic
calculations, local variables or to keep track of internal operations (the sequence of routines called in a
program). For example, one routine calls another, which calls another and so on. As each routine is
completed, the computer returns control to the calling routine all the way back to the first one that started
the sequence.” PC Magazine, http://www.pcmag.com/encyclopedia/term/51954/stack (last visited June
10, 2015).
8
9 Mr. Jones described what he meant when using the term simulator: “I’m just talking about a simulator
that will execute software, recognizing that’s [sic] not interacting with any real hardware. Because it
allows me to do things like measure how long a task takes to execute, measure how much stack size they
use, and so on.” (ECF No. 531 at 61).
5
59-60). By using a software simulator, Mr. Jones explained that he could “change[]
[Ford’s] code to force it to accept two inputs that then get processed,” so that he could
see the affect that those changes have on the output. (Id. at 60). Mr. Jones
acknowledged that by forcing the code to accept certain values, the resulting code was
not the same code that is contained in Ford’s vehicles; however, he asserted that he
would perform such testing if he were developing code for Ford and that fault injection
using simulation software was necessary as some faults are “very difficult to create in
physical hardware.” (Id. at 59-60). When questioned by the Court whether the inputs
that Mr. Jones required the code to accept could ever occur in “real life,” Mr. Jones
responded: “[T]ypically, if, for example, you have a frayed wire, a loose connection, or
you’ve got a lot of interference from a cell tower, signals can take on just about any
value. And so if I was designing something like [ETC source code], I would design the
software to accommodate any value.” (Id. at 63). Mr. Jones then further explained his
method of testing code using a simulator: “I would examine the code. I would develop
some hypothesis about maybe this code doesn’t work the way it should. And then I
would potentially construct an experiment to test that hypothesis. Now, sometimes I can
run the test purely as a thought experiment. Other times I do need to go into the
simulation environment because it is so complicated and say, I am going to put these
numbers in and see what comes out the other end.” (Id. at 64). Mr. Jones insisted that
any changes made to source code can “always” be identified and presented one program
that tracks source code changes. (Id. at 57). With regard to source code format
limitations, Mr. Jones testified that he could use search and “comparison” tools on readonly source code, but could not use a debugger on read-only code because “you have to
be able to create [] files in order to run the debugger.” (Id. at 65). It is also apparent that
6
the fault injection in the way described by Mr. Jones could not occur with read-only
code.10
Dr. Kelly testified that he is a software engineer and that he has experience
working with nuclear reactor safety shutdown systems, flight control systems, medical
devices, radiation devices, train control systems, and diesel engine control systems. (Id.
at 81, 92). He stated that he had been hired as an expert in a number of cases “to assess
and analyze” source code. (Id. at 81). Dr. Kelly asserted that he had “signed onto many
protective orders” in some of those cases and that Ford’s proposed protective order for
producing its ETC source code was “entirely consistent” with those past protective
orders. (Id. at 82). According to Dr. Kelly, Plaintiffs are “looking for conditions that
would cause the software to fail,” which can be accomplished by examining the software,
developing a hypothesis, and then “actually testing the real binary on the real PCM
[Powertrain Control Module] to see if that can occur.” (Id. at 83). Dr. Kelly explained
that such testing could be performed on the hardware module, which is connected to a
computer, by forcing inputs into various input pins through the application of voltages
to specific places on the hardware module. (Id. at 84). He insisted that write access to
the source code was unnecessary to perform the testing that Plaintiffs desired and that
there is no need “to change the code in order to test the binary.” (Id. at 83-84). In
response to Mr. Jones’s demonstration where he changed certain portions of the code,
At the conclusion of Mr. Jones’s testimony, the Court inquired of Plaintiffs’ counsel whether Plaintiffs’
experts had performed black-box testing on any of the purportedly defective vehicles. (ECF No. 531 at 71).
Black-box testing is the “[t]esting [of] software based on output requirements and without any knowledge
of
the
internal
structure
or
coding
in
the
program.”
PC
Magazine,
http://www.pcmag.com/encyclopedia/term/38733/black-box-testing (last visited June 10, 2015).
According to Dr. Kelly, black-box testing may be utilized to test usability problems, concurrency errors,
initialization errors, termination errors, ordering errors, and timing errors. (ECF No. 531 at 87-88).
Plaintiffs’ counsel responded that testing “akin” to black-box testing had been performed on two vehicle
models and that Plaintiffs’ experts were able to recreate sudden unexpected acceleration events during
that testing. (Id. at 72, 75).
10
7
Dr. Kelly testified that the “numbers set in the [binary] code cannot change” in the real
world. (Id. at 84).
In relation to whether a read-only format would obstruct Plaintiffs’ experts’
ability to analyze the source code, Dr. Kelly asserted that the integrated development
environment utilized by Mr. Jones could still be used with read-only access to the source
code, as could a compiler and a debugger. (Id. at 86, 98, 100). Dr. Kelly also testified
that read-only source code could be “put out on a simulator, as an emulator,” and be
tested.11 (Id. at 87). However, Dr. Kelly acknowledged that read-only source code could
not be rewritten to cause it to fail, and could not be changed to inject faults into the
code. (Id. at 89). On cross-examination, Dr. Kelly opined that whether the source code
was produced in read-only format or write-access format would “make no difference” to
the type of code analysis that he thought should be performed in this case. (Id. at 94).
Dr. Kelly explained that “if [the source code] is writable, then you run into the problem
with changing the code and tracking those changes and determining what code exactly it
is that you are testing. … The problem is, in a complex system … tracking all of the
changes to the source code and how they are propagated down to the binary is very
difficult.” (Id. at 94-95). Dr. Kelly acknowledged that it was possible to track any
changes made to the source code, but insisted that changing the code “adds an
incredible complexity and it becomes extremely difficult to find out what files are
actually on a binary. There are potentially thousands of files that can be changed in
many different ways, and it becomes next to impossible to determine accurately which
file made up a particular binary. … [I]t becomes extremely difficult to figure out what it
11 An emulator is “[h]ardware, software or a combination of the two that enables a computer to run
programs for another platform. … For example, Apple's iOS ‘simulator’ and Google's Android ‘emulator’
are both software utilities that run their respective mobile apps in the computer for testing purposes.” PC
Magazine, http://www.pcmag.com/encyclopedia/term/42579/emulator (last visited June 10, 2015).
8
is you did at all.” (Id. at 96-97).
On the subject of efficiency in analyzing the source code, Dr. Kelly opined that
having the source code in write-access format would not decrease the amount of time
necessary to perform most of the tasks that Mr. Jones demonstrated. (Id. at 99). Dr.
Kelly remarked that the only process made easier with write-access source code would
be testing the code by “forc[ing] it to have certain inputs,” which could also be done with
black-box testing or white-box testing.12 (Id. at 99). With regard to fault injection, Dr.
Kelly testified that “injecting faults into the code is changing the code,” and that he was
unaware whether Ford injects fault into its source code during its testing process. (Id. at
101-02). Dr. Kelly also stated that, in his experience, deliberately modifying source code
so that it fails in order to determine how the code operates is not done as part of the
source code development and testing process. (Id. at 102). In addition, Dr. Kelly
explained that he did not believe that there was any benefit to be gained from
introducing errors into the code. (Id. at 103). He added that faults could be injected
“into the as-built system” by “set[ting] various ends to certain values” or corrupting
certain memory locations. (Id. at 106).
Before the hearing concluded, the parties discussed how the source code would
be viewed by Plaintiffs’ experts. Plaintiffs’ counsel suggested that a physically secure
room containing Ford’s server be created by Ford wherever it likes and that a virtual
private network (“VPN”) be established between Ford’s server and another location near
Plaintiffs’ experts. (ECF No. 531 at 112). The location near Plaintiffs’ experts would
contain computers for Plaintiffs’ experts’ use that would access Ford’s source code and
12 White-box testing is “[t]esting software with the knowledge of the internal structure and coding inside
the program.” PC Magazine, http://www.pcmag.com/encyclopedia/term/54432/white-box-testing (last
visited June 10, 2015).
9
any tools used to analyze the source code over the VPN. (Id. at 113). To access the
computers at the Plaintiffs’ experts’ location, Plaintiffs’ counsel suggested that the
computers be equipped with biometrics scanners (e.g. fingerprint or retina scanners).
(Id.) Plaintiffs’ counsel averred that any increased risk that Ford’s source code might be
intercepted by a hacker who accesses the VPN was “not severe enough” to justify the
cost and additional time that would be necessary if all of the source code review were to
take place at a facility designated by Ford. (Id. at 114). In response, Ford’s counsel
objected to Plaintiffs’ counsel’s VPN proposal and asserted that Ford’s source code has
never been placed anywhere outside of a secured location at a Ford facility and that the
code did not “get carried around or put up on other networks.” (Id. at 115, 119); see also
(ECF No. 502 at 2) (Ford’s brief on the issue stating that Ford has never produced its
ETC source code in the manner proposed by Plaintiffs). Yet, Ford’s counsel
acknowledged that pieces of the source code during the design stage might be
transported out of Ford’s facility. (ECF No. 531 at 119-20). After hearing the parties’
positions, the Court made the parties aware that it shared Ford’s concerns as to
Plaintiffs’ counsel’s VPN proposal.
II.
Positions of the Parties
In its brief filed before the hearing, Ford contends that Plaintiffs should only be
permitted to inspect and search a read-only version of the ETC source code, and that
during Plaintiffs’ review of the source code, they should not be allowed to use
“additional tools or applications,” including compilers and simulators.13 (ECF No. 502 at
1, 10). Ford argues that Plaintiffs desire the production of the ETC source code in writeAt the hearing, Ford’s counsel indicated that he would have to speak with his client as to whether it
objects to Plaintiffs’ experts’ use of a compiler or debugger on the ETC source code in read-only format.
(ECF No. 531 at 109).
13
10
access format so that they can manipulate it, which Ford believes contradicts the Court’s
prior ruling that ordered the production of the ETC source code for review. (Id. at 7).
According to Ford’s brief, source code may be manipulated “without leaving physical
evidence or a trail of changes.” (Id. at 8). Ford argues that source code altered by
Plaintiffs’ experts “would not be representative of the ETC system in vehicles sold by
Ford,” because the binary image flashed into the PCM ROM (read only memory) cannot
be changed without a special tool to erase the ROM and flash a different binary image
into the ROM. (Id.) In addition, Ford insists that providing write access to the source
code increases a wrongdoer’s ability to create a counterfeit Ford ETC system binary
image. (Id. at 8-9). Furthermore, Ford asserts that the ETC source code has never been
provided to any non-Ford personnel, including suppliers, for anything other than “eyes
on” review on “an as needed basis at a Ford facility on a Ford supplied computer with a
Ford employee present and operating the computer.” (Id. at 9).
In their brief submitted before the hearing, Plaintiffs contend that they must have
write access to the source code in order to perform an adequate and efficient analysis of
the code. Plaintiffs claim that they must be able to test the source code by injecting
faults “into Ford’s system to see how it handles or addresses those faults.” (ECF No. 504
at 2). They assert that “[t]his is a common and critical testing mechanism.” (Id.)
Plaintiffs also maintain that they must be able to analyze the “interactions between code
sections,” and that this cannot occur just by reading the code. (Id. at 4). In addition,
Plaintiffs insist that any changes that they make to the source code during their testing
of the code can be revealed by using a file comparison tool. (Id.) Finally, Plaintiffs argue
that they must be permitted to use a compiler and any related tools in order to “gain a
full understanding of how the system actually communicates,” and that if they do not
11
have access to a compiler, then the source code review process will be protracted and
more expensive. (Id. at 7-8).
After the hearing, Plaintiffs submitted to the Court a spreadsheet containing the
names and descriptions of approximately fifty software tools that they hope to use in
analyzing the ETC source code. Plaintiffs also indicated whether those tools were
compatible with a read-only version of the source code. Plaintiffs’ tool list includes
virtual machines for operating systems; compilers, linkers, assemblers, simulators,
debuggers, integrated development environments, static analysis tools, scripts, and
batch files used by Ford; custom tool development programs; script development
software; searching and matching programs; word processing programs; Adobe
Acrobat; file compression programs; an encryption program; Linux editors; Windows
editors; version control software; static analysis tools other than those used by Ford; file
comparison software; a metrics tool; and source code “formatters and beautifiers.”
According to Plaintiffs, a majority of these programs can be used with a read-only
version of source code, including the simulators and debuggers used by Ford. In
contrast, Plaintiffs assert that the compilers used by Ford “probably” require source
code in a write-access format. Nearly all of the tools listed by Plaintiffs would require
write access permission for the file system. In other words, those tools would require
the ability to create files or data on the computer’s hard drive or on the computer’s
network drive, but would not require the ability to change or edit the source code itself.
In response, Ford’s counsel sent a letter informing the Court that Ford objected
to Plaintiffs’ experts’ use of any compiling tools on Ford’s ETC source code. In the event
that the Court allows Plaintiffs’ experts to utilize a compiler, Ford has requested that
Plaintiffs’ experts’ compiler be similar to that which Ford uses (the Green Hills Software
12
Integrated Development Environment). Ford also objects to a multitude of Plaintiffs’
proposed tools on the grounds that they pose “a significant safety or security threat to
Ford,” they require the source code in write-access format in order to run, they
“implicate[] the production by Ford of source code for systems far beyond the ETC
system,” or they are duplicative of other tools that Ford has agreed Plaintiffs may use.
Ford disagrees with Plaintiffs that many of their listed tools may be used with source
code in a read-only format. Importantly, Ford asserts that its simulators require source
code to be in write-access format; however, Ford’s compilers, assemblers, linkers, and
debuggers do not require write-access format. Ultimately, Ford has agreed to the use of
six tools or types of tools proposed by Plaintiffs, including the static analysis tools used
by Ford, Windows PowerShell version 5 (a script developing tool), Veracrypt (an
encryption tool), PC-Lint (a static analysis tool), RSM (a metrics tool), and Beyond
Compare (a file comparison tool).
III.
Relevant Law
Federal Rule of Civil Procedure 26(b)(1) provides that:
Parties may obtain discovery regarding any matter, not privileged, that is
relevant to the claim or defense of any party, including the existence,
description, nature, custody, condition, and location of any books,
documents, or other tangible things and the identity and location of
persons having knowledge of any discoverable matter ... Relevant
information need not be admissible at the trial if the discovery appears
reasonably calculated to lead to the discovery of admissible evidence.
While the claims and defenses raised in the case should be the focus of discovery,
broader discovery is permitted when justified by the particular needs of the case. Fed. R.
Civ. P. 26(b)(1), advisory committee notes (2000). In general, information is relevant,
and thus discoverable, if it ‘“bears on, or ... reasonably could lead to other matter[s] that
could bear on, any issue that is or may be in the case. Although ‘the pleadings are the
13
starting point from which relevancy and discovery are determined ... [r]elevancy is not
limited by the exact issues identified in the pleadings, the merits of the case, or the
admissibility of discovered information.’” Kidwiler v. Progressive Paloverde Ins. Co.,
192 F.R.D. 193, 199 (N.D.W.Va. 2000) (internal citations omitted). In many cases, “the
general subject matter of the litigation governs the scope of relevant information for
discovery purposes.” Id. The party resisting discovery, not the party seeking discovery,
bears the burden of persuasion. See Kinetic Concepts, Inc. v. ConvaTec Inc., 268 F.R.D.
226, 243-44 (M.D.N.C. 2010)(citing Wagner v. St. Paul Fire & Marine Ins. Co., 238
F.R.D. 418, 424-25 (N.D.W.Va. 2006)).
Simply because information is discoverable under Rule 26, however, “does not
mean that discovery must be had.” Schaaf v. SmithKline Beecham Corp., 233 F.R.D.
451, 453 (E.D.N.C. 2005) (citing Nicholas v. Wyndham Int'l, Inc., 373 F.3d 537, 543
(4th Cir. 2004)). For good cause shown under Rule 26(c), the court may restrict or
prohibit discovery when necessary to protect a person or party from annoyance,
embarrassment, oppression, or undue burden or expense. Fed. R. Civ. P. 26(c). In
addition, Rule 26(b)(2)(C) requires the court, on motion or on its own, to limit the
frequency and extent of discovery, when (1) “the discovery sought is unreasonably
cumulative or duplicative;” (2) the discovery “can be obtained from some other source
that is more convenient, less burdensome, or less expensive;” (3) “the party seeking the
discovery has already had ample opportunity to collect the requested information by
discovery in the action;” or (4) “the burden or expense of the proposed discovery
outweighs its likely benefit, considering the needs of the case, the amount in
controversy, the parties’ resources, the importance of the issues at stake in the action,
and the importance of the discovery in resolving the issues.” Fed. R. Civ. P.
14
26(b)(2)(C)(i)-(iii). This rule “‘cautions that all permissible discovery must be measured
against the yardstick of proportionality.’” Lynn v. Monarch Recovery Mgmt., Inc., 285
F.R.D. 350, 355 (D.Md. 2012) (quoting Victor Stanley, Inc. v. Creative Pipe, Inc., 269
F.R.D. 497, 523 (D.Md. 2010)). To insure that discovery is sufficient, yet reasonable,
district courts have “substantial latitude to fashion protective orders.” Seattle Times Co.
v. Rhinehart, 467 U.S. 20, 36, 104 S.Ct. 2199, 81 L.Ed.2d 17 (1984).
Federal Rule of Civil Procedure 26(c)(1)(G) allows the court, for good cause, to
issue an order “requiring that a trade secret or other confidential research, development,
or commercial information not be revealed or be revealed only in a specified way.” In
order for the court to apply the rule, two criteria must exist. First, the material sought to
be protected must be “a trade secret or other confidential research, development, or
commercial information.” Second, there must be a “good cause” basis for granting the
restriction. The party seeking protection bears the burden of establishing both the
confidentiality of the material and the harm associated with its disclosure. Deford v.
Schmid Prods. Co., 120 F.R.D. 648, 653 (D.Md. 1987) (citing Cipollone v. Liggett
Group, Inc., 785 F.2d 1108, 1121 (3d Cir. 1986)). Once these elements are demonstrated,
the burden shifts to the party seeking disclosure to show that the material is relevant
and necessary to its case. Empire of Carolina, Inc. v. Mackle, 108 F.R.D. 323, 326 (D.C.
Fla. 1985). The court “must balance the requesting party’s need for information against
the injury that might result if uncontrolled disclosure is compelled.” Pansy v. Borough
of Stroudsburg, 23 F.3d 772, 787 (3d Cir. 1994) (quoting Arthur R. Miller,
Confidentiality, Protective Orders, and Public Access to the Courts, 105 Harv. L. Rev.
427, 432-33 (1991)).
15
If the court determines that disclosure is required, the issue becomes whether the
materials should be “revealed only in a specified way.” Fed. R. Civ. P. 26(c)(1)(G).
“Whether this disclosure will be limited depends on a judicial balancing of the harm to
the party seeking protection (or third persons) and the importance of disclosure to the
public.” Id. Factors to consider when deciding if and how to limit disclosure include:
(1) whether disclosure will violate any privacy interests; (2) whether the
information is being sought for a legitimate purpose or for an improper
purpose; (3) whether disclosure of the information will cause a party
embarrassment; (4) whether confidentiality is being sought over
information important to public health and safety; (5) whether the sharing
of information among litigants will promote fairness and efficiency; (6)
whether a party benefiting from the order of confidentiality is a public
entity or official; and (7) whether the case involves issues important to the
public.
Pansy, 23 F.3d at 787-91. The court exercises broad discretion in deciding “‘when a
protective order is appropriate and what degree of protection is required.’” Furlow v.
United States, 55 F. Supp. 2d 360, 366 (D.Md. 1999) (quoting Seattle Times Co., 467
U.S. at 36).
IV.
Discussion
As discussed above, the parties disagree over the format in which the ETC source
code must be produced. Beyond that, the parties also differ on what software tools
Plaintiffs should be permitted to use in viewing and analyzing the ETC source code.
Having considered the parties’ submissions and the testimony adduced at the hearing,
the Court finds that Plaintiffs are entitled to a read-only version of the ETC source code
because the benefits that Plaintiffs seek from gaining access to the source code can be
realized through a read-only format. As Dr. Kelly testified, most of what Mr. Jones
demonstrated at the hearing can be performed on source code in a read-only format.
(ECF No. 531 at 99). The only possible limitations of a read-only format are that the
16
source code cannot be changed and faults cannot be injected directly into the source
code by editing the code. (Id. at 89). However, as Ford and Dr. Kelly have pointed out,
changing the source code would make the resulting code unrepresentative of the code
contained in Plaintiffs’ vehicles. (Id. at 32-33, 104). Moreover, Dr. Kelly asserted that
fault injection can be performed in a way that does not implicate editing or adding lines
to the source code. (Id. at 106). Under a proportionality analysis, at this juncture, any
additional benefit of providing Plaintiffs with the code in a write-access format does not
outweigh Ford’s concerns about the security of its highly proprietary ETC source code.14
Furthermore, it would be premature, and possibly inefficient, to grant Plaintiffs write
access to the ETC source code when they have yet to even review or analyze the code in a
read-only format. Accordingly, the Court ORDERS that Ford must produce the ETC
source code in a read-only format.
As for the specific programs that Plaintiffs’ experts may use, Ford informed the
Court that its compiler and debugger could be used on read-only code, but objected to
allowing Plaintiffs to use a compiler. The Court finds that Plaintiffs should be permitted
to use a compiler in analyzing the source code as the compiler may alert Plaintiffs to
certain errors in the code. Moreover, the debugger requires compiled code. (ECF No. 531
at 49, 55-56, 86, 100). The Court agrees with both parties that Plaintiffs should be
Under the protective order related to source code in the Toyota unintended acceleration litigation, the
plaintiffs’ expert, Michael Barr, was able to use the Green Hills Compiler and Simulator as well as a
number of static analysis tools. Plaintiff’s Opposition to Toyota’s Daubert Motion to Exclude the Expert
Testimony of Michael Barr at 25, In Re: Toyota Motor Corp. Unintended Acceleration Marketing, Sales
Practices, & Products Liability Litigation, No. 8:10-cv-01460-JVS-FMO (C.D. Cal. Sept. 9, 2013), ECF
No. 100. It is unclear whether the plaintiffs were given write access to Toyota’s source code. Many of the
provisions related to analysis of the source code have been redacted from the version of the protective
order available to this Court. See Redacted Stipulated Protective Order Governing the Exchange and
Handling of Source Code and Source Code Related Material at 13-16, In Re: Toyota Motor Corp.
Unintended Acceleration Marketing, Sales Practices, & Products Liability Litigation, No. 8:10-ML02151-JVS-FMO (C.D. Cal. Mar. 31, 2011), ECF No. 1344.
14
17
required to use a compiler that is similar to the compiler that Ford used in developing
the ETC source code, so that there is no discrepancy as to how the code was compiled
when it was analyzed by Plaintiffs’ experts.
Ford offers multiple arguments against the remainder of Plaintiffs’ proposed
tools. First, Ford argues that certain programs may allow modification of the ETC source
code in RAM (random access memory) without being able to trace any edits, or permit
Plaintiffs’ experts to create additional tools for source code review without Ford’s
knowledge. However, Plaintiffs and their experts are instructed by this opinion that they
are not to change or edit the ETC source code in any way when performing their
analysis of the code. Furthermore, Plaintiffs shall be required to disclose all of the
software tools that they plan to use in analyzing the source code ten business days
before they plan to use each tool. Plaintiffs’ obligation will be continuous so that
any new tools may not be used to analyze the source code without first notifying Ford
and providing Ford the opportunity to voice any concerns to Plaintiffs and lodge an
objection to the use of the tool with this Court within five days of Plaintiffs’
proposed use of the tool. If Ford objects to the use of the tool, then the tool will not
be used until the parties reach a resolution on the issue or the Court resolves the issue.15
With these parameters in mind, the parties are advised that Plaintiffs will likely be
permitted to use standard read-only compatible tools on the ETC source code so long as
repeated testing does not unnecessarily delay the discovery process.
In addition, Ford contends that some of Plaintiffs’ proposed tools will require the
production of source code beyond the ETC source code. The Court need not address that
15 This is similar to the tool provision in the Toyota unintended acceleration litigation. Redacted
Stipulated Protective Order Governing the Exchange and Handling of Source Code and Source Code
Related Material at 14, In Re: Toyota Motor Corp. Unintended Acceleration Marketing, Sales Practices,
& Products Liability Litigation, No. 8:10-ML-02151-JVS-FMO (C.D. Cal. Mar. 31, 2011), ECF No. 1344.
18
concern at this time given that Plaintiffs have formally requested the ETC source code
only; therefore, they have been granted access to the ETC source code only. If after
reviewing the ETC source code, Plaintiffs believe that other portions of source code are
indispensable to completing a proper and efficient analysis of the ETC source code as it
is relevant to the claims and defenses in this litigation, then Plaintiffs may return to the
Court and demonstrate good cause for the production of additional portions of source
code.
Ford also objects to the use of certain proposed tools on the grounds that they
are duplicative of other proposed tools, or that Ford does not use those tools in
developing its source code. For example, Ford suggests that Plaintiffs’ experts should
not be permitted to use multiple static analysis tools. Yet, the NASA Engineering and
Safety Center recognized the value of utilizing different static analysis tools in
conducting its review of Toyota’s source code:
The team’s experience is that there is no single analysis technique today
that can reliably intercept all vulnerabilities, but that it is strongly
recommended to deploy a range of different leading tools. Each tool used
can excel at a different aspect of static analysis, which results in
remarkably little overlap in the set of warnings that is produced. The
combination of different analyzers achieves the highest value in analysis
results.
NASA Engineering and Safety Center, National Highway Traffic Safety Administration
Toyota Unintended Acceleration Investigation – Appendix A, 9 (2011), available at
www.nhtsa.gov/staticfiles/nvs/pdf/NASA_FR_Appendix_A_Software.pdf.16 When the
potentially differing strengths and weaknesses of the tools are taken into account, the
tools about which Ford complains may not be duplicative at all. Moreover, there is no
Plaintiffs’ proposed tool list includes some of the tools used by NASA’s engineers in testing Toyota’s
source code, such as Coverity (a static analysis tool) and Simulink (a program used to design, simulate,
and test systems). National Highway Traffic Safety Administration Toyota Unintended Acceleration
Investigation – Appendix A at 10, 12.
16
19
evidence that deploying similar tools to analyze the ETC source code will unreasonably
delay the analysis process. Furthermore, there is some value in allowing Plaintiffs’
experts to use certain analytical tools that Ford does not typically use in the
development process. See id. at 26 (stating that NASA engineers “intentionally applied
analysis tools” not used by Toyota).
In sum, Ford’s objections to the read-only compatible tools proposed by Plaintiffs
are unconvincing. The Court ORDERS that Plaintiffs’ experts are entitled to use those
programs listed in their spreadsheet that are compatible with a read-only format so long
as the use of multiple tools does not unnecessarily delay the discovery process. For any
additional tools that Plaintiffs’ experts wish to use, Plaintiffs must follow the process
described above.
Having concluded that Plaintiffs are entitled to the ETC source code in a readonly format and that Plaintiffs may use the listed tools that are compatible with such a
format, the Court finds that Plaintiffs should start their analysis of the ETC source code
with the vehicle models that they have already tested. As noted above, Plaintiffs’ counsel
has represented that testing was performed on two models and that Plaintiffs’ experts
were able to recreate sudden acceleration events during that testing. (ECF No. 531 at 72,
75). Plaintiffs’ experts should begin their source code analysis on the source code used in
those two models (the 2005 Explorer and the 2005 Mustang). (Id. at 76). After an initial
review of the ETC source code for those models, Plaintiffs shall report back to the Court
with an estimate of the time that it will take to complete the ETC source code analysis
for those two models. Plaintiffs shall also provide the Court with an estimate of time that
it will take to analyze the ETC source code for the additional models at issue in this
litigation, as well as a proposed schedule setting forth in order of priority the vehicle
20
models and years from which the ETC source code shall be analyzed.
Finally, the parties are ORDERED to meet and confer regarding the remaining
terms of the special protective order and shall tender an Agreed Order Governing
Disclosure of Ford’s Source Code to the undersigned no later than June 22, 2015. As
previously stated, if the parties cannot agree on the terms not resolved by this opinion,
they shall submit their respective versions to the undersigned by electronic mail no later
than 12:00 noon on June 22, 2015 and the disputed issues shall be addressed at the
regularly scheduled telephone conference on June 24, 2015. The Court urges both
parties to review the redacted source code protective order filed in the Toyota sudden
unintended acceleration litigation17 to ascertain its applicability in the instant matter. As
the Court stated at the hearing on this issue, the Court is not inclined to adopt Plaintiffs’
VPN proposal. The source code review should occur in a secure room in a secure facility
where access to the room may be strictly controlled.18 Additionally, the Court suggests
that any person who reviews the source code be required to sign a confidentiality
agreement that includes a penalty clause for failure to comply with the terms of the
agreement. Furthermore, no copies of the source code or any portion thereof (including
electronic copies) shall be made, unless the parties otherwise agree, and no outside
electronic devices shall be permitted in the secure room. To the extent that the parties
have not already reached agreement on the following issues, the parties should also
consider: the number of computers that will be placed in the secure room; what
specifications those computers will have; how Plaintiffs’ experts’ tools (software) will be
17
ECF No. 1344 in Case No. 8:10-ML-02151-JVS-FMO (C.D. Cal. Mar. 31, 2011).
18 By electronic mail sent on June 11, 2015, Ford suggests that the review occur at “a secure location at a
Ford Building with Ford IT infrastructure” to offer the best protection, and access to IT support if
necessary. Ford indicates that it is in the process of identifying a location that will accommodate Plaintiffs’
desire to analyze the source code after regular business hours.
21
loaded onto the computers; how access to the secure room will be monitored (e.g. by a
security guard, with cameras, etc.); if the inside of the secure room will be monitored;
whether a neutral system administrator will be required; the days and hours that
Plaintiffs’ authorized experts or representatives may access the secure room; how
Plaintiffs will notify Ford of the identity of those who will be accessing the secure room;
whether note taking will be allowed, and if so, how those notes will be stored; the
process for returning the notes to Ford at the conclusion of the litigation; where expert
reports will be written; and if there will be a limitation on citing to the source code in
any documents filed with the Court.
The Clerk is directed to provide a copy of this Order to counsel of record and any
unrepresented party.
ENTERED: June 12, 2015
22
Disclaimer: Justia Dockets & Filings provides public litigation records from the federal appellate and district courts. These filings and docket sheets should not be considered findings of fact or liability, nor do they necessarily reflect the view of Justia.
Why Is My Information Online?