AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC. et al v. PUBLIC.RESOURCE.ORG, INC.

Filing 134

MOTION for Summary Judgment , MOTION for Permanent Injunction by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 Memorandum in Support, # 2 Statement of Facts, # 3 Declaration Hutter Decl., # 4 Exhibit 1, # 5 Exhibit 2, # 6 Exhibit 3, # 7 Exhibit 4, # 8 Declaration Hudis Decl., # 9 Exhibit A, # 10 Exhibit B, # 11 Exhibit C, # 12 Exhibit T, # 13 Exhibit U, # 14 Exhibit Z, # 15 Exhibit BB, # 16 Exhibit CC, # 17 Exhibit EE, # 18 Exhibit GG, # 19 Exhibit HH, # 20 Exhibit II, # 21 Exhibit JJ, # 22 Exhibit KK, # 23 Exhibit LL, # 24 Exhibit MM, # 25 Declaration Ernesto Decl., # 26 Exhibit NN, # 27 Exhibit OO, # 28 Exhibit PP, # 29 Exhibit QQ, # 30 Exhibit RR, # 31 Exhibit SS, # 32 Exhibit TT, # 33 Exhibit UU, # 34 Declaration Wise Decl., # 35 Exhibit KKK, # 36 Exhibit LLL, # 37 Declaration Camara Decl., # 38 Exhibit MMM, # 39 Declaration Levine Decl., # 40 Exhibit NNN, # 41 Exhibit PPP, # 42 Exhibit QQQ, # 43 Exhibit UUU, # 44 Declaration Geisinger Decl., # 45 Declaration Schneider Decl., # 46 Exhibit Levine Depo Tr., # 47 Exhibit No. 1207 to Levine Depo Tr., # 48 Exhibit No. 1308 to Levine Depo Tr., # 49 No. 1308 to Levine Depo Tr., # 50 Text of Proposed Order)(Elgarten, Clifton)

IN THE UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF COLUMBIA

AMERICAN EDUCATION RESEARCH
ASSOCIATION, INC.,
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC.,
and
NATIONAL COUNCIL ON MEASUREMENT IN
EDUCATION, INC.
Plaintiff-Counterdefendants,
v.
PUBLIC RESOURCE ORG., INC.,
Defendant-Counterclaimant.

)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)

Case No. 1:14-cv-00857 (TSC)

STATEMENT OF MATERIAL FACTS

John I. Stewart, Jr. (D.C. Bar No. 913905)
Clifton S. Elgarten (D.C. Bar No. 366898)
Amanda Shafer Berman (D.C. Bar No. 497860)
CROWELL & MORING LLP
1001 Pennsylvania Avenue, NW
Washington, DC 20004-2595
[phone]
[e-mail]
Attorneys for Plaintiff-Counterdefendants

TABLE OF CONTENTS
Page
STATEMENT OF MATERIAL FACTS ...................................................................................................... 1 
A. 

The Plaintiff Sponsoring Organizations.............................................................................. 1 

B. 

Creation of the Standards for Educational and Psychological Testing ............................... 2 

C. 

Why the Standards Are Published and Their Benefits ........................................................ 7 

D. 

The Standards Are For Professional, Not Governmental, Use. .......................................... 9 

E. 

Plaintiffs’ Development of the 1999 Edition of the Standards ......................................... 10 

F. 

Description of the 1999 Standards (Hutter Decl. Exhibit 1) ............................................. 11 

G. 

The Federal Practice of Incorporation by Reference. ....................................................... 13 

H. 

DOE’s Incorporation By Reference of Some Parts of the 1999 Standards....................... 15 

I. 

Defendants Copied the 1999 Standards In Toto and Verbatim, and Enabled
Others to Copy Them, Without Plaintiffs’ Permission ..................................................... 20 

J. 

This Litigation, and the Court of Appeals’ Remand Direction ......................................... 24 

STATEMENT OF MATERIAL FACTS1
A.

The Plaintiff Sponsoring Organizations

1.

Plaintiffs, the Sponsoring Organizations, are each District of Columbia not-for-

profit corporations (Levine Decl., ¶ 4; Ernesto Decl., ¶ 3; Wise Decl., ¶ 3). Each is an active
professional organization that engages in many activities for many purposes other than to create
the Standards for Educational and Psychological Testing at issue in this case.
2.

American Education Resource Association (AERA) is the major national

scientific society for research on education and learning. AERA’s mission is to advance
knowledge about education, to encourage scholarly inquiry related to education, and to promote
the use of research to improve education and serve the public good (Levine Decl., ¶ 5).
3.

The American Psychological Association (APA) is the largest scientific and

professional organization representing psychology in the United States. APA is the world’s
largest association of psychologists and counts a vast number of researchers, educators,
clinicians, consultants and students among its members. APA’s mission is to advance the
creation, communication, and application of psychological knowledge to benefit society and
improve people’s lives (Ernesto Decl., ¶ 4).
4.

The National Council on Measurement in Education (NCME) is a professional

organization for individuals involved in assessment, evaluation, testing, and other aspects of
educational measurement. NCME’s members are involved in the construction and use of
standardized tests; new forms of assessment, including performance- based assessment; program
design; and program evaluation (Wise Decl., ¶ 4).
1

Much of the supporting material cited in this Statement was provided previously in connection
with the earlier summary judgment briefing. It is tendered again, for the convenience of the
Court. In a number of instances, only deposition excerpts are provided.

B.

Creation of the Standards for Educational and Psychological Testing

5.

The Sponsoring Organizations have been preparing and publishing versions of the

Standards for over fifty years. In 1954, APA prepared and published the “Technical
Recommendations for Psychological Tests and Diagnostic Techniques” (Camara Decl., ¶ 7;
Ernesto Decl., ¶ 5). In 1955, AERA and NCME prepared and published a companion document
titled, “Technical Recommendations for Achievement Tests” (Levine Decl., ¶ 6; Camara Decl., ¶
7; Wise Decl., ¶ 5). Subsequently, a joint committee of the three organizations modified, revised,
and consolidated the two documents into the first Joint Standards. Beginning with the 1966
revision, the three organizations collaborated in developing the “Joint Standards” (or simply, the
“Standards”). Each subsequent revision of the Standards has been careful to note that it is a
revision and update of the prior version (Levine Decl., ¶ 6; Camara Decl., ¶ 7; Ernesto Decl., ¶ 6;
Wise Decl., ¶ 6).
6.

Specifically, beginning in the mid-1950s, Plaintiffs formed and periodically

reconstituted a committee of experienced experts in psychological and educational assessment,
charged with the initial development of the Technical Recommendations and then each
subsequent revision of the (renamed) Standards. These committees were formed by the
Plaintiffs’ presidents (or designees), who would meet and agree on the committees’ membership.
Beginning with the 1966 version of the Standards, this committee has been called the “Joint
Committee” (Levine Decl., ¶ 7; Camara Decl., ¶ 8; Ernesto Decl., ¶ 7; Wise Decl., ¶ 7).
7.

Financial and operational oversight for revising, promoting, distributing, and

selling the 1999 and 2014 Standards has been undertaken by a periodically reconstituted
“Management Committee,” comprised of designees of the three Plaintiffs (Levine Decl., ¶ 8;
Camara Decl., ¶ 9; Schneider Decl., ¶ 4; Ernesto Decl., ¶ 8; Wise Decl., ¶ 8). All members of the
Joint Committee(s) and the Management Committee(s) are unpaid volunteers. The expenses
2

associated with the ongoing development and publication of the Standards include travel and
lodging expenses (for the Joint Committee and Management Committee members), support staff
time, printing and shipment of bound volumes, and advertising costs (Levine Decl., ¶ 9; Camara
Decl., ¶ 10; Schneider Decl., ¶ 5; Ernesto Decl., ¶ 9; Wise Decl., ¶ 9).
8.

Many different fields of endeavor rely on assessments, and Plaintiffs seek to

ensure that the range of these fields of endeavor is represented in the Joint Committees’
membership — e.g., admissions, achievement, clinical counseling, educational, licensingcredentialing, employment, policy, and program evaluation. Similarly, the Joint Committee’s
members represent expertise across major functional assessment areas, including validity,
equating, reliability, test development, scoring, reporting, interpretation, and large scale
interpolation (Levine Decl., ¶ 10; Ernesto Decl., ¶ 10; Wise Decl., ¶ 10).
9.

AERA currently serves as publisher of the Standards, which are made available

for purchase through the AERA website. Levine Tr. 29, 55. AERA is a small organization, and a
purchaser may ordinarily contact some in person at AERA if they are having difficulty placing
an order. Levine Tr. 61-62. Sales are made to individual entities, libraries across the country,
college bookstores and academic institutions, with spikes in sales sometimes associate with the
academic calendar. See Levine Tr. 49-50. AERA sells the volumes for hard copy purchase, and
one can also purchase an electronic (read only) version of the 2014 Standards – or one can make
a combined purchase of both the electronic and hard copy for a discounted price. Levine Tr. 5254, 61, Exh. 1308, 1309.
10.

Plaintiffs promote and sell copies of the Standards via a variety of referrals to the

AERA website, at annual meetings, in public offerings to students, and to educational institution
faculty. Advertisements promoting the Standards have appeared in meeting brochures, in

3

scholarly journals, and in the hallways at professional meetings (Levine Decl., ¶ 14, Exh. NNN;
Ernesto Decl., ¶ 28, Exh. UU; Wise Decl., ¶ 21, Exh. KKK).
11.

Distribution of the Standards is monitored by the Sponsoring Organizations.

AERA, the now-designated publisher of the Standards, sometimes does provide promotional
complementary print copies to students or professors. Except for these few complementary print
copies, however, the Standards are not given away for free; they are not made available to the
public by any of the three organizations, or given to anyone to copy free of charge (Levine Decl.,
¶ 16; Ernesto Decl., ¶ 29; Wise Decl., ¶ 22). To date, Plaintiffs have never posted, or authorized
the posting of, a digitized copy of the 1999 Standards on any publicly accessible website (Levine
Decl., ¶ 16; Ernesto Decl., ¶ 30; Wise Decl., ¶ 23). Thus, the Sponsoring Organizations do not
make the Standards accessible (except for purchase) on-line in read only format. (C.f. American
Society for Testing and Materials, et al. v. Public.Resource.Org, Inc., 896 F.3d 437, 453 (2018)
(“the SDOs, by their own admission, make copies of their standards freely available online in
controlled reading rooms”)).

Doing so would logically reduce AERA’s ability to sell the

Standards, and thus reduce revenue for the overall enterprise of maintaining the Standards.
12.

The 1999 Standards have historically been sold at modest retail prices ranging

from $25.95 to $49.95 per copy. (Levine Decl., ¶ 17). Current pricing for members for either a
hard copy or e-book (single user) of the 2014 edition is $49.95, or as a bundle (hard copy and
single user e-book) at $59.95. The non-member prices for either a hard copy or single user ebook is $69.95, with $79.95 for the bundle. The 1999 edition is available in hardcopy at $35.95
for members and $45.95 for non-members, but the advertisement/offer of the 1999 Standards
expressly sets forth the caution that the volume has been superseded. Levine Tr. Exh. 1308,
1309.

4

13.

The 1999 Standards were first offered for sale in mid-1999. Sales in the year

2000 were estimated at 3797; for 2001, at 3755. They were 5592 in 2002; 3310 in 2003; 3218 in
in 2004; 3803 in 2005; 3888 in 2006; 3077 in 2007; 3358 in 2008; 2590 in 2009; 3043 in 2010;
2132 in 2011; 1649 in 2012; 1732 in 2013; and 855 in 2014. Levine Tr. Exh. 1207. Declining
sales 2011-2014 are coincident with, and might be attributed to, a combination of factors
including: PRO’s uploading of the 1999 Standards on-line beginning in May 2012 and running
through June 2014; saturation of parts of the market; or anticipation of the impending release of a
new edition, the 2014 Standards, which was introduced in the middle of 2014.
14.

After the 2014 version of the Standards was published in late summer of 2014,

AERA for a time discontinued sales of the 1999 Standards in order to encourage sales of the
newly-revised edition — the 2014 Standards (Levine Decl., ¶ 19, Exh. PPP). However, so long
as purchasers are aware that it is no longer the current edition – which is made clear on the
AERA website – the 1999 Standards continue to have value for those in the testing and
assessment profession who (i) need to know the state of best testing practices as they existed
between 1999 and 2014, (ii) believe they still may be held accountable to the guidance of the
1999 Standards, or (iii) study changes in best testing and assessment practices over time. In the
summer of 2015, AERA resumed sales of the 1999 Standards (Levine Decl., ¶ 20, Exh. QQQ).
15.

The production and updating of the Standards is financed solely through sales

revenues. All revenue from the sale of the 1999 Standards above expenses is used to cover the
publishing costs of the Standards and for the preparation of subsequent editions of the Standards.
The Sponsoring Organizations do not distribute any proceeds from the sales of the Standards to
the Sponsoring Organizations. Thus, generation of revenue allows the Sponsoring Organizations
to develop up-to-date, high quality Standards that otherwise would not be developed due to the

5

time and effort that goes into producing them (Levine Decl., ¶ 21; Geisinger Decl., ¶ 22; Camara
Decl., ¶ 19; Ernesto Decl., ¶ 31). Without income from the sales of the Standards to offset
production costs and to allow for further revisions, it is very likely that the Sponsoring
Organizations would no longer undertake to periodically update them (Levine Decl., ¶ 22;
Ernesto Decl., ¶ 32; Wise Decl., ¶ 24; Geisinger Decl., ¶ 22).
16.

Plaintiffs at one time considered soliciting funding for the revision process from

third party sources like governmental agencies, foundations, and other associations interested in
testing and assessment issues. But Plaintiffs rejected this option due to the difficulty of procuring
the funding as well as potential conflicts of interest that could arise from such a system. The
Sponsoring Organizations therefore concluded that revisions should be self-funding — that is,
from sale of prior editions of the Standards (Levine Decl., ¶ 23; Camara Decl., ¶ 20).
17.

The Sponsoring Organizations do not, so far as the record shows, market any

associated services for sale, such as seminars, as a way to generate revenue or profit through
collateral businesses or related products. (Compare 896 F.3d at 453 (“[C]an the SDOs continue
to make money on derivative good such that they have an adequate incentive to continue
producing the standards?”)). The revenue to finance the important work of updating and revising
the Standards as appropriate is generated solely through sales of the Standards.
18.

Due to the small membership size of Plaintiff NCME, and the relative minor

portion of the membership of Plaintiffs AERA and APA who devote their careers to testing and
assessment, it is highly unlikely that the members of the Sponsoring Organizations would vote
for a dues increase to fund future revisions if sales revenue is lost by virtue of an entity like PRO
allowing the Standards to be accessed on-line and downloaded or printed for free. As a result,

6

the Sponsoring Organizations would likely abandon their practice of periodically updating the
Standards (Levine Decl., ¶ 24; Camara Decl., ¶ 24; Geisinger Decl., ¶ 23; Ernesto Decl., ¶ 33).
19.
AERA.

As described above, the 1999 Standards are available for sale on-line through

However, the 1999 Standards have been widely sold to professionals, academic

institutions, libraries, professionals, students and academics. The 1999 Standards already in
circulation are, therefore, available for review and consultation through those outlets.
C.

Why the Standards Are Published and Their Benefits

20.

The Standards are published “to promote the sound and ethical use of tests and to

provide a basis for evaluating the quality of testing practices.” See 1999 Standards at 1 (quoted
by the Court of Appeals, 896 F.3d at 441). The Standards set forth principles and guidelines,
designed to provide a set of best practices to improve testing and assessment across multiple
settings, including education and various areas of psychology. The Standards can and should be
used in the sound and ethical development and use of tests, and also to evaluate the quality of
tests and testing practices (Geisinger Decl., ¶ 18; Camara Decl., ¶ 13; Wise Decl., ¶ 12).
21.

The Standards are not simply intended for Plaintiffs’ members. They are intended

for a broad audience that cuts across professions, backgrounds, and training. For example, they
can guide test developers, sponsors, publishers, and users by providing criteria for the evaluation
of tests, testing practices, and the effects of test use. Test user standards refer to those standards
that help test users decide how to choose certain tests, interpret scores, or make decisions based
on tests results. Test users include clinical or industrial psychologists, research directors, school
psychologists, counselors, employment supervisors, teachers, and various administrators who
select or interpret tests for their organizations. There is no mechanism, however, to enforce
compliance with the Standards on the part of the test developer or test user (Camara Decl., ¶ 14;
Wise Decl., ¶ 13; Geisinger Decl., ¶ 19; Ernesto Decl., ¶ 12).
7

22.

The Standards promote the development of high quality tests and the sound use of

results from such tests. Without such high quality standards, tests might produce scores that are
not defensible or accurate, not an adequate reflection of the characteristic they were intended to
measure, and not fair to the person tested. Thus, the Standards help ensure that measures of
student achievement are relevant, that admissions decisions are fair, that employment hiring and
professional credentialing results in qualified individuals being selected, and patients with
psychological needs are diagnosed properly and treated accordingly. Quality tests protect the
public from harmful decision-making and provide opportunities for education and employment
that are fair to all who seek them (Camara Decl., ¶ 15; Wise Decl., ¶ 14).
23.

The Standards apply broadly to a wide range of standardized vehicles and

procedures that sample an individual’s behavior, including tests, assessments, inventories, scales,
and other testing vehicles. The Standards apply equally to standardized multiple-choice tests,
performance assessments (including tests comprised of only open-ended essays), and hands-on
assessments or simulations. The main exceptions are that the Standards do not apply to
unstandardized questionnaires (e.g., unstructured behavioral checklists or observational forms),
teacher-made tests, and subjective decision processes (e.g., a teacher’s evaluation of students’
classroom participation over the course of a semester) (Camara Decl., ¶ 16; Wise Decl., ¶ 15;
Geisinger Decl., ¶ 20; Ernesto Decl., ¶ 13).
24.

The Standards have been used to develop testing guidelines for such activities as

college admissions, personnel selection, test translations, test user qualifications, and computerbased testing. The Standards also have been widely cited to address technical, professional, and
operational norms for all forms of assessments that are professionally developed and used in a
variety of settings. The Standards additionally provide a valuable public service to state and

8

federal governments that choose to use them. For instance, each testing company, when
submitting proposals for testing administration, instead of relying on a patchwork of local, or
even individual and proprietary, testing design and implementation criteria, may rely instead on
the Standards to afford the best guidance for testing and assessment practices (Camara Decl., ¶
17; Wise Decl., ¶ 16; Geisinger Decl., ¶ 21; Ernesto Decl.,¶ 14).
D.

The Standards Are For Professional, Not Governmental, Use.

25.

As described above, the Standards are developed and designed for professional

use. The Standards were not created or updated to address any governmental or regulatory need,
nor in response to any legislative action or judicial decision. However, the Standards have been
cited in judicial decisions related to the proper use and evidence for assessment, as well as by
state and federal legislators. These citations in judicial decisions and during legislative
deliberations occurred without any lobbying by the Plaintiffs (Levine Decl., ¶ 12; Camara Decl.,
¶ 18; Ernesto Decl., ¶ 15; Wise Decl., ¶ 17).
26.

During the first round of discovery in this case, APA located some

correspondence or draft correspondence relating to APA’s support for legislation proposed in
2001 by Senator Paul Wellstone (D-MN) on Fairness and Accuracy in High Stakes Educational
Decisions for Students — a suggested amendment to the Elementary and Secondary Education
Act (“No Child Left Behind Act”) 147 Cong. Rec. S. 4,644 (daily ed. May 9, 2001) (Ernesto
Decl., ¶¶ 16–22, Exhs. NN-SS). Some of these letters are unsigned and are not printed on APA
letterhead. Therefore, in accordance with APA practices, it is likely that the unsigned letters (not
printed on letterhead) were internal discussion drafts that were never sent (Ernesto Decl., ¶ 23).
27.

Regarding the signed letters printed on APA letterhead, they relate to Senator

Wellstone’s proposed legislation mandating that tests and assessments administered by the states
be of high quality and used appropriately for the benefit of test administrators and test takers.
9

These are goals that are consistent with APA policy as then reflected in the 1999 Standards.
Even though Senator Wellstone’s amendments sought, in part, to mandate States’ compliance
with the Standards, the Sponsoring Organizations had not actively advocated for this. In any
event, Senator Wellstone’s proposed amendment never became law (Ernesto Decl., ¶ 24, Exh.
TT).
28.

APA’s search of its records did not disclose any further communications with

Congress relating to the use of the Standards in legislation or proposed legislation, and, to the
best of APA’s or the other Sponsoring Organizations’ knowledge, the Standards have not been
referenced in legislation since 2001 (Ernesto Decl., ¶ 25). Moreover, neither AERA nor NCME
has ever communicated with Congress for the purpose of encouraging reference to the Standards
in law (Levine Decl., ¶¶ 12-13; Wise Decl., ¶ 18). None of the Plaintiffs has solicited any
government agency to incorporate the Standards into the Code of Federal Regulations or other
rules of federal or state agencies (Levine Decl., ¶ 13; Ernesto Decl., ¶ 26; Wise Decl., ¶ 19).
E.

Plaintiffs’ Development of the 1999 Edition of the Standards

29.

Preparing and revising the Standards entails intensive labor and considerable

cross-disciplinary expertise. Each time the Standards are revised, the Sponsoring Organizations
select and arrange for meetings of the Joint Committee, composed of leading authorities in
psychological and educational assessments. During these meetings, certain individual standards
are combined, pared down, or augmented, while others are deleted altogether, and some are
created as whole new individual standards.
30.

The 1999 Standards took more than five years to complete. It is the result of

work put in by the Joint Committee to generate a set of best practices on educational and
psychological testing that are respected and relied upon by leaders in their fields (Levine Decl., ¶
11; Camara Decl., ¶ 11; Wise Decl., ¶ 11). Draft revisions of the 1985 Standards, which became
10

the 1999 Standards, were widely distributed for public review and comment three times during
this revision effort to gauge whether the testing community believed the revised drafts to be
current and inclusive of the topics at issue (Schneider Decl., ¶ 6). The Joint Committee received
thousands of pages of comments and proposed text revisions from Plaintiffs’ members;
scientific, professional, trade, and advocacy groups; credentialing boards; state and federal
government agencies; test publishers and developers; and academic institutions. While the Joint
Committee reviewed and took under advisement these comments, the final language of the 1999
Standards was produced by the Joint Committee members (Camara Decl., ¶ 12; Schneider Decl.,
¶ 7). More than half the content of the 1999 Standards resulted from newly written prose of the
Joint Committee (Camara Decl., ¶ 12).
F.

Description of the 1999 Standards (Hutter Decl. Exhibit 1)

31.

Cover-to-cover, the 1999 Standards are 205 pages long. The vast bulk of those

205 pages is explanatory and background material and commentary, not the black-letter
standards themselves. For instance, the 1999 Standards include:


A front cover, a back cover, a cover page, and a page showing copyright and
publisher information;



A table of contents showing how the 1999 Standards are organized;



A preface providing a short history of the Standards and identifying the people
and organizations that participated in creating the 1999 Standards;



A five-plus page Introduction that gives an overview of the testing process;
introduces the 1999 Standards; offers some cautions to be exercised in using
the Standards; explains some updates specific to the 1999 edition of the

11

Standards; clarifies how the 1999 Standards use the term “construct”; and
elaborates on how the 1999 Standards are organized;


An index pointing users to the pages in the 1999 Standards that refer to
specific terms; and



A glossary defining how certain terms are specifically used in the 1999
Standards.

32.

The remaining 164 pages of the 1999 Standards consist of three substantive parts.

Part I is titled “Test Construction, Evaluation, and Documentation.” Part II is titled “Fairness in
Testing.” Part III is titled “Testing Applications.”
chapters.

Each of the three parts is broken up into

Part I, for example, contains chapters on validity; reliability and errors of

measurement; test development and revision; scaling, norming, and score comparability; test
administration, scoring, and reporting; and supporting documentation for tests.
33.

Each chapter in the 1999 Standards consists, in turn, of two overall sections. First

there is a section of background and explanatory material organized under various topical
subheadings. As the introduction to the 1999 Standards makes clear, while the section of
background material opening each chapter may be helpful to understanding the black-letter
standards, “it should not be interpreted as imposing additional standards.”
34.

The background material at the outset of each chapter is followed in each chapter

by a section that includes the individual, black-letter standards themselves. Each black-letter
standard is stated in bold-face type, one paragraph long, and is written in a prescriptive manner.
Each individual standard is followed by a Comment on which may help explain, elaborate on or
offer examples. The comments are designed to assist in applying the black-letter standards, but
they do not impose additional requirements beyond the black-letter standards. Even in the part

12

of each chapter that contains the standards (which is less than half the chapter), more words tend
to be devoted to the commentary than to the black letter standards themselves.
35.

The 1999 Standards, including all chapters, contains 272 standards in total. The

standards take up less than 15% of 1999 Standards (as determined by comparing the number of
lines of text devoted to standards in relation to the lines in the text overall). See Hutter Decl. ¶ 6.
G.

The Federal Practice of Incorporation by Reference.

36.

The National Technology Transfer and Advancement Act of 1995 (“NTTAA”)

requires federal agencies to use privately-developed standards to achieve federal objectives
whenever possible.2 Pub. L. No. 104-113 § 12, 110 Stat. 775, 782–83 (1996), codified at 15
U.S.C. § 272. Specifically it declares that “all Federal agencies and departments shall use
technical standards that are developed or adopted by voluntary consensus bodies, using such
technical standards as a means to carry out policy objectives or activities.” Id.
37.

One method through which federal agencies avail themselves of privately-

developed standards is by incorporating the standards by reference.

The Code of Federal

Regulations currently contains more than 23,000 incorporations by reference, though not all of
the material incorporated are standards developed by private standards-setting bodies. See

2

The NTTAA was the product of a study of the system of privately-developed consensus standards as it had
evolved in the United States, which emerged with the recommendation that government agencies continue to avail
itself of that system in aid of its own mission.
Specifically, in 1991 Congress enacted P.L. 102-245, requesting the National Research Council study
standards development. See National Research Council, Standards, Conformity Assessment, and Trade into the 21st
Century vii (National Academy Press 1995), available at http://www.nap.edu/read/4921/chapter/1. That study’s
overview of the U.S. standards-development system noted that many standards developers “offset expenses and
generate income through sales of standards documents, to which they hold the copyright. For many SDOs,
publishing is a significant source of operating revenue.” Id. at 32 (emphasis added). The study concluded that the
“U.S. standards development system serves the national interest well” by “support[ing] efficient and timely
development of product and process standards that meet economic and public interests.” Id. at 157. The study
recommended that Congress pass a law to promote the use of privately developed voluntary consensus standards by
federal agencies. Congress responded by enacting the NTTAA, codified at 15 U.S.C. § 272.

13

Bremer, Teaching Guide: Incorporation by Reference, 2019 Administrative Law Review at 321;
Bremer, Private Standards in Public Law, 63 Kansas L. Rev. 279.
38.

The Office of Management and Budget has explained that incorporation by

reference (i) saves the government the cost of developing standards on its own; (ii) provides
incentives to establish standards serving national needs; (iii) promotes efficiency and economic
competition through harmonized standards; and (iv) furthers the federal policy of relying on the
private sector to meet government needs for goods and services. OMB Circular No. A-119, 63
Fed. Reg. 8546 (Revised Feb. 10, 1998). See also Final Revision of OMB Circular A-119, 81
Fed. Reg. 4673 (Jan. 27, 2016).
39.

OMB has recognized that it is essential to not interfere with the ability of the

standard-setting organizations to charge for the use of their works: “If we required that all
materials IBR’d into the CFR be available for free, that requirement would compromise the
ability of regulators to rely on voluntary consensus standards, possibly requiring them to create
their own standards, which is contrary to the NTTAA and the OMB Circular A-119.” See
Incorporation by Reference, Announcement of Final Rule, Office of the Federal Register, 79
Fed. Reg. 66267, 66268 (Nov. 7, 2014), available at https://federalregister.gov/a/2014-26445.
40.

The practice of incorporation by reference is currently codified at 5 U.S.C.

§552(a)(1), which clarifies that no material incorporated by reference can be deemed binding –
in the sense of adversely affecting any person – unless the material is reasonably available to the
class of affected persons. Thus, any standard incorporated by reference is self-limiting in its
application if not “reasonably available.”
Except to the extent that a person has actual and timely notice of
the terms thereof, a person may not in any manner be required to
resort to, or be adversely affected by, a matter required to be
published in the Federal Register and not so published. For
14

purposes of this paragraph, matter reasonably available to the
class of persons affected thereby is deemed published in the
Federal Register when incorporated by reference therein with the
approval of the Director of the Federal Register.
41.

The Office of Federal Register in turn has promulgated regulations governing the

process for agencies to follow in order to obtain approval to incorporate materials by reference in
the CFR. 1 C.F.R. §51 (2014). Those requirements interpret and implement the statutory
mandate that, in order for matters set forth in the register to be binding, they must be “reasonably
available to the class of persons affected.” 5 U.S.C. § 552(a)(1); 1 C.F.R. § 51.7(a)(3). They
require that (i) a copy of the incorporated material must be on file with the Office of the Federal
Register and (ii) that the regulations incorporating such material must state the ways those
incorporated materials are reasonably available to interested parties. 1 C.F.R. §§ 51.3, 51.5.
There is no requirement that such materials be available to the public at no cost, or on the
internet
H.

DOE’s Incorporation By Reference of Some Parts of the 1999 Standards

42.

Plaintiffs learned primarily during the course of these proceedings that the

Department of Education has incorporated certain standards set forth in the 1999 Standards in
three sections of the Code of Federal Regulations. As seen below, none of these provisions
apply to conventional primary conduct or purport to apply mandatory enforceable rules with
penalties attached.

Rather, they address how tests may qualify for purposes relevant to

Department of Education grant programs.
43.

34 C.F.R. §668.146(b)(6) is the only regulation that was cited by PRO as the basis

for its incorporation-by-reference rationale for posting the 1999 Standards. That regulation is
part of a subpart of regulations setting forth provisions under which a student without a high
school diploma or its recognized equivalent may become eligible to receive funds under Title IV

15

of the Higher Education Act. The student may become eligible by obtaining a diploma or its
equivalent. Another way for a student to become eligible for Title IV funds is to pass a test that
satisfies certain criteria including “all standards for test construction provided in the 1999 edition
of the Standards for Educational and Psychological Testing, prepared by a joint committee of the
American Educational Research Association, the American Psychological Association, and the
National Council on Measurement in Education incorporated by reference in this section.” The
regulation then goes on to note that the incorporation by reference was properly approved,
identifies where the material referenced (the 1999 Standards) is on file, and provides a phone
number and website for the National Archives and Records Administration “[f]or information on
the availability of this material at NARA.” The regulation also identifies the AERA website,
noting that the document may also be obtained from there.
44.

The individual standards incorporated by reference in 34 C.F.R. 668.146(b)(6) are

set forth in Part I of the 1999 Standards, entitled “Test Construction, Evaluation and
Documentation.” Part I actually consists of six chapters, covering Validity, Reliability and
Errors of Measurement, Test Development and Revision, Scales, Norms and Score
Comparability, Test Administration, Scoring, and Reporting, and Supporting Documentation for
Tests. Even assuming that all six chapters of Part I are being referenced by the regulation, the
actual bold-face “standards” set forth in these chapters are very limited.
45.

For example, the chapter within Part I on “validity” runs from pp. 9-24, begins

with background and explanatory material, and contains standards only on pages 17-24
(Standards 1.1-1.24), including comments on each standard. The comments on each standard are
typically as long as, or longer than the standards themselves.

16

46.

The chapter within Part I on Reliability and Errors is eleven pages long and

begins with 6 pages of background and discussion of various topics and considerations.
Standards (2.1-2.20) are set forth on 6 pages, and the comments to the standards are again longer
than the referenced standards themselves.
47.

The chapter on Test Development and Revision runs from pages 37-48 and the

first 6 pages describe background and discussions of various related topics and considerations.
The actual standards are set forth within pages 43-48 (standards 3.1- 3.27), and again, most of
that appears to be commentary, rather than the individual standards themselves (though some of
the standards have no immediately following “comment” and some comments are as brief as the
standard).
48.

The next three chapters in Part I follow a similar format.

49.

34 C.F.R. §668.148(a)(1)(iv) is also a part of the subpart of the regulations setting

forth provisions under which a student without a high school diploma or its recognized
equivalent may become eligible to receive funds under Title IV. This provision covers tests
developed for non-native speakers of English enrolled in a program taught in their native
language. Such tests must, according to this provision, be “[d]eveloped in accordance with
guidelines provided in the “’Testing Individuals of Diverse Linguistic Backgrounds’ section of
the” 1999 Standards.
50.

That chapter of the 1999 Standards runs from pages 91 to 100, with the first 6

pages covering background and explanatory topics. The actual standards are found at pages 97
to 100, and again, the actual standards themselves appear to occupy less than half of those pages,
with comments taking up more space.

17

51.

34 C.F.R. §462.13(c)(1) and (f)(1) are part of the regulations establishing a

process for reviewing the suitability of tests, as submitted by test publishers, for use in the
National Reporting System for Adult Education, which is an outcome-based accountability
system for evaluating state-administered, federally-funded adult education programs. Among the
criteria used to determine whether a test is suitable for use in the that system is that “[t]he test
must meet all applicable and feasible standards for test construction and validity provided in the
1999 edition of the Standards[.]” But a test or publisher that seeks to qualify under this program
is specifically offered the opportunity to demonstrate why a particular standard is not feasible or
applicable.
52.

This regulation, like 34 C.F.R. §668.146(b)(6), references Part I of the 1999

Standards. The regulation itself contains similar information and directions as to how and where
to find the incorporated standards.
53.

In addition, subsection (f)(1) provides that “For a test that has been modified for

individuals with disabilities, the Error! Hyperlink reference not valid.must,” among other
things, “[p]rovide documentation that it followed the guidelines provided in the Testing
Individuals With Disabilities section of the 1999 edition of the Standards for Educational and
Psychological Testing.” That is reference to a different chapter of the 1999 Standards, contained
in Part 2.

That chapter comprises 8 pages, the first 5 of which provide background and

explanatory material, which is followed by 3 pages containing the black letter standards (10.110.12), many of which are followed a Comment.
54.

Less than 6 per cent of the 1999 Standards is occupied by the standards

incorporated by reference in the three regulation (as determined by comparing the number of

18

lines of text in the referenced standards with the number of lines in the text overall). Hutter
Decl. ¶ 6.
55.

Plaintiffs are not aware that DOE has ever had occasion to enforce the provisions

of its regulations referencing the 1999 Standards.
56.

In addition, Plaintiffs are not specifically aware of tests developed specifically to

meet these DOE regulations. Rather, tests meeting these criteria may have been developed
entirely independent of the regulations because high quality tests generally would, as a matter of
basic professionalism, be developed in a manner consistent with the Standards, but without
regard to the fact that some of the standards are cited in DOE regulations. The exception could,
of course, be for 34 C.F.R. §462.13 (f)(1), which requires the test publisher to provide
documentation that it affirmatively meets the requirements of the referenced portion of the 1999
Standards. But plaintiffs are not in possession of any information regarding how or whether that
provision has been applied.
57.

Copies of the 1999 Standards are available, as identified in the CFR, in the

reading room and from AERA, either by ordering on-line or over the telephone. In addition,
existing copies of the 1999 Standards are available and in circulation throughout academia
(including schools of education and departments of psychology), are in the possession of
organizations involved in testing, can be accessed in public and university libraries, and would
expectably be accessible and referenced in connection with the development and use of tests
irrespective of whether at some point the test would be in any way subject to the DOE
regulations.

19

I.

Defendants Copied the 1999 Standards In Toto and Verbatim, and Enabled
Others to Copy Them, Without Plaintiffs’ Permission

58.

Defendant Public.Resource.Org (PRO) is a California non-profit corporation

founded in 2007 by Mr. Carl Malamud, with the explicit aim of making government information
more accessible, with particular emphasis on the law (Hudis Decl., ¶ 2, Exh. A, pp. 77, 93–94,
163–164). The identified purpose and objective of PRO is to create and maintain so-called
informational “public works projects for the Internet” (Hudis Decl., ¶ 2, Exh. A, pp. 94–95, 105–
09, ¶ 3, Exh. B, Section II.B., ¶ 4, Exh. C. Section 2.1).
59.

PRO creates digital copies of standards referenced or incorporated in regulations,

without the permission of the copyright owners. It does so as a matter of principle. That is, PRO
does not purport to identify any specific need or reason for such copying. Rather, PRO simply
believes, and acts on the belief, that all such material should copied and made available for
copying on the internet. Its mission includes the broad and indiscriminate objective “to make the
law and other government materials more widely available.”

896 F.3d at 444 (quoting

Malamud’s Declaration in this case).
60.

In March 2012, PRO began copying standards incorporated by reference into the

Code of Federal Regulations, and facilitating copying by others. In May 2012, PRO began the
process of posting copies these standards to its website. “Between 2012 and 2014, PRO uploaded
hundreds of technical standards, which, collectively, were downloaded tens of thousands of
times.” Id.
61.

Specifically, on May 17, 2012, PRO bought a used hard copy of the 1999

Standards from an Amazon re-seller (Hudis Decl., ¶ 2, Exh. A, pp. 232-240, ¶ 21, Exh. T, Int.
Ans. 1, ¶¶ 22-23, Exh. U). Upon receipt of the purchased paper copy, Malamud disassembled
the book, removed the spine, trimmed the pages to give them an even border, scanned the pages
20

to create a PDF, and named the PDF file “aera.standards.1999.pdf.” Malamud then appended a
cover sheet, a self-made “Certificate,” to the front of the PDF file giving a false semblance of
governmental imprimatur to the unauthorized copying and online posting of the 1999 Standards
(Hudis Decl., ¶ 2, Exh. A, pp. 257–59, 261–64, ¶ 21, Exh. T, Int. Ans. 3–4, ¶ 26. Malamud did
nothing else to modify or transform the PDF file or the Standards. He simply copied and posted
the 1999 Standards, in its entirety, to his company’s website. He did not provide for wordsearching, online identification, or text-to-speech utilization for the blind and visually impaired
(Hudis Decl., ¶ 27, Exh. Z, pp. 30, 122, 200–01, 206, 271–72, 315–16). PRO published the
infringing digital copy of the 1999 Standards on a website titled https://law.resource.org.
62.

The Internet Archive is a nonprofit organization whose mission is to build and

maintain a digital library of the Internet. The Internet Archive builds this internet library —
which it makes available for public use — by scanning, digitally capturing, and saving
electronically scanned and captured third-party websites, and by receiving submissions from
third parties who have user accounts enabling them to upload content (Hudis Decl., ¶ 29, Exh.
BB, pp. 31–41). Malamud has such user account access (Hudis Decl., ¶ 29, Exh. BB, pp. 51–
56), and he uploaded the entirety of the 1999 Standards to the Internet Archive’s website on May
26–27, 2012 (Hudis Decl., ¶ 29, Exh. BB, pp. 59-112, ¶ 30, Exh. CC (¶¶ 3–18 therein), ¶ 32,
Exh. EE, ¶ 33). PRO posted Plaintiffs’ 1999 Standards to its website and the Internet Archive
website without the permission or authorization of any of the Sponsoring Organizations (Hudis
Decl., ¶ 35, Exh. HH, Admission Nos. 4–5; Levine Decl., ¶ 29; Ernesto Decl., ¶ 35; Wise Decl.,
¶ 26).
63.

The copy of the Sponsoring Organizations’ 1999 Standards that Malamud

published to the Internet Archive at https://archive.org/details/gov.law.aera.standards.1999 was

21

in the same format, using the same cover sheet or “Certificate” employed by PRO in the posting
of the Sponsoring Organizations’ 1999 Standards to Defendant’s own website. All of the
surrounding text associated with the posting to the Internet Archive website was inserted by
Malamud — including the insertion of “Creative Commons License: CC0 1.0 Universal,”
indicating that no rights are being asserted over the item (Hudis Decl., ¶ 2, Exh. A, pp. 275–84, ¶
29, Exh. BB, pp. 57–63, ¶ 30, Exh. CC (¶ 2 therein), ¶ 34, Exh. GG):
64.

Although based on incomplete reporting (Hudis Decl., ¶ 2, Exh. A, pp. 272–74,

328–36), during the near two-year period that the Sponsoring Organizations’ 1999 Standards
were first posted on PRO’s https://law.resource.org website, they were accessed at least 4,164
times (Hudis Decl., ¶ 21, Exh. T, Int. Ans. 2 and Amended Ans. 5 (labeled 6)). During that same
period, the Sponsoring Organizations’ 1999 Standards were accessed on the Internet Archive
https://archive.org website 1,290 times (Hudis Decl., ¶ 29, Exh. BB, pp. 124-132, ¶ 37, Exh. II).
65.

The Internet Archive’s website is open to the public and does not restrict an

Internet user’s ability to download or print the Sponsoring Organizations’ 1999 Standards. PRO
also placed no such restrictions on its website (Hudis Decl., ¶ 2, Exh. A, pp. 347–48). There
were no sign-up procedures to enter PRO’s https://law.resource.org website, nor was there any
Digital Rights Management (or “DRM”) plan to protect against, or identify, further copying of
the files accessed from PRO’s site (Hudis Decl., ¶ 27, Exh. Z, pp. 324–27, 167–73).
66.

There is no way for Plaintiffs to calculate the number of university/college

professors, students, testing companies and others who would have purchased Plaintiffs’
Standards but for their wholesale posting on PRO’s https://law.resource.org website and the
Internet Archive http://archive.org website (Levine Decl., ¶ 30; Geisinger Decl., ¶ 24).

22

67.

In late 2013 and early 2014, the Sponsoring Organizations became aware that the

1999 Standards had been posted on the Internet without their authorization, and that students
were obtaining free copies from the posting source. Upon further investigation, they discovered
that PRO was the source of the online posting (Camara Decl., ¶ 21, Exh. MMM; Wise Decl., ¶¶
27-28, Exh. LLL).
68.

In December 2013, Plaintiff AERA requested in writing that PRO remove the

1999 Standards from its online postings (Levine Decl., ¶ 31, Exh. UUU). Defendant refused
(Hudis Decl., ¶ 2, Exh. A, pp. 310–19, ¶ 38, Exh. JJ, ¶ 39, Exh. KK). Once this lawsuit was filed
and the Sponsoring Organizations threatened to file a motion for a preliminary injunction, PRO
agreed in June 2014 to remove its postings of the 1999 Standards from its
https://law.resource.org website and from Internet Archive’s https://archive.org website, pending
a resolution of this litigation on the merits. PRO’s undertaking included the promise not to post
any revision of the 1999 Standards (i.e., the 2014 Standards) pending the outcome of this
litigation on the merits (Hudis Decl., ¶ 2, Exh. A, pp. 322-28, ¶ 40, Exh. LL, ¶ 41, Exh. MM).
69.

Notwithstanding that undertaking, it is currently posted on the Internet Archive

website, stating that it was uploaded on May 26, 2012. That posting includes the full volume of
the 1999 Standards, verbatim, covered with the self-created “Certificate” stating “By Authority
of the United States of America … Legally Binding Document,” and stating that the document is
posted by Public.Resource.Org. Inc. The website cites a single section of the CFR, 34 C.F.R.
§668.148(a)(2)(iv).

The website posting provides a variety of means by which the 1999

Standards can be downloaded. See Hutter Decl. Exh 3. The website reflected that as of
September

11,

2019,

it

was

viewed

https://archive.org/details/gov.law.aera.standards.1999/page/n1

23

1445

times.

See

J.

This Litigation, and the Court of Appeals’ Remand Direction

70.

In 2017, this Court granted Plaintiffs’ motion for summary judgment on their

copyright claim. The Court held that Plaintiffs own the copyright to the 1999 Standards; that
Plaintiffs’ copyright is valid; that nothing about the 1999 Standards’ incorporation into federal
law negated the copyright; that PRO’s posting on the internet was infringing; and that PRO’s
wholesale copying and posting of the entirety of the 1999 Standards to the internet, from where it
could be downloaded, was not fair use. Regarding the fair use issue, this Court’s decision
recognized that PRO had the burden on that issue, and the Court addressed the specific
arguments that PRO had made – arguments that PRO advanced across the board, without
addressing the differences between the many and varied standards at issue in the ASTM case, or
the differences between how those standards were referenced in federal regulations. Based on
those holdings, the Court enjoined PRO from continuing to post the 1999 Standards.
71.

In 2018, the Court of Appeals for the District of Columbia Circuit vacated this

Court’s summary judgment and injunction order and remanded the case for further proceedings.
The Court of Appeals’ remand order was limited to a single issue: fair use. The Court of
Appeals noted that PRO had offered an “undifferentiated” theory of why its publication of any
standards incorporated by reference in federal regulations qualified as fair use (or should be
denied copyright protection at all). This Court had therefore addressed those arguments of PRO
in the undifferentiated terms under which they were presented.
72.

But the Court of Appeals believed that approach “failed to account for the

variation among the standards at issue and … failed to consider each fair use claim ‘on its own
facts.’” Thus, it noted the many and varied standards and incorporations by reference at issue (at
least in the ASTM case), and directed that:

24

On remand, the district court will need to develop a fuller record
regarding the nature of each of the standards at issue, the way in
which they are incorporated, and the manner and extent to which
they were copied by PRO in order to resolve this “mixed question
of law and fact.”
73.

In light of that direction, this Court allowed PRO an extended period to take

additional discovery, which it did. Plaintiffs did not take any additional discovery.
Respectfully submitted,
s/ Clifton S. Elgarten
John I. Stewart, Jr. (D.C. Bar No. 913905)
Clifton S. Elgarten (D.C. Bar No. 366898)
Amanda Shafer Berman (D.C. Bar No.
497860)
CROWELL & MORING LLP
1001 Pennsylvania Avenue, NW
Washington, DC 20004-2595
[phone]
[e-mail]
Attorneys for Plaintiff-Counterdefendants
October 4, 2019

25