Filing 134

MOTION for Summary Judgment , MOTION for Permanent Injunction by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 Memorandum in Support, # 2 Statement of Facts, # 3 Declaration Hutter Decl., # 4 Exhibit 1, # 5 Exhibit 2, # 6 Exhibit 3, # 7 Exhibit 4, # 8 Declaration Hudis Decl., # 9 Exhibit A, # 10 Exhibit B, # 11 Exhibit C, # 12 Exhibit T, # 13 Exhibit U, # 14 Exhibit Z, # 15 Exhibit BB, # 16 Exhibit CC, # 17 Exhibit EE, # 18 Exhibit GG, # 19 Exhibit HH, # 20 Exhibit II, # 21 Exhibit JJ, # 22 Exhibit KK, # 23 Exhibit LL, # 24 Exhibit MM, # 25 Declaration Ernesto Decl., # 26 Exhibit NN, # 27 Exhibit OO, # 28 Exhibit PP, # 29 Exhibit QQ, # 30 Exhibit RR, # 31 Exhibit SS, # 32 Exhibit TT, # 33 Exhibit UU, # 34 Declaration Wise Decl., # 35 Exhibit KKK, # 36 Exhibit LLL, # 37 Declaration Camara Decl., # 38 Exhibit MMM, # 39 Declaration Levine Decl., # 40 Exhibit NNN, # 41 Exhibit PPP, # 42 Exhibit QQQ, # 43 Exhibit UUU, # 44 Declaration Geisinger Decl., # 45 Declaration Schneider Decl., # 46 Exhibit Levine Depo Tr., # 47 Exhibit No. 1207 to Levine Depo Tr., # 48 Exhibit No. 1308 to Levine Depo Tr., # 49 No. 1308 to Levine Depo Tr., # 50 Text of Proposed Order)(Elgarten, Clifton)

Download PDF
IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLUMBIA AMERICAN EDUCATION RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., and NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. Plaintiff-Counterdefendants, v. PUBLIC RESOURCE ORG., INC., Defendant-Counterclaimant. ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) Case No. 1:14-cv-00857 (TSC) STATEMENT OF MATERIAL FACTS John I. Stewart, Jr. (D.C. Bar No. 913905) Clifton S. Elgarten (D.C. Bar No. 366898) Amanda Shafer Berman (D.C. Bar No. 497860) CROWELL & MORING LLP 1001 Pennsylvania Avenue, NW Washington, DC 20004-2595 (202) 624-2500 Attorneys for Plaintiff-Counterdefendants TABLE OF CONTENTS Page STATEMENT OF MATERIAL FACTS ...................................................................................................... 1  A.  The Plaintiff Sponsoring Organizations.............................................................................. 1  B.  Creation of the Standards for Educational and Psychological Testing ............................... 2  C.  Why the Standards Are Published and Their Benefits ........................................................ 7  D.  The Standards Are For Professional, Not Governmental, Use. .......................................... 9  E.  Plaintiffs’ Development of the 1999 Edition of the Standards ......................................... 10  F.  Description of the 1999 Standards (Hutter Decl. Exhibit 1) ............................................. 11  G.  The Federal Practice of Incorporation by Reference. ....................................................... 13  H.  DOE’s Incorporation By Reference of Some Parts of the 1999 Standards....................... 15  I.  Defendants Copied the 1999 Standards In Toto and Verbatim, and Enabled Others to Copy Them, Without Plaintiffs’ Permission ..................................................... 20  J.  This Litigation, and the Court of Appeals’ Remand Direction ......................................... 24  STATEMENT OF MATERIAL FACTS1 A. The Plaintiff Sponsoring Organizations 1. Plaintiffs, the Sponsoring Organizations, are each District of Columbia not-for- profit corporations (Levine Decl., ¶ 4; Ernesto Decl., ¶ 3; Wise Decl., ¶ 3). Each is an active professional organization that engages in many activities for many purposes other than to create the Standards for Educational and Psychological Testing at issue in this case. 2. American Education Resource Association (AERA) is the major national scientific society for research on education and learning. AERA’s mission is to advance knowledge about education, to encourage scholarly inquiry related to education, and to promote the use of research to improve education and serve the public good (Levine Decl., ¶ 5). 3. The American Psychological Association (APA) is the largest scientific and professional organization representing psychology in the United States. APA is the world’s largest association of psychologists and counts a vast number of researchers, educators, clinicians, consultants and students among its members. APA’s mission is to advance the creation, communication, and application of psychological knowledge to benefit society and improve people’s lives (Ernesto Decl., ¶ 4). 4. The National Council on Measurement in Education (NCME) is a professional organization for individuals involved in assessment, evaluation, testing, and other aspects of educational measurement. NCME’s members are involved in the construction and use of standardized tests; new forms of assessment, including performance- based assessment; program design; and program evaluation (Wise Decl., ¶ 4). 1 Much of the supporting material cited in this Statement was provided previously in connection with the earlier summary judgment briefing. It is tendered again, for the convenience of the Court. In a number of instances, only deposition excerpts are provided. B. Creation of the Standards for Educational and Psychological Testing 5. The Sponsoring Organizations have been preparing and publishing versions of the Standards for over fifty years. In 1954, APA prepared and published the “Technical Recommendations for Psychological Tests and Diagnostic Techniques” (Camara Decl., ¶ 7; Ernesto Decl., ¶ 5). In 1955, AERA and NCME prepared and published a companion document titled, “Technical Recommendations for Achievement Tests” (Levine Decl., ¶ 6; Camara Decl., ¶ 7; Wise Decl., ¶ 5). Subsequently, a joint committee of the three organizations modified, revised, and consolidated the two documents into the first Joint Standards. Beginning with the 1966 revision, the three organizations collaborated in developing the “Joint Standards” (or simply, the “Standards”). Each subsequent revision of the Standards has been careful to note that it is a revision and update of the prior version (Levine Decl., ¶ 6; Camara Decl., ¶ 7; Ernesto Decl., ¶ 6; Wise Decl., ¶ 6). 6. Specifically, beginning in the mid-1950s, Plaintiffs formed and periodically reconstituted a committee of experienced experts in psychological and educational assessment, charged with the initial development of the Technical Recommendations and then each subsequent revision of the (renamed) Standards. These committees were formed by the Plaintiffs’ presidents (or designees), who would meet and agree on the committees’ membership. Beginning with the 1966 version of the Standards, this committee has been called the “Joint Committee” (Levine Decl., ¶ 7; Camara Decl., ¶ 8; Ernesto Decl., ¶ 7; Wise Decl., ¶ 7). 7. Financial and operational oversight for revising, promoting, distributing, and selling the 1999 and 2014 Standards has been undertaken by a periodically reconstituted “Management Committee,” comprised of designees of the three Plaintiffs (Levine Decl., ¶ 8; Camara Decl., ¶ 9; Schneider Decl., ¶ 4; Ernesto Decl., ¶ 8; Wise Decl., ¶ 8). All members of the Joint Committee(s) and the Management Committee(s) are unpaid volunteers. The expenses 2 associated with the ongoing development and publication of the Standards include travel and lodging expenses (for the Joint Committee and Management Committee members), support staff time, printing and shipment of bound volumes, and advertising costs (Levine Decl., ¶ 9; Camara Decl., ¶ 10; Schneider Decl., ¶ 5; Ernesto Decl., ¶ 9; Wise Decl., ¶ 9). 8. Many different fields of endeavor rely on assessments, and Plaintiffs seek to ensure that the range of these fields of endeavor is represented in the Joint Committees’ membership — e.g., admissions, achievement, clinical counseling, educational, licensingcredentialing, employment, policy, and program evaluation. Similarly, the Joint Committee’s members represent expertise across major functional assessment areas, including validity, equating, reliability, test development, scoring, reporting, interpretation, and large scale interpolation (Levine Decl., ¶ 10; Ernesto Decl., ¶ 10; Wise Decl., ¶ 10). 9. AERA currently serves as publisher of the Standards, which are made available for purchase through the AERA website. Levine Tr. 29, 55. AERA is a small organization, and a purchaser may ordinarily contact some in person at AERA if they are having difficulty placing an order. Levine Tr. 61-62. Sales are made to individual entities, libraries across the country, college bookstores and academic institutions, with spikes in sales sometimes associate with the academic calendar. See Levine Tr. 49-50. AERA sells the volumes for hard copy purchase, and one can also purchase an electronic (read only) version of the 2014 Standards – or one can make a combined purchase of both the electronic and hard copy for a discounted price. Levine Tr. 5254, 61, Exh. 1308, 1309. 10. Plaintiffs promote and sell copies of the Standards via a variety of referrals to the AERA website, at annual meetings, in public offerings to students, and to educational institution faculty. Advertisements promoting the Standards have appeared in meeting brochures, in 3 scholarly journals, and in the hallways at professional meetings (Levine Decl., ¶ 14, Exh. NNN; Ernesto Decl., ¶ 28, Exh. UU; Wise Decl., ¶ 21, Exh. KKK). 11. Distribution of the Standards is monitored by the Sponsoring Organizations. AERA, the now-designated publisher of the Standards, sometimes does provide promotional complementary print copies to students or professors. Except for these few complementary print copies, however, the Standards are not given away for free; they are not made available to the public by any of the three organizations, or given to anyone to copy free of charge (Levine Decl., ¶ 16; Ernesto Decl., ¶ 29; Wise Decl., ¶ 22). To date, Plaintiffs have never posted, or authorized the posting of, a digitized copy of the 1999 Standards on any publicly accessible website (Levine Decl., ¶ 16; Ernesto Decl., ¶ 30; Wise Decl., ¶ 23). Thus, the Sponsoring Organizations do not make the Standards accessible (except for purchase) on-line in read only format. (C.f. American Society for Testing and Materials, et al. v. Public.Resource.Org, Inc., 896 F.3d 437, 453 (2018) (“the SDOs, by their own admission, make copies of their standards freely available online in controlled reading rooms”)). Doing so would logically reduce AERA’s ability to sell the Standards, and thus reduce revenue for the overall enterprise of maintaining the Standards. 12. The 1999 Standards have historically been sold at modest retail prices ranging from $25.95 to $49.95 per copy. (Levine Decl., ¶ 17). Current pricing for members for either a hard copy or e-book (single user) of the 2014 edition is $49.95, or as a bundle (hard copy and single user e-book) at $59.95. The non-member prices for either a hard copy or single user ebook is $69.95, with $79.95 for the bundle. The 1999 edition is available in hardcopy at $35.95 for members and $45.95 for non-members, but the advertisement/offer of the 1999 Standards expressly sets forth the caution that the volume has been superseded. Levine Tr. Exh. 1308, 1309. 4 13. The 1999 Standards were first offered for sale in mid-1999. Sales in the year 2000 were estimated at 3797; for 2001, at 3755. They were 5592 in 2002; 3310 in 2003; 3218 in in 2004; 3803 in 2005; 3888 in 2006; 3077 in 2007; 3358 in 2008; 2590 in 2009; 3043 in 2010; 2132 in 2011; 1649 in 2012; 1732 in 2013; and 855 in 2014. Levine Tr. Exh. 1207. Declining sales 2011-2014 are coincident with, and might be attributed to, a combination of factors including: PRO’s uploading of the 1999 Standards on-line beginning in May 2012 and running through June 2014; saturation of parts of the market; or anticipation of the impending release of a new edition, the 2014 Standards, which was introduced in the middle of 2014. 14. After the 2014 version of the Standards was published in late summer of 2014, AERA for a time discontinued sales of the 1999 Standards in order to encourage sales of the newly-revised edition — the 2014 Standards (Levine Decl., ¶ 19, Exh. PPP). However, so long as purchasers are aware that it is no longer the current edition – which is made clear on the AERA website – the 1999 Standards continue to have value for those in the testing and assessment profession who (i) need to know the state of best testing practices as they existed between 1999 and 2014, (ii) believe they still may be held accountable to the guidance of the 1999 Standards, or (iii) study changes in best testing and assessment practices over time. In the summer of 2015, AERA resumed sales of the 1999 Standards (Levine Decl., ¶ 20, Exh. QQQ). 15. The production and updating of the Standards is financed solely through sales revenues. All revenue from the sale of the 1999 Standards above expenses is used to cover the publishing costs of the Standards and for the preparation of subsequent editions of the Standards. The Sponsoring Organizations do not distribute any proceeds from the sales of the Standards to the Sponsoring Organizations. Thus, generation of revenue allows the Sponsoring Organizations to develop up-to-date, high quality Standards that otherwise would not be developed due to the 5 time and effort that goes into producing them (Levine Decl., ¶ 21; Geisinger Decl., ¶ 22; Camara Decl., ¶ 19; Ernesto Decl., ¶ 31). Without income from the sales of the Standards to offset production costs and to allow for further revisions, it is very likely that the Sponsoring Organizations would no longer undertake to periodically update them (Levine Decl., ¶ 22; Ernesto Decl., ¶ 32; Wise Decl., ¶ 24; Geisinger Decl., ¶ 22). 16. Plaintiffs at one time considered soliciting funding for the revision process from third party sources like governmental agencies, foundations, and other associations interested in testing and assessment issues. But Plaintiffs rejected this option due to the difficulty of procuring the funding as well as potential conflicts of interest that could arise from such a system. The Sponsoring Organizations therefore concluded that revisions should be self-funding — that is, from sale of prior editions of the Standards (Levine Decl., ¶ 23; Camara Decl., ¶ 20). 17. The Sponsoring Organizations do not, so far as the record shows, market any associated services for sale, such as seminars, as a way to generate revenue or profit through collateral businesses or related products. (Compare 896 F.3d at 453 (“[C]an the SDOs continue to make money on derivative good such that they have an adequate incentive to continue producing the standards?”)). The revenue to finance the important work of updating and revising the Standards as appropriate is generated solely through sales of the Standards. 18. Due to the small membership size of Plaintiff NCME, and the relative minor portion of the membership of Plaintiffs AERA and APA who devote their careers to testing and assessment, it is highly unlikely that the members of the Sponsoring Organizations would vote for a dues increase to fund future revisions if sales revenue is lost by virtue of an entity like PRO allowing the Standards to be accessed on-line and downloaded or printed for free. As a result, 6 the Sponsoring Organizations would likely abandon their practice of periodically updating the Standards (Levine Decl., ¶ 24; Camara Decl., ¶ 24; Geisinger Decl., ¶ 23; Ernesto Decl., ¶ 33). 19. AERA. As described above, the 1999 Standards are available for sale on-line through However, the 1999 Standards have been widely sold to professionals, academic institutions, libraries, professionals, students and academics. The 1999 Standards already in circulation are, therefore, available for review and consultation through those outlets. C. Why the Standards Are Published and Their Benefits 20. The Standards are published “to promote the sound and ethical use of tests and to provide a basis for evaluating the quality of testing practices.” See 1999 Standards at 1 (quoted by the Court of Appeals, 896 F.3d at 441). The Standards set forth principles and guidelines, designed to provide a set of best practices to improve testing and assessment across multiple settings, including education and various areas of psychology. The Standards can and should be used in the sound and ethical development and use of tests, and also to evaluate the quality of tests and testing practices (Geisinger Decl., ¶ 18; Camara Decl., ¶ 13; Wise Decl., ¶ 12). 21. The Standards are not simply intended for Plaintiffs’ members. They are intended for a broad audience that cuts across professions, backgrounds, and training. For example, they can guide test developers, sponsors, publishers, and users by providing criteria for the evaluation of tests, testing practices, and the effects of test use. Test user standards refer to those standards that help test users decide how to choose certain tests, interpret scores, or make decisions based on tests results. Test users include clinical or industrial psychologists, research directors, school psychologists, counselors, employment supervisors, teachers, and various administrators who select or interpret tests for their organizations. There is no mechanism, however, to enforce compliance with the Standards on the part of the test developer or test user (Camara Decl., ¶ 14; Wise Decl., ¶ 13; Geisinger Decl., ¶ 19; Ernesto Decl., ¶ 12). 7 22. The Standards promote the development of high quality tests and the sound use of results from such tests. Without such high quality standards, tests might produce scores that are not defensible or accurate, not an adequate reflection of the characteristic they were intended to measure, and not fair to the person tested. Thus, the Standards help ensure that measures of student achievement are relevant, that admissions decisions are fair, that employment hiring and professional credentialing results in qualified individuals being selected, and patients with psychological needs are diagnosed properly and treated accordingly. Quality tests protect the public from harmful decision-making and provide opportunities for education and employment that are fair to all who seek them (Camara Decl., ¶ 15; Wise Decl., ¶ 14). 23. The Standards apply broadly to a wide range of standardized vehicles and procedures that sample an individual’s behavior, including tests, assessments, inventories, scales, and other testing vehicles. The Standards apply equally to standardized multiple-choice tests, performance assessments (including tests comprised of only open-ended essays), and hands-on assessments or simulations. The main exceptions are that the Standards do not apply to unstandardized questionnaires (e.g., unstructured behavioral checklists or observational forms), teacher-made tests, and subjective decision processes (e.g., a teacher’s evaluation of students’ classroom participation over the course of a semester) (Camara Decl., ¶ 16; Wise Decl., ¶ 15; Geisinger Decl., ¶ 20; Ernesto Decl., ¶ 13). 24. The Standards have been used to develop testing guidelines for such activities as college admissions, personnel selection, test translations, test user qualifications, and computerbased testing. The Standards also have been widely cited to address technical, professional, and operational norms for all forms of assessments that are professionally developed and used in a variety of settings. The Standards additionally provide a valuable public service to state and 8 federal governments that choose to use them. For instance, each testing company, when submitting proposals for testing administration, instead of relying on a patchwork of local, or even individual and proprietary, testing design and implementation criteria, may rely instead on the Standards to afford the best guidance for testing and assessment practices (Camara Decl., ¶ 17; Wise Decl., ¶ 16; Geisinger Decl., ¶ 21; Ernesto Decl.,¶ 14). D. The Standards Are For Professional, Not Governmental, Use. 25. As described above, the Standards are developed and designed for professional use. The Standards were not created or updated to address any governmental or regulatory need, nor in response to any legislative action or judicial decision. However, the Standards have been cited in judicial decisions related to the proper use and evidence for assessment, as well as by state and federal legislators. These citations in judicial decisions and during legislative deliberations occurred without any lobbying by the Plaintiffs (Levine Decl., ¶ 12; Camara Decl., ¶ 18; Ernesto Decl., ¶ 15; Wise Decl., ¶ 17). 26. During the first round of discovery in this case, APA located some correspondence or draft correspondence relating to APA’s support for legislation proposed in 2001 by Senator Paul Wellstone (D-MN) on Fairness and Accuracy in High Stakes Educational Decisions for Students — a suggested amendment to the Elementary and Secondary Education Act (“No Child Left Behind Act”) 147 Cong. Rec. S. 4,644 (daily ed. May 9, 2001) (Ernesto Decl., ¶¶ 16–22, Exhs. NN-SS). Some of these letters are unsigned and are not printed on APA letterhead. Therefore, in accordance with APA practices, it is likely that the unsigned letters (not printed on letterhead) were internal discussion drafts that were never sent (Ernesto Decl., ¶ 23). 27. Regarding the signed letters printed on APA letterhead, they relate to Senator Wellstone’s proposed legislation mandating that tests and assessments administered by the states be of high quality and used appropriately for the benefit of test administrators and test takers. 9 These are goals that are consistent with APA policy as then reflected in the 1999 Standards. Even though Senator Wellstone’s amendments sought, in part, to mandate States’ compliance with the Standards, the Sponsoring Organizations had not actively advocated for this. In any event, Senator Wellstone’s proposed amendment never became law (Ernesto Decl., ¶ 24, Exh. TT). 28. APA’s search of its records did not disclose any further communications with Congress relating to the use of the Standards in legislation or proposed legislation, and, to the best of APA’s or the other Sponsoring Organizations’ knowledge, the Standards have not been referenced in legislation since 2001 (Ernesto Decl., ¶ 25). Moreover, neither AERA nor NCME has ever communicated with Congress for the purpose of encouraging reference to the Standards in law (Levine Decl., ¶¶ 12-13; Wise Decl., ¶ 18). None of the Plaintiffs has solicited any government agency to incorporate the Standards into the Code of Federal Regulations or other rules of federal or state agencies (Levine Decl., ¶ 13; Ernesto Decl., ¶ 26; Wise Decl., ¶ 19). E. Plaintiffs’ Development of the 1999 Edition of the Standards 29. Preparing and revising the Standards entails intensive labor and considerable cross-disciplinary expertise. Each time the Standards are revised, the Sponsoring Organizations select and arrange for meetings of the Joint Committee, composed of leading authorities in psychological and educational assessments. During these meetings, certain individual standards are combined, pared down, or augmented, while others are deleted altogether, and some are created as whole new individual standards. 30. The 1999 Standards took more than five years to complete. It is the result of work put in by the Joint Committee to generate a set of best practices on educational and psychological testing that are respected and relied upon by leaders in their fields (Levine Decl., ¶ 11; Camara Decl., ¶ 11; Wise Decl., ¶ 11). Draft revisions of the 1985 Standards, which became 10 the 1999 Standards, were widely distributed for public review and comment three times during this revision effort to gauge whether the testing community believed the revised drafts to be current and inclusive of the topics at issue (Schneider Decl., ¶ 6). The Joint Committee received thousands of pages of comments and proposed text revisions from Plaintiffs’ members; scientific, professional, trade, and advocacy groups; credentialing boards; state and federal government agencies; test publishers and developers; and academic institutions. While the Joint Committee reviewed and took under advisement these comments, the final language of the 1999 Standards was produced by the Joint Committee members (Camara Decl., ¶ 12; Schneider Decl., ¶ 7). More than half the content of the 1999 Standards resulted from newly written prose of the Joint Committee (Camara Decl., ¶ 12). F. Description of the 1999 Standards (Hutter Decl. Exhibit 1) 31. Cover-to-cover, the 1999 Standards are 205 pages long. The vast bulk of those 205 pages is explanatory and background material and commentary, not the black-letter standards themselves. For instance, the 1999 Standards include:  A front cover, a back cover, a cover page, and a page showing copyright and publisher information;  A table of contents showing how the 1999 Standards are organized;  A preface providing a short history of the Standards and identifying the people and organizations that participated in creating the 1999 Standards;  A five-plus page Introduction that gives an overview of the testing process; introduces the 1999 Standards; offers some cautions to be exercised in using the Standards; explains some updates specific to the 1999 edition of the 11 Standards; clarifies how the 1999 Standards use the term “construct”; and elaborates on how the 1999 Standards are organized;  An index pointing users to the pages in the 1999 Standards that refer to specific terms; and  A glossary defining how certain terms are specifically used in the 1999 Standards. 32. The remaining 164 pages of the 1999 Standards consist of three substantive parts. Part I is titled “Test Construction, Evaluation, and Documentation.” Part II is titled “Fairness in Testing.” Part III is titled “Testing Applications.” chapters. Each of the three parts is broken up into Part I, for example, contains chapters on validity; reliability and errors of measurement; test development and revision; scaling, norming, and score comparability; test administration, scoring, and reporting; and supporting documentation for tests. 33. Each chapter in the 1999 Standards consists, in turn, of two overall sections. First there is a section of background and explanatory material organized under various topical subheadings. As the introduction to the 1999 Standards makes clear, while the section of background material opening each chapter may be helpful to understanding the black-letter standards, “it should not be interpreted as imposing additional standards.” 34. The background material at the outset of each chapter is followed in each chapter by a section that includes the individual, black-letter standards themselves. Each black-letter standard is stated in bold-face type, one paragraph long, and is written in a prescriptive manner. Each individual standard is followed by a Comment on which may help explain, elaborate on or offer examples. The comments are designed to assist in applying the black-letter standards, but they do not impose additional requirements beyond the black-letter standards. Even in the part 12 of each chapter that contains the standards (which is less than half the chapter), more words tend to be devoted to the commentary than to the black letter standards themselves. 35. The 1999 Standards, including all chapters, contains 272 standards in total. The standards take up less than 15% of 1999 Standards (as determined by comparing the number of lines of text devoted to standards in relation to the lines in the text overall). See Hutter Decl. ¶ 6. G. The Federal Practice of Incorporation by Reference. 36. The National Technology Transfer and Advancement Act of 1995 (“NTTAA”) requires federal agencies to use privately-developed standards to achieve federal objectives whenever possible.2 Pub. L. No. 104-113 § 12, 110 Stat. 775, 782–83 (1996), codified at 15 U.S.C. § 272. Specifically it declares that “all Federal agencies and departments shall use technical standards that are developed or adopted by voluntary consensus bodies, using such technical standards as a means to carry out policy objectives or activities.” Id. 37. One method through which federal agencies avail themselves of privately- developed standards is by incorporating the standards by reference. The Code of Federal Regulations currently contains more than 23,000 incorporations by reference, though not all of the material incorporated are standards developed by private standards-setting bodies. See 2 The NTTAA was the product of a study of the system of privately-developed consensus standards as it had evolved in the United States, which emerged with the recommendation that government agencies continue to avail itself of that system in aid of its own mission. Specifically, in 1991 Congress enacted P.L. 102-245, requesting the National Research Council study standards development. See National Research Council, Standards, Conformity Assessment, and Trade into the 21st Century vii (National Academy Press 1995), available at That study’s overview of the U.S. standards-development system noted that many standards developers “offset expenses and generate income through sales of standards documents, to which they hold the copyright. For many SDOs, publishing is a significant source of operating revenue.” Id. at 32 (emphasis added). The study concluded that the “U.S. standards development system serves the national interest well” by “support[ing] efficient and timely development of product and process standards that meet economic and public interests.” Id. at 157. The study recommended that Congress pass a law to promote the use of privately developed voluntary consensus standards by federal agencies. Congress responded by enacting the NTTAA, codified at 15 U.S.C. § 272. 13 Bremer, Teaching Guide: Incorporation by Reference, 2019 Administrative Law Review at 321; Bremer, Private Standards in Public Law, 63 Kansas L. Rev. 279. 38. The Office of Management and Budget has explained that incorporation by reference (i) saves the government the cost of developing standards on its own; (ii) provides incentives to establish standards serving national needs; (iii) promotes efficiency and economic competition through harmonized standards; and (iv) furthers the federal policy of relying on the private sector to meet government needs for goods and services. OMB Circular No. A-119, 63 Fed. Reg. 8546 (Revised Feb. 10, 1998). See also Final Revision of OMB Circular A-119, 81 Fed. Reg. 4673 (Jan. 27, 2016). 39. OMB has recognized that it is essential to not interfere with the ability of the standard-setting organizations to charge for the use of their works: “If we required that all materials IBR’d into the CFR be available for free, that requirement would compromise the ability of regulators to rely on voluntary consensus standards, possibly requiring them to create their own standards, which is contrary to the NTTAA and the OMB Circular A-119.” See Incorporation by Reference, Announcement of Final Rule, Office of the Federal Register, 79 Fed. Reg. 66267, 66268 (Nov. 7, 2014), available at 40. The practice of incorporation by reference is currently codified at 5 U.S.C. §552(a)(1), which clarifies that no material incorporated by reference can be deemed binding – in the sense of adversely affecting any person – unless the material is reasonably available to the class of affected persons. Thus, any standard incorporated by reference is self-limiting in its application if not “reasonably available.” Except to the extent that a person has actual and timely notice of the terms thereof, a person may not in any manner be required to resort to, or be adversely affected by, a matter required to be published in the Federal Register and not so published. For 14 purposes of this paragraph, matter reasonably available to the class of persons affected thereby is deemed published in the Federal Register when incorporated by reference therein with the approval of the Director of the Federal Register. 41. The Office of Federal Register in turn has promulgated regulations governing the process for agencies to follow in order to obtain approval to incorporate materials by reference in the CFR. 1 C.F.R. §51 (2014). Those requirements interpret and implement the statutory mandate that, in order for matters set forth in the register to be binding, they must be “reasonably available to the class of persons affected.” 5 U.S.C. § 552(a)(1); 1 C.F.R. § 51.7(a)(3). They require that (i) a copy of the incorporated material must be on file with the Office of the Federal Register and (ii) that the regulations incorporating such material must state the ways those incorporated materials are reasonably available to interested parties. 1 C.F.R. §§ 51.3, 51.5. There is no requirement that such materials be available to the public at no cost, or on the internet H. DOE’s Incorporation By Reference of Some Parts of the 1999 Standards 42. Plaintiffs learned primarily during the course of these proceedings that the Department of Education has incorporated certain standards set forth in the 1999 Standards in three sections of the Code of Federal Regulations. As seen below, none of these provisions apply to conventional primary conduct or purport to apply mandatory enforceable rules with penalties attached. Rather, they address how tests may qualify for purposes relevant to Department of Education grant programs. 43. 34 C.F.R. §668.146(b)(6) is the only regulation that was cited by PRO as the basis for its incorporation-by-reference rationale for posting the 1999 Standards. That regulation is part of a subpart of regulations setting forth provisions under which a student without a high school diploma or its recognized equivalent may become eligible to receive funds under Title IV 15 of the Higher Education Act. The student may become eligible by obtaining a diploma or its equivalent. Another way for a student to become eligible for Title IV funds is to pass a test that satisfies certain criteria including “all standards for test construction provided in the 1999 edition of the Standards for Educational and Psychological Testing, prepared by a joint committee of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education incorporated by reference in this section.” The regulation then goes on to note that the incorporation by reference was properly approved, identifies where the material referenced (the 1999 Standards) is on file, and provides a phone number and website for the National Archives and Records Administration “[f]or information on the availability of this material at NARA.” The regulation also identifies the AERA website, noting that the document may also be obtained from there. 44. The individual standards incorporated by reference in 34 C.F.R. 668.146(b)(6) are set forth in Part I of the 1999 Standards, entitled “Test Construction, Evaluation and Documentation.” Part I actually consists of six chapters, covering Validity, Reliability and Errors of Measurement, Test Development and Revision, Scales, Norms and Score Comparability, Test Administration, Scoring, and Reporting, and Supporting Documentation for Tests. Even assuming that all six chapters of Part I are being referenced by the regulation, the actual bold-face “standards” set forth in these chapters are very limited. 45. For example, the chapter within Part I on “validity” runs from pp. 9-24, begins with background and explanatory material, and contains standards only on pages 17-24 (Standards 1.1-1.24), including comments on each standard. The comments on each standard are typically as long as, or longer than the standards themselves. 16 46. The chapter within Part I on Reliability and Errors is eleven pages long and begins with 6 pages of background and discussion of various topics and considerations. Standards (2.1-2.20) are set forth on 6 pages, and the comments to the standards are again longer than the referenced standards themselves. 47. The chapter on Test Development and Revision runs from pages 37-48 and the first 6 pages describe background and discussions of various related topics and considerations. The actual standards are set forth within pages 43-48 (standards 3.1- 3.27), and again, most of that appears to be commentary, rather than the individual standards themselves (though some of the standards have no immediately following “comment” and some comments are as brief as the standard). 48. The next three chapters in Part I follow a similar format. 49. 34 C.F.R. §668.148(a)(1)(iv) is also a part of the subpart of the regulations setting forth provisions under which a student without a high school diploma or its recognized equivalent may become eligible to receive funds under Title IV. This provision covers tests developed for non-native speakers of English enrolled in a program taught in their native language. Such tests must, according to this provision, be “[d]eveloped in accordance with guidelines provided in the “’Testing Individuals of Diverse Linguistic Backgrounds’ section of the” 1999 Standards. 50. That chapter of the 1999 Standards runs from pages 91 to 100, with the first 6 pages covering background and explanatory topics. The actual standards are found at pages 97 to 100, and again, the actual standards themselves appear to occupy less than half of those pages, with comments taking up more space. 17 51. 34 C.F.R. §462.13(c)(1) and (f)(1) are part of the regulations establishing a process for reviewing the suitability of tests, as submitted by test publishers, for use in the National Reporting System for Adult Education, which is an outcome-based accountability system for evaluating state-administered, federally-funded adult education programs. Among the criteria used to determine whether a test is suitable for use in the that system is that “[t]he test must meet all applicable and feasible standards for test construction and validity provided in the 1999 edition of the Standards[.]” But a test or publisher that seeks to qualify under this program is specifically offered the opportunity to demonstrate why a particular standard is not feasible or applicable. 52. This regulation, like 34 C.F.R. §668.146(b)(6), references Part I of the 1999 Standards. The regulation itself contains similar information and directions as to how and where to find the incorporated standards. 53. In addition, subsection (f)(1) provides that “For a test that has been modified for individuals with disabilities, the Error! Hyperlink reference not valid.must,” among other things, “[p]rovide documentation that it followed the guidelines provided in the Testing Individuals With Disabilities section of the 1999 edition of the Standards for Educational and Psychological Testing.” That is reference to a different chapter of the 1999 Standards, contained in Part 2. That chapter comprises 8 pages, the first 5 of which provide background and explanatory material, which is followed by 3 pages containing the black letter standards (10.110.12), many of which are followed a Comment. 54. Less than 6 per cent of the 1999 Standards is occupied by the standards incorporated by reference in the three regulation (as determined by comparing the number of 18 lines of text in the referenced standards with the number of lines in the text overall). Hutter Decl. ¶ 6. 55. Plaintiffs are not aware that DOE has ever had occasion to enforce the provisions of its regulations referencing the 1999 Standards. 56. In addition, Plaintiffs are not specifically aware of tests developed specifically to meet these DOE regulations. Rather, tests meeting these criteria may have been developed entirely independent of the regulations because high quality tests generally would, as a matter of basic professionalism, be developed in a manner consistent with the Standards, but without regard to the fact that some of the standards are cited in DOE regulations. The exception could, of course, be for 34 C.F.R. §462.13 (f)(1), which requires the test publisher to provide documentation that it affirmatively meets the requirements of the referenced portion of the 1999 Standards. But plaintiffs are not in possession of any information regarding how or whether that provision has been applied. 57. Copies of the 1999 Standards are available, as identified in the CFR, in the reading room and from AERA, either by ordering on-line or over the telephone. In addition, existing copies of the 1999 Standards are available and in circulation throughout academia (including schools of education and departments of psychology), are in the possession of organizations involved in testing, can be accessed in public and university libraries, and would expectably be accessible and referenced in connection with the development and use of tests irrespective of whether at some point the test would be in any way subject to the DOE regulations. 19 I. Defendants Copied the 1999 Standards In Toto and Verbatim, and Enabled Others to Copy Them, Without Plaintiffs’ Permission 58. Defendant Public.Resource.Org (PRO) is a California non-profit corporation founded in 2007 by Mr. Carl Malamud, with the explicit aim of making government information more accessible, with particular emphasis on the law (Hudis Decl., ¶ 2, Exh. A, pp. 77, 93–94, 163–164). The identified purpose and objective of PRO is to create and maintain so-called informational “public works projects for the Internet” (Hudis Decl., ¶ 2, Exh. A, pp. 94–95, 105– 09, ¶ 3, Exh. B, Section II.B., ¶ 4, Exh. C. Section 2.1). 59. PRO creates digital copies of standards referenced or incorporated in regulations, without the permission of the copyright owners. It does so as a matter of principle. That is, PRO does not purport to identify any specific need or reason for such copying. Rather, PRO simply believes, and acts on the belief, that all such material should copied and made available for copying on the internet. Its mission includes the broad and indiscriminate objective “to make the law and other government materials more widely available.” 896 F.3d at 444 (quoting Malamud’s Declaration in this case). 60. In March 2012, PRO began copying standards incorporated by reference into the Code of Federal Regulations, and facilitating copying by others. In May 2012, PRO began the process of posting copies these standards to its website. “Between 2012 and 2014, PRO uploaded hundreds of technical standards, which, collectively, were downloaded tens of thousands of times.” Id. 61. Specifically, on May 17, 2012, PRO bought a used hard copy of the 1999 Standards from an Amazon re-seller (Hudis Decl., ¶ 2, Exh. A, pp. 232-240, ¶ 21, Exh. T, Int. Ans. 1, ¶¶ 22-23, Exh. U). Upon receipt of the purchased paper copy, Malamud disassembled the book, removed the spine, trimmed the pages to give them an even border, scanned the pages 20 to create a PDF, and named the PDF file “aera.standards.1999.pdf.” Malamud then appended a cover sheet, a self-made “Certificate,” to the front of the PDF file giving a false semblance of governmental imprimatur to the unauthorized copying and online posting of the 1999 Standards (Hudis Decl., ¶ 2, Exh. A, pp. 257–59, 261–64, ¶ 21, Exh. T, Int. Ans. 3–4, ¶ 26. Malamud did nothing else to modify or transform the PDF file or the Standards. He simply copied and posted the 1999 Standards, in its entirety, to his company’s website. He did not provide for wordsearching, online identification, or text-to-speech utilization for the blind and visually impaired (Hudis Decl., ¶ 27, Exh. Z, pp. 30, 122, 200–01, 206, 271–72, 315–16). PRO published the infringing digital copy of the 1999 Standards on a website titled 62. The Internet Archive is a nonprofit organization whose mission is to build and maintain a digital library of the Internet. The Internet Archive builds this internet library — which it makes available for public use — by scanning, digitally capturing, and saving electronically scanned and captured third-party websites, and by receiving submissions from third parties who have user accounts enabling them to upload content (Hudis Decl., ¶ 29, Exh. BB, pp. 31–41). Malamud has such user account access (Hudis Decl., ¶ 29, Exh. BB, pp. 51– 56), and he uploaded the entirety of the 1999 Standards to the Internet Archive’s website on May 26–27, 2012 (Hudis Decl., ¶ 29, Exh. BB, pp. 59-112, ¶ 30, Exh. CC (¶¶ 3–18 therein), ¶ 32, Exh. EE, ¶ 33). PRO posted Plaintiffs’ 1999 Standards to its website and the Internet Archive website without the permission or authorization of any of the Sponsoring Organizations (Hudis Decl., ¶ 35, Exh. HH, Admission Nos. 4–5; Levine Decl., ¶ 29; Ernesto Decl., ¶ 35; Wise Decl., ¶ 26). 63. The copy of the Sponsoring Organizations’ 1999 Standards that Malamud published to the Internet Archive at was 21 in the same format, using the same cover sheet or “Certificate” employed by PRO in the posting of the Sponsoring Organizations’ 1999 Standards to Defendant’s own website. All of the surrounding text associated with the posting to the Internet Archive website was inserted by Malamud — including the insertion of “Creative Commons License: CC0 1.0 Universal,” indicating that no rights are being asserted over the item (Hudis Decl., ¶ 2, Exh. A, pp. 275–84, ¶ 29, Exh. BB, pp. 57–63, ¶ 30, Exh. CC (¶ 2 therein), ¶ 34, Exh. GG): 64. Although based on incomplete reporting (Hudis Decl., ¶ 2, Exh. A, pp. 272–74, 328–36), during the near two-year period that the Sponsoring Organizations’ 1999 Standards were first posted on PRO’s website, they were accessed at least 4,164 times (Hudis Decl., ¶ 21, Exh. T, Int. Ans. 2 and Amended Ans. 5 (labeled 6)). During that same period, the Sponsoring Organizations’ 1999 Standards were accessed on the Internet Archive website 1,290 times (Hudis Decl., ¶ 29, Exh. BB, pp. 124-132, ¶ 37, Exh. II). 65. The Internet Archive’s website is open to the public and does not restrict an Internet user’s ability to download or print the Sponsoring Organizations’ 1999 Standards. PRO also placed no such restrictions on its website (Hudis Decl., ¶ 2, Exh. A, pp. 347–48). There were no sign-up procedures to enter PRO’s website, nor was there any Digital Rights Management (or “DRM”) plan to protect against, or identify, further copying of the files accessed from PRO’s site (Hudis Decl., ¶ 27, Exh. Z, pp. 324–27, 167–73). 66. There is no way for Plaintiffs to calculate the number of university/college professors, students, testing companies and others who would have purchased Plaintiffs’ Standards but for their wholesale posting on PRO’s website and the Internet Archive website (Levine Decl., ¶ 30; Geisinger Decl., ¶ 24). 22 67. In late 2013 and early 2014, the Sponsoring Organizations became aware that the 1999 Standards had been posted on the Internet without their authorization, and that students were obtaining free copies from the posting source. Upon further investigation, they discovered that PRO was the source of the online posting (Camara Decl., ¶ 21, Exh. MMM; Wise Decl., ¶¶ 27-28, Exh. LLL). 68. In December 2013, Plaintiff AERA requested in writing that PRO remove the 1999 Standards from its online postings (Levine Decl., ¶ 31, Exh. UUU). Defendant refused (Hudis Decl., ¶ 2, Exh. A, pp. 310–19, ¶ 38, Exh. JJ, ¶ 39, Exh. KK). Once this lawsuit was filed and the Sponsoring Organizations threatened to file a motion for a preliminary injunction, PRO agreed in June 2014 to remove its postings of the 1999 Standards from its website and from Internet Archive’s website, pending a resolution of this litigation on the merits. PRO’s undertaking included the promise not to post any revision of the 1999 Standards (i.e., the 2014 Standards) pending the outcome of this litigation on the merits (Hudis Decl., ¶ 2, Exh. A, pp. 322-28, ¶ 40, Exh. LL, ¶ 41, Exh. MM). 69. Notwithstanding that undertaking, it is currently posted on the Internet Archive website, stating that it was uploaded on May 26, 2012. That posting includes the full volume of the 1999 Standards, verbatim, covered with the self-created “Certificate” stating “By Authority of the United States of America … Legally Binding Document,” and stating that the document is posted by Public.Resource.Org. Inc. The website cites a single section of the CFR, 34 C.F.R. §668.148(a)(2)(iv). The website posting provides a variety of means by which the 1999 Standards can be downloaded. See Hutter Decl. Exh 3. The website reflected that as of September 11, 2019, it was viewed 23 1445 times. See J. This Litigation, and the Court of Appeals’ Remand Direction 70. In 2017, this Court granted Plaintiffs’ motion for summary judgment on their copyright claim. The Court held that Plaintiffs own the copyright to the 1999 Standards; that Plaintiffs’ copyright is valid; that nothing about the 1999 Standards’ incorporation into federal law negated the copyright; that PRO’s posting on the internet was infringing; and that PRO’s wholesale copying and posting of the entirety of the 1999 Standards to the internet, from where it could be downloaded, was not fair use. Regarding the fair use issue, this Court’s decision recognized that PRO had the burden on that issue, and the Court addressed the specific arguments that PRO had made – arguments that PRO advanced across the board, without addressing the differences between the many and varied standards at issue in the ASTM case, or the differences between how those standards were referenced in federal regulations. Based on those holdings, the Court enjoined PRO from continuing to post the 1999 Standards. 71. In 2018, the Court of Appeals for the District of Columbia Circuit vacated this Court’s summary judgment and injunction order and remanded the case for further proceedings. The Court of Appeals’ remand order was limited to a single issue: fair use. The Court of Appeals noted that PRO had offered an “undifferentiated” theory of why its publication of any standards incorporated by reference in federal regulations qualified as fair use (or should be denied copyright protection at all). This Court had therefore addressed those arguments of PRO in the undifferentiated terms under which they were presented. 72. But the Court of Appeals believed that approach “failed to account for the variation among the standards at issue and … failed to consider each fair use claim ‘on its own facts.’” Thus, it noted the many and varied standards and incorporations by reference at issue (at least in the ASTM case), and directed that: 24 On remand, the district court will need to develop a fuller record regarding the nature of each of the standards at issue, the way in which they are incorporated, and the manner and extent to which they were copied by PRO in order to resolve this “mixed question of law and fact.” 73. In light of that direction, this Court allowed PRO an extended period to take additional discovery, which it did. Plaintiffs did not take any additional discovery. Respectfully submitted, s/ Clifton S. Elgarten John I. Stewart, Jr. (D.C. Bar No. 913905) Clifton S. Elgarten (D.C. Bar No. 366898) Amanda Shafer Berman (D.C. Bar No. 497860) CROWELL & MORING LLP 1001 Pennsylvania Avenue, NW Washington, DC 20004-2595 (202) 624-2500 Attorneys for Plaintiff-Counterdefendants October 4, 2019 25

Disclaimer: Justia Dockets & Filings provides public litigation records from the federal appellate and district courts. These filings and docket sheets should not be considered findings of fact or liability, nor do they necessarily reflect the view of Justia.

Why Is My Information Online?