AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC. et al v. PUBLIC.RESOURCE.ORG, INC.

Filing 67

MOTION to Strike #60 the declaration of Kurt P. Geisinger by PUBLIC.RESOURCE.ORG, INC. (Attachments: #1 Memorandum in Support of Defendant-Counterclaimant Public.Resource.Orgs Motion to Strike [PUBLIC], #2 Declaration of Matthew Becker [PUBLIC], #3 Exhibit 1 (Filed Under Seal), #4 Exhibit 2 (Filed Under Seal), #5 Exhibit 3, #6 Exhibit 4, #7 Exhibit 5, #8 Exhibit 6 (Filed Under Seal), #9 Exhibit 7 (Filed Under Seal), #10 Exhibit 8, #11 Exhibit 9, #12 Exhibit 10, #13 Exhibit 11, #14 Exhibit 12, #15 Exhibit 13, #16 Exhibit 14, #17 Exhibit 15, #18 Exhibit 16, #19 Text of Proposed Order, #20 Certificate of Service)(Bridges, Andrew) Modified on 1/21/2016 linkage and text(td).

EXHIBIT 3

IN THE UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF COLUMBIA
AMERICAN EDUCATIONAL RESEARCH
ASSOCIATION, INC., AMERICAN
PSYCHOLOGICAL ASSOCIATION, INC.,
and NATIONAL COUNCIL ON
MEASUREMENT IN EDUCATION, INC.,
Plaintiffs/Counterclaim Defendants,
v.
PUBLIC.RESOURCE.ORG, INC.,
Defendant/Counterclaimant.

)
)
)
)
)
)
)
)
)
)
)
)
)
)

Civil Action No. 1:14-cv-00857-TSC-DAR

EXPERT’S DECLARATION AND
REPORT OF KURT F. GEISINGER,
Ph. D. PURSUANT TO FED. R. CIV. P.
26(a)(2)(B)

I, KURT F. GEISINGER, Ph. D., declare:
1.

I am currently Director of the Buros Center on Testing and W. C. Meierhenry

Distinguished University Professor at the University of Nebraska-Lincoln.
2.

The following constitutes my expert’s report in this action on behalf of Plaintiffs,

the American Educational Research Association, Inc. (“AERA”), the American Psychological
Association, Inc. (“APA”) and the National Council on Measurement in Education, Inc.
(“NCME”) (collectively, “Plaintiffs”), complaining of certain activities engaged in by
Defendant, Public.Resource.Org, Inc. (“Public Resource”).
3.

This Declaration and Report contains my opinions to date. The basis for my

opinions, the materials I considered in reaching my opinions, and my qualifications for rendering
such opinions are set forth in this Declaration and attached Exhibits. I reserve the right to
supplement my Declaration to address any additional documents and testimony introduced in this
action that come to my attention between now and the time of any deposition, hearing or trial.

My Qualifications
4.

I received my doctoral degree in Educational Psychology in 1977 from the

Pennsylvania State University, after previously receiving my masters’ degree in Psychology at
the University of Georgia and my bachelor’s degree from Davidson College (with honors). I
also studied German, Psychology and other topics as an undergraduate at the Phillips Universität
in Marburg, Germany and at Harvard University when I attended the Institute for Educational
Management in 1995.
5.

Previously, I served as the Vice President of Academic Affairs and Professor of

Psychology at the University of St. Thomas in Houston, Texas, where I was responsible for four
academic schools, approximately 200 faculty members, and over 4,000 students. I also served as
Academic Vice President and Professor of Psychology at Le Moyne College, Dean of the
College of Arts and Sciences and Professor of Psychology at the State University of New York at
Oswego, and Professor of Psychology at Fordham University in New York City, where I was
department chair for the Department of Psychology and director of the Doctoral program in
Psychometrics.
6.

Over the past forty years, I have researched, studied, and taught psychometrics.

Psychometrics, defined in more detail later in this report, is the quantitative study of tests and
measures in terms of the value, usefulness, and interpretation of the results of such measures. I
also am a fellow, diplomate, and member of numerous professional societies involving
educational and psychological testing, such as the APA (fellow), the American Association for
Assessment Psychology (diplomate), the AERA (fellow), and the NCME, as well as other
professional associations. I have represented the APA by serving on and chairing the Joint
Committee on Testing Practices (which is separate from the joint committee of the AERA, the

2

APA and the NCME responsible for the 1999 Standards for Educational and Psychological
Testing) and have served on the APA’s Committee on Psychological Tests and Assessment. In
2010, I was elected to serve two terms (2006-2008 and 2009-2011) as the representative on the
Council of Representatives for the APA’s Division of Evaluation, Measurement and Statistics.
My second term was cut short by one year when I was elected to serve as a member-at-large on
the APA’s Board of Directors in 2010, a position I held for a three-year term (2011-2013).
7.
testing.

I have authored numerous publications about psychological and educational
I have worked at the Educational Testing Service (“ETS”), chaired its Technical

Advisory Committee for the Graduate Record Examination (“GRE”), served on the Board of
Directors for the GRE (a Board that I also chaired), and have been a member of the College
Board, (formerly known as the College Entrance Examination Board) for which I served on its
SAT Committee (from 2000-2002). I recently concluded a four-year term (from 2011-2014) on
the Advisory Research Committee for the College Board, serving the last two years as its chair.
I currently serve on the Technical Advisory Committee for the Educational Records Bureau.1
8.

In 2010, I was elected to the Council (i.e., Board of Directors) for the

International Test Commission—the primary international testing body. In 2012, I also was
elected as its Treasurer and to serve on its Executive Council. I am the only American on its
Executive Council.
9.

I was asked to review and share my comments on chapters of the 1999 Standards

for Educational and Psychological Testing, published jointly by the AERA, the APA, and the

1

The Educational Record Bureau specializes in the development and use of tests and testing
products for private and independent educational institutions at the p-12 levels.
3

NCME (the “1999 Standards”). The Joint Standards2 embody the professionally accepted
practices for testing and measurement. One of the chapters I reviewed was based upon the
testing of individuals with disabilities, an area in which I have engaged in research and have
served as an expert witness in federal courts as well as state courts in New York, New Jersey,
and California. The other chapter related to the rights and responsibilities of test takers. See
Exh. A. I note that the Joint Standards were revised in 2014.
10.

In addition to my 130 plus journal articles and book chapters, I have written,

edited, or co-edited approximately 15 books and monographs.

The vast majority of these

publications deal with testing and measurement issues. For example, I have edited two books on
the psychological testing of Hispanics and another I co-edited related to fairness in testing. I also
have co-edited several books of reviews of published tests and measures. I also was Editor-inChief for the three-volume Handbook of Testing and Assessment in Psychology (published by the
APA in 2013).

Additionally, I have been editor of the journal Applied Measurement in

Education for the past 8 plus years. Taylor & Francis, in conjunction with the Buros Center for
Testing, publishes this journal.
11.

I also co-chaired a sub-committee of the APA’s Joint Committee on Testing

Practices and the overall committee itself that developed a document on the rights and
responsibilities of test takers (from 1993-2001). This document has been endorsed by a number
of professional associations related to proper test use, including the APA, the National
Association of School Psychologists, the American Counseling Association, and the NCME.
While chairing the Joint Committee on Testing Practices, the committee developed a book
entitled Assessing Individuals with Disabilities, in which I wrote a chapter. I also served on a
2

I use the term Joint Standards to refer to the Standards for Educational and Psychological
Testing as a whole, not a specific version of the Standards, i.e. 1999 or 2014
4

task force charged to illuminate issues related to the testing of individuals with disabilities as
well as ethnic minorities. The task force wrote and edited a book entitled Test Interpretation and
Diversity: Achieving Equity in Assessment, which was published by the APA’s publication unit
in 1997. I authored three chapters in that volume.
12.

I additionally served on an APA task force (from 2007-2010) that considered the

assessment and intervention of individuals with disabilities. The results of our work, Guidelines
for the “Assessment of and Intervention with Individuals with Disabilities,” was published in the
American Psychologist, the premier publication of the APA (Geisinger et al., 2012) and endorsed
as the policy of the APA by its governance. A reference for the American Psychologist article
may be found on my curriculum vitae, which is attached as Exhibit A.
13.

In the past two years (2014-2015), I have served on two task forces related to the

use of measures in clinical psychology. One of these has written a policy, recently accepted by
the APA’s Board of Directors, that differentiates the use of tests and other measures, for
screening and assessment, two highly related types of testing, but which differ in specificity and
focus. Tests are usually standardized measures that are given to a number of people for a
specific purpose. A bar examination would be an example of a test. Measures are other
typically quantitative values used to evaluate a person and include tests. A bathroom scale
results in a measure (weight), but would not normally be considered as a test.
14.

During 2013-2014, I served on a committee of the Institute of Medicine (a

component of the National Academy of Sciences) that evaluated the use of psychological and
clinical neuropsychological measures by the Social Security Administration in determining
disability status. The final report, entitled Psychological Testing in the Service of Disability

5

Determination, is in the process of being published, but is also available from the Institute of
Medicine’s website.
15.

For approximately four years (from 2008-2012), I jointly represented three

professional associations (the AERA, the APA, and the NCME) in developing the International
Organization for Standardization’s (“ISO”) first standard on psychological testing. The results
of the work of the committee that engaged in this activity was ISO Standard 10677. The
standard is divided into two parts. The first part establishes requirements and guidance for a
client working with a service provider to carry out the assessment of an individual, a group, or an
organization for work-related purposes.

ISO 10667-1:2011 enables the client to base its

decisions on sound assessment results. ISO 10667-1:2011 also specifies the responsibilities of a
service provided in terms of the assessment methods and procedures that can be carried out for
various work-related purposes made by or affecting individuals, groups or organizations. The
second part lays out the responsibilities of the service provider in terms of the same assessment
project.
16.

I also developed or helped to develop a number of testing measures. Specifically,

I served as the primary consultant on a number of civil service examinations given in New York
City for police officer, sergeant, lieutenant, and captain, fire fighter, fire lieutenant, fire captain,
sanitation supervisor, and a variety of other civil service occupations over a period of at least a
decade ending in 1992. I sometimes defended these measures in court. I also represented the
Public Service Alliance of Canada against the Public Service of Canada in two cases related to
their national testing efforts and Disability Rights Advocates with regard to several testing
disputes concerning individuals with disabilities. See Exh. A.

6

17.

In recent years, my primary efforts have been to assure testing fairness for those

with disabilities, language minorities, and ethnic minorities.
18.

My curriculum vitae is attached to this Declaration and Report as part of Exhibit

19.

A list of all publications that I have authored in the past 10 years is included in

A.

my curriculum vitae. See Exh. A.
20.

To the best of my memory, during the past 4 years, I have not testified as an

expert at trial or by deposition. Previously, I have been accepted as an expert on testing in state
courts in New York, New Jersey, and California, and in federal courts in New York, New Jersey,
and Canada. Within the past four years (2011-2015), I have been identified as an expert in cases
that were settled prior to trial. I wrote a report on the use of testing to deny an individual with
disability benefits for a state agency in Nebraska this past fall, but the matter was resolved prior
to going to court or arbitration. In none of these cases have I given deposition testimony.
21.

To explain what psychometrics is, I provide below the first two paragraphs of my

entry in the Corsini Encyclopedia of Psychology (2010, Wiley, 3rd edition) on the topic of
“Psychometrics: Norms, Reliability, Validity, and Item Analysis”.
The field of psychometrics generally considers the data from educational and
psychological tests and assessments from a quantitative perspective. Such data normally
emerges from test responses, although it may come from a wide variety of measurement
instruments. Two divisions might be identified within psychometrics: theoretical and
applied psychometrics. Psychometric theory (as portrayed by Embretson & Reise, 2000;
Lord, 1980; McDonald, 1999; Nunnally, 1978) provides researchers and psychologists
with mathematical models used in considering responses to individual test items, entire

7

tests and sets of tests. Applied psychometrics is the implementation of these models and
their analytic procedures to test data (e.g., Thorndike, 1982).
22.

For over five years (1989-1995), I taught courses for the Cornell University

School of Industrial and Labor Relations on the topic of affirmative action and equal opportunity
hiring. These courses related to the use of tests in a fair and valid way to make personnel
decisions such as hiring and promotion while attempting to increase diversity and to be in
conformance with federal laws and guidelines.
23.

I have had past and ongoing relationships, as a member or fellow, with each of the

three Plaintiff associations. I currently serve as a committee chair of one of AERA’s divisions’
(Division D – Measurement and Research Methodology) International Committee.

I have

presented at AERA’s annual conferences regularly.
24.

I have served on and chaired NCME’s professional development committee

(1990-1992), served as a program co-chair for its annual meeting (1993), ran for its board (and
was defeated) (1993), represented it on the committee that developed its code of professional
conduct (ethics), and was a representative and advisory board member of a doctoral program in
psychometrics that was being developed at Morgan State University (2007-2012).

I have

published in several of NCME’s journals and have served on the editorial committee for the
journal, Educational Measurement: Issues and Practice (1992-1995).
25.

I also was elected, and have served, as a member of APA’s Committee on

Psychological Tests and Assessment (1998-2000); on its Committee on International Relations in
Psychology (2010); on its Joint Committee on Testing Practices (1992-1996), on its Council of
Representatives (two terms from (2006-2010) representing the Division of Measurement,
Evaluation and Statistics; and on its Board of Directors. I was appointed to serve on APA’s

8

Good Governance Task Force that prepared a plan to reorganize its governance. I also was
appointed to serve on perhaps a half dozen APA task forces over the years related to testing
issues of one type or another (e.g., the testing of individuals with disabilities, the testing of
individuals who are ethnic minorities, the use of testing in clinical psychology). I served for
eight years (from 1992-2000) as a member of APA’s editorial board for its journal,
Psychological Assessment, and recently served on the committee that selected a new editor for
that journal. Further, I served all three of these organizations by representing them on an
American National Standards Institute (“ANSI”)/International Organization for Standardization
(“ISO”) committee that developed an international standard for industrial testing.
Materials that I have Considered
26.

A list of the facts, data, and materials that I have considered in forming my

opinions in this case is attached to this Declaration and Report as Exhibit B.
My Opinions Relevant to this Case, and the Basis and Reasons for my Opinions
27.

The Joint Standards serve as the foundation for the testing profession. It is the

most authoritative single source about the best practices in testing. Other associations (i.e., the
International Test Commission and the American Counseling Association) have much shorter
and less comprehensive guidelines or standards related to testing or certain aspects of testing, but
none have achieved the prominence that the Joint Standards currently enjoy. Also, none have
the pervasive influence across different aspects of the use of tests. That is, the Joint Standards
are appropriate in a wide range of diverse clinical, counseling, educational, and industrial
settings with a variety of populations. It is because of the widespread respect in which the Joint
Standards are held that they are sometimes cited in court cases. Elaborations of this conclusive
statement follow.

9

28.

I currently direct and have directed for the past nine years the Buros Center for

Testing, formerly known as the Buros Institute of Mental Measurements. It was founded some
80 years ago by Oscar Buros, then a faculty member at the Rutgers University, to be essentially
the Consumer Reports of the testing industry. The Buros Center publishes comprehensive,
critical reviews of testing. These reviews are available through our published volumes entitled
Mental Measurement Yearbooks. I have spoken with the other editors of our primary document,
the Mental Measurements Yearbook, and we agree that the most commonly cited document in
the reviews of tests is the Joint Standards. Those who review tests and testing practices refer to
the Joint Standards constantly, and this is reflected in our Mental Measurements Yearbooks. We
believe it to be the most frequently used comprehensive yardstick against which the quality of
tests and measures and the quality of test use is evaluated.
29.

In the 1990s, I co-chaired APA’s Joint Committee on Testing Practices (which

had no direct or formal relationship with the joint committee of the AERA, APA and NCME that
develops and revises the Joint Standards). The role of this committee on testing practices was to
develop documents and products that could improve testing practices. One document that the
committee developed and subsequently revised was entitled the Code of Fair Testing Practices
in Education, a very brief document written for parents and users of educational tests alike. The
members of the Joint Committee on Testing Practices agreed that the principles espoused in the
Code of Fair Testing Practices in Education had to be consistent with the Joint Standards (given
their pre-eminent status). I also co-chaired a working group of that committee, which developed
a document entitled the Rights and Responsibilities of Test Takers. We began our work this
document by reviewing everything that the Joint Standards had to say about this aspect of
testing.

10

30.

In the multiple editions of the Joint Standards, various psychometric concepts

have evolved or changed. Most testing experts believe that the single most important quality in
testing is the validity of test scores—that they are used and interpreted in appropriate and useful
ways. One can trace the history of our profession’s perceptions of validity and how it may be
estimated and determined by studying its portrayals throughout the seven versions of the Joint
Standards that have been published to date.
31.

Background of the Standards for Educational and Psychological Testing, and its

importance to the testing professions: The history of the Joint Standards is not brief. This
history reflects changes in psychometric technology, our understandings of testing, and
psychological/educational characteristics, societal fluctuations, technological improvements, as
well as general zeitgeist differences. The first set of standards was published in 1954 by the
APA and was entitled, Technical Recommendations for Psychological Tests and Diagnostic
Techniques. Its impact on the testing field was monumental. Shortly after the publication of this
volume in 1955, AERA and NCMUE (the original name of NCME was the National Council on
Measurements Used in Education) published a similar document devoted almost exclusively to
educational measures entitled, Technical Recommendations for Achievement Tests. There had
been collaboration across these organizations in the development of these two initial documents
(Eignor, 2013). The three organizations subsequently decided that their work should continue
collaboratively. The two preceding documents and their first joint effort all described what test
publishers should include in their test-related documentation and, to generate such information,
what research efforts should be made in test development and use.
32.

In 1963, the Joint Committee (across the three associations – AERA, APA and

NCME) was formed, and the first Joint Standards were ultimately published in 1966 as the

11

Standards for Educational and Psychological Tests and Manuals (the “1966 Standards”). The
next effort began only five years after the publication of the 1966 Standards.

The Joint

Committee worked from 1971 through the publication of their revised Standards in 1974 (the
“1974 Standards”). This 1974 set of test standards focused less on documentation and more on
topics such as how tests should best be developed, used, scored, and results reported. To
emphasize the reduced focus on documentation, the title of the 1974 Standards was changed to
Standards for Educational and Psychological Tests.
33.

In the early 1980s, still another Joint Committee was empaneled and charged with

the revision of the Joint Standards, a process that concluded with the publication of the 1985
Standards for Educational and Psychological Testing (“the 1985 Standards”).

This minor

change in the title from “tests” to “testing” emphasizes the changed focus from the tests
themselves to the process of, use of, and interpretation of tests and the results of testing.
34.

The 1999 Standards in question in this case held with the same title.

The

development process of the 1999 Standards began with open meetings where many individuals
were able to speak to the Joint Committee to provide input into the ways that they believed the
Joint Standards should change. I was one such participant at meetings in Alexandria, VA in
October of 1994. Whereas the time needed to write the Test Standards had been approximately 3
years prior to the 1999 Standards, it appears that the time was closer to 4 years before the 1999
Standards became available.
35.

The revision process for the current Standards that were published in 2014 was

longer, although the final publication was delayed due to the present dispute with Public
Resource. The Joint Committee was formalized in or around 2007, the first meeting was held in

12

January 2008, and the Committee completed its work in 2013. This revision process took five
years.
36.

The members of each Joint Committee are all volunteers, but staff support is

needed. Not counting the very real costs of contributed staff support, the budget for the Joint
Committee’s meetings was approximately $400,000. Had the staff salaries been covered in this
budget, and had members of the Joint Committee received even minimal compensation for the
work they performed, the budget for revising and updating the Joint Standards would have been
approximately $2,000,000. These costs will continue to increase.
37.

There are several reasons I expect such cost increases. Testing is becoming both

increasingly complex and increasingly technological. For example, 20 years ago all testing was
“in person” or “paper-and-pencil.” Now there is testing via computer, testing via tablet, and
testing via phone, but the “in-person” and “paper-and-pencil” testing method still continues.
Unproctored testing on the internet is presently the most common type of personnel selection
testing in non-public settings in the United States. In northern Europe, the most common type of
testing is now via the internet.

Secondly, society is putting increasing emphasis upon

measurement concerns. Teachers are being evaluated in many settings based upon how their
students perform on tests. Every year more professions and positions require tests to justify
access to positions (e.g., via licensure and certification testing). As more and more testing cases
are litigated, the need for clarification of professional practice increases concomitantly. Finally,
travel and hotel expenses only continue to rise.
38.

Prior to the 1985 Standards, all of the individual standards composing the

Standards volume were considered separately as “essential, very desirable, or desirable”. These
statements indicated relative degrees of importance. The 1985 Standards were identified as

13

primary, secondary, or conditional, depending upon both the importance and the nature of their
use. Beginning in 1999, no status was assigned to individual standards, a practice that was also
continued for the 2014 Standards.

I heard informally from members of the 1999 Joint

Committee that the reason for the change was to de-emphasize the role that the Joint Standards
might play in litigation.
39.

One can glean from the prior discussion that the Joint Standards are changed and

updated approximately every 10-15 years. The revision process has been taking an ever-longer
amount of time, probably due at least in part to the increased focus on educational testing in the
accountability movement across education in the United States. Testing is now used in part to
affect federal budgets allocated to states to provide education and, for example, for teacher and
professional staff evaluation. It is likely that the time frame will continue to increase as the focus
on educational testing shows no letup in the foreseeable future. In fact, it is clear that the Joint
Standards were once revised every 10 years and for the past two revisions, it has taken about 15
years. One reason for this temporal increase relates to the complexity of the changes.
40.

Having three important professional associations involved in the development and

updating of the Joint Standards helps bring much credence to them. It also, however, makes the
process more cumbersome due to the communications and decision-making necessitated by
having three professional associations involved that may all see various professional issues
differently.
41.

Since the revision of the 1999 Standards, the development of revised Joint

Standards is controlled by a Management Committee. The three associations (AERA, APA, and
NCME) established a Management Committee consisting of three representatives, one from each
sponsoring organization. These members are appointed by the Chief Executive Officers of

14

AERA and APA and by the President of NCME. Each member normally serves a 3-year,
renewable term. Members are usually appointed so that terms are staggered, providing for
continuity. Members may be reappointed for one additional three-year term, and in years when
the Joint Standards are in active revision, members’ terms for the duration of the publication
period may be extended by each sponsoring organization.
42.

Members of the Management Committee oversee all aspects of the Joint

Standards on an ongoing basis, including, but not limited to: publication and distribution,
oversight of the Development Fund, protecting the copyright of the Standards, archival activities,
gathering information about their use, and potential revision issues.

The Management

Committee represents both the interests of the Joint Standards and the interests of all three
sponsoring organizations in conducting its administrative duties.
43.

At least once every five years, members of the Management Committee confer

with the leadership of their respective organizations to assess the need for revision of the Joint
Standards.
44.

If it is determined that the Joint Standards need revision, the Management

Committee first appoints co-chairs of a “Joint Committee” that addresses the specifics of the
revision effort. In the case of the 2014 Standards, a website was developed whereby members of
the sponsoring organizations could provide feedback concerning the changes that they believe
necessary or important.

Following this process, in collaboration with the co-chairs, the

Management Committee appoints members of the Joint Committee who represent the sponsoring
organizations. The Joint Committee provides a structure for communication with the sponsoring
organizations throughout the revision process.

15

45.

One of the reasons that the process is so lengthy is that the Joint Committee is

composed of nationally prominent experts cutting across clinical psychology, counseling
psychology, school psychology, industrial/organizational psychology, clinical neuropsychology,
and educational testing. The Joint Committee is composed of highly regarded professionals at
the top of their respective fields, who have specialized knowledge regarding the use of tests and
measures in their respective disciplines. Getting such extraordinarily busy individuals to agree to
serve in a volunteer, unpaid fashion is difficult enough. Working with their schedules to set up
meetings when all can attend is administratively an arduous process. My understanding is that
the individuals take on this task as service to their profession, their associations, and the Joint
Standards themselves.
46.

The Management Committee oversees the process throughout the development of

new standards, and reports back to their respective associations.
47.

The entire Joint Standards revision process is financed through sales of a prior

version of the Joint Standards. As noted above, the direct costs for the development process has
been estimated at $400,000 for the 2014 Standards and between $500,000 and $600,000 for the
1999 Standards.
48.

What Public Resource did with the 1999 Standards: I reviewed Public Resource’s

discovery responses and the transcripts (with exhibits) from the depositions of Carl Malamud
(President and Founder of Public Resource) and Chris Butler (Office Manager of the Internet
Archive). In May 2012, I understand that Mr. Malamud purchased a used copy of the 1999
Standards. I understand further that he then sliced the printed pages out from their bindings,
scanned the pages to a PDF file, and posted the file to Public Resource’s website as well as to a
publicly available collection on the Internet Archive. See Exh. C at 5-6. A graduate psychology

16

student who engaged in such actions would probably be dismissed from his or her program.
Most certainly he or she would be subject to ethics charges that could follow them throughout
the person’s career.
49.

The PDF file posted to Public Resource’s website and to the Internet Archive

contained a cover page prepared by Mr. Malamud, leading Internet users who came upon it to
believe that the 1999 Standards were freely available for download, copying, or whatever use
someone wanted to make of the text. See Exh. D. To the extent that individuals accessed the
1999 Standards in this fashion, it would appear to be theft of services.
50.

The PDF file with Mr. Malamud’s cover page was posted to Public Resource’s

website and to the Internet Archive from July 2012 until June 2014. See Exh. C at 5. Based
upon incomplete records provided by Public Resource, during the time it was posted to the
Internet, the PDF file containing the entire text of the 1999 Standards that was posted to Public
Resource’s website was accessed by Internet users at least 4,405 times. See Exh. C at 9-10. The
same PDF file containing the entire text of the 1999 Standards that was posted to the Internet
Archive website was accessed by Internet users at least 1,113 times. See Exh. E. No restrictions
were placed on this PDF file to prevent Internet users from downloading the file to their local
hard drives or printing it using a printer attached to an Internet user’s computer. See Exh. F at
346-47. These accessions represent considerable lost revenue to the organizations supporting the
Joint Committee. However, even if the incomplete records kept by Public Resource and Internet
Archive were completely accurate, such numbers would not represent the real numbers as lost
revenue because one person can download the Joint Standards and share them with hundreds of
colleagues.

17

51.

In December 2013, AERA asked Mr. Malamud to remove the 1999 Standards

from the Internet locations where he posted the document.

He refused.

Ultimately, Mr.

Malamud did remove the 1999 Standards from public view on the Internet, but only after Public
Resource was sued for copyright infringement and threatened with a motion for a preliminary
injunction. See Exhs. F at 324-26, G.
52.

During his deposition, Mr. Malamud testified that, should Public Resource

succeed in this litigation, it would be a very easy matter for him to re-post the 1999 Standards to
his company’s website and to the Internet Archive website. See Exh. F at 307. Further, Mr.
Malamud contemplated that Public Resource might do the same with the 2014 Standards. See
Exh. F at 308-09. Such actions would almost certainly lead to the 2014 Standards being the last
one developed or published. I have heard from definitive sources within the three organizations
(AERA, APA and NCME) that without the revenue from the sale of the Joint Standards, there
would not be funding from other sources to continue updating them. There simply would not be
funding to continue the updates without these needed sales revenues.
53.

Given the changes happening concomitantly in testing, the testing profession,

and society generally, the cessation of updates to the Joint Standards would be a travesty. The
profession relies on the Joint Standards increasingly in a changing world where high stakes
decisions are often buttressed by information gleaned from tests and measures. Indeed, the Joint
Standards are needed. The Joint Standards are critically important to professionals who work
with tests and measures in education and psychology.
54.

Public Resource’s justification for posting the 1999 Standards to the Internet: It is

Public Resource’s view that, once the 1999 Standards were incorporated by reference into
federal and/or state regulations, the 1999 Standards lost their copyright protection.

18

As a

consequence, Public Resource believes that it and others can freely reproduce the 1999 Standards
(in this case, in electronic format), and post the document to the Internet so that it is freely
available to everyone. See Exh. H.
55.

The past and continued harm that electronically reproducing and posting the 1999

Standards to the Internet will cause AERA, APA and NCME: The three associations that are the
Plaintiffs in this case (AERA, APA, and NCME) are integrally involved in the revision and
publishing of the Joint Standards. I have discussed the prospects of the continuation of the Joint
Standards with knowledgeable representatives of all three organizations. I fear that the 2014
Standards will be the final version should AERA’s, APA’s, and NCME’s copyright infringement
claims against Public Resource not succeed. The loss of income caused by the document being
made freely available has already had a significant and negative impact. The loss of sales
revenue negatively affected the three associations' budget for the development of the 2014
Standards, and cost the associations some credibility for seeming to permit an organization such
as Public Resource to violate copyrights that the testing profession considers so sacred (because
tests too are copyrighted).
56.

Almost certainly, none of the three associations would be willing or able to

finance the continuation of revisions to the Joint Standards if they are made freely available.
The Plaintiff associations would not be financially able to continue the re-development process
into the future. None of the associations would even have the inclination of their governance or
membership to carry on with publishing the Joint Standards, given the additional burden this
would place on membership costs.
57.

The past and continued harm that electronically reproducing and posting the 1999

Test Standards to the Internet will cause to the testing professions and the public: The primary

19

consequences of not revising the Joint Standards would be twofold: to the public, who are
impacted by changes in testing practices, and to test users and their clients. Because of society’s
reliance on test results, a significant portion of the population is benefitted by proper testing
practices (e.g., employers select the best and most appropriate job candidates, colleges and
universities choose the applicants most likely to succeed in their programs, students receive
credit for their learning, programs can assess their successfulness). Changes are necessitated to
testing practices when societal norms and technology change. Recent editions of the Joint
Standards have included chapters on the testing of ethnic minorities, individuals with disabilities,
and fairness, for example. The Joint Standards represent something of a gold standard to which
test developers and users aspire. The Joint Standards also have been modified as the needs for
various kinds of measurement have changed. Should the practice of posting the Joint Standards
to the Internet continue, it is likely that there will be no formally sanctioned process for their
continuation. The agency that I run is essentially one for consumer protection. Without the Joint
Standards, I fear that many customers (clinical psychologists, counseling psychologists,
industrial psychologists, school psychologists, test developers, psychometricians, and the
organizations that each works in) will be the ones losing.
Conclusions
58.

The Joint Standards represent the single best and most complete statement of how

tests and other measurements should be developed, used, evaluated, and interpreted.
The Joint Standards have a long development history (for the social sciences). It is a history that
is currently endangered by what I consider copyright theft. We in academe and the scholarly
professions often report that all we have is our ideas and our writing. If someone is able to steal
our ideas so openly and callously, our professions and indeed our society suffer.

20

59.

AERA, APA, and NCME have engaged in a laborious and time consuming

project (actually a history of projects) that they expected to be rewarded with resultant modest
revenues. The vast portion of these revenues has gone to funding the process for the continual
revision of the Joint Standards. Without such a revenue stream, the Joint Standards may end
with the current edition.
60.

If there is not a next edition of the Joint Standards in the 2020s, then needed

changes in professional practice would not be acknowledged in as formal and yet aspirational a
manner as permitted by the Joint Standards. While the Joint Standards are not published to
make money per se, there is an expectation of modest revenues. The Joint Standards are
debated, considered, and written to improve practice. Funding is needed to continue this effort.
61.

Extremely well qualified members of our professions are willing to volunteer

their time to serve on the Joint Committee to work on the Joint Standards. They do so for two
primary reasons: i) to improve professional practice in their area of expertise, and ii0 to benefit
their professional associations.

If the latter goal is removed due to lost revenues to the

professional associations, then the quality of those willing to serve is likely to be reduced.
People may only engage in this work if they are compensated, which again would be extremely
difficult given the lack of or severe reduction in revenues.
62.

The current Joint Standards were published in 2014. Yet the version that Public

Resource placed on the Internet for free access was the 1999 version of the Joint Standards.
Unsuspecting people (e.g., students) may well access these freely available standards and believe
that they are the “current” Joint Standards. Moreover, given the high stakes nature of our
society presently, suppose a small test developer (and there are many of them) accessed the
outdated Joint Standards and used them to develop, use, and/or interpret the results of a test. It

21

is possible that their decisions and actions would be out of date. Further, given that the Joint
Standards have been quoted as having been given great deference by the courts, it is possible
that should a situation like the above occur, a test developer or test user could be placed in a
situation whereby the advice that they follow is outdated, perhaps inappropriate advice. They
could even be subjected to legal liability for such a well-intended, if somewhat naïve, action.
63.

I expect that when one successively publishes a document and revisions to that

document, one has a reasonable expectation that the publisher does so on the basis of exclusivity.
No one else can publish that same document. The Mental Measurement Yearbooks, which the
Buros Center for Testing that I direct publishes, lost considerable money in its first four or five
editions. We are now publishing the 20th edition, and we have the expectation to earn revenues
with each successive edition. I would hope that if someone else would be prohibited from
simply copying our books and distributing them for free. The well-earned reputation of solid and
professional work should be rewarded with an expectation of a fair return. Giving an item away
for free, such as the Joint Standards, violates that very principle. If the revenue loss is serious
enough, it may remove the desire and expectation for the sponsoring organizations (AERA, APA
and NCME) to continue publishing the document. That is exactly the situation in which these
three professional organizations find themselves.
Engagement and Compensation
64.

A Letter of Agreement engaging my services in this action and stating the

compensation that I will be paid for my study and testimony in the case is attached to this Report
as Exhibit I.
I DECLARE, und the penalty of perjury, that the foregoing is true and correct.

22