Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 1370

Unredacted Declaration of Ravin Balakrishnan, Ph.D in Support of Apple's Opposition to Samsung's Motion for Summary Judgment re 1256 Order on Administrative Motion to File Under Seal, (Dkt. No. 1024) by Apple Inc.. (Jacobs, Michael) (Filed on 7/26/2012) Modified text on 7/27/2012 (dhm, COURT STAFF).

1
2
3
4
5
6
7
8
9

HAROLD J. MCELHINNY (CA SBN 66781)
[e-mail]
MICHAEL A. JACOBS (CA SBN 111664)
[e-mail]
JENNIFER LEE TAYLOR (CA SBN 161368)
[e-mail]
ALISON M. TUCHER (CA SBN 171363)
[e-mail]
RICHARD S.J. HUNG (CA SBN 197425)
[e-mail]
JASON R. BARTLETT (CA SBN 214530)
[e-mail]
MORRISON & FOERSTER LLP
425 Market Street
San Francisco, California 94105-2482
Telephone: [phone]
Facsimile: [phone]

WILLIAM F. LEE
[e-mail]
WILMER CUTLER PICKERING
HALE AND DORR LLP
60 State Street
Boston, MA 02109
Telephone: [phone]
Facsimile: [phone]
MARK D. SELWYN (SBN 244180)
[e-mail]
WILMER CUTLER PICKERING
HALE AND DORR LLP
950 Page Mill Road
Palo Alto, California 94304
Telephone: [phone]
Facsimile: [phone]

10
11

Attorneys for Plaintiff and
Counterclaim-Defendant APPLE INC.

12
13

UNITED STATES DISTRICT COURT

14

NORTHERN DISTRICT OF CALIFORNIA

15

SAN JOSE DIVISION

16
17

APPLE INC., a California corporation,
Plaintiff,

18
19
20
21
22

v.
SAMSUNG ELECTRONICS CO., LTD., a
Korean business entity; SAMSUNG
ELECTRONICS AMERICA, INC., a New
York corporation; SAMSUNG
TELECOMMUNICATIONS AMERICA,
LLC, a Delaware limited liability company,

Case No.

11-cv-01846-LHK (PSG)

DECLARATION OF RAVIN
BALAKRISHNAN, PH.D. IN
SUPPORT OF APPLE’S
OPPOSITION TO SAMSUNG’S
MOTION FOR SUMMARY
JUDGMENT

23
Defendants.
24
25
26
27

**CONFIDENTIAL – CONTAINS MATERIAL DESIGNATED AS HIGHLY
CONFIDENTIAL – ATTORNEYS’ EYES ONLY PURSUANT
TO A PROTECTIVE ORDER**
SUBMITTED UNDER SEAL

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

1

TABLE OF CONTENTS

2
3
4

Page
I.
II.
II.

5
6
7
8

III.

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
IV.

QUALIFICATIONS ........................................................................................................... 1
MATERIALS CONSIDERED ........................................................................................... 5
LEGAL PRINCIPLES ........................................................................................................ 5
A.
Anticipation ............................................................................................................. 6
B.
Obviousness ............................................................................................................ 7
C.
Invention and Patent Application Dates.................................................................. 8
A.
Claim Construction ................................................................................................. 9
DETAILED OPINION ....................................................................................................... 9
A.
Background of the ’381 Patent................................................................................ 9
B.
Person of Ordinary Skill in the Art ....................................................................... 13
C.
Validity of the ’381 Patent .................................................................................... 14
1.
The Tablecloth program is not equivalent to DTFlash ............................. 15
2.
The Van Dam Declaration does not demonstrate that the Tablecloth
program was used in public, publicly known, or commercially sold
before the critical date for the ’381 patent ................................................ 16
3.
The DiamondTouch system required precise calibration in order to
function as intended .................................................................................. 16
4.
The “electronic document” in the Tablecloth program differs in the
Van Dam Declaration, Dr. Van Dam’s Expert Report, and
Samsung’s Invalidity Contentions ............................................................ 17
5.
The DiamondTouch system does not have a touch screen display ........... 21
6.
Dr. Van Dam’s opinion requires a particular motion ................................ 21
7.
The Tablecloth program does not display an area beyond the edge
of an electronic document in response to an edge being reached ............. 22
8.
The Tablecloth program does not translate an electronic document
in a second direction until the area beyond the edge of the
electronic document is no longer displayed in response to the edge
of the electronic document being reached ................................................. 23
9.
Tablecloth does not always snap back ...................................................... 25
10.
The Tablecloth program does not anticipate or render obvious claim
19 of the ’381 patent ................................................................................. 27
11.
Secondary considerations of non-obviousness ......................................... 28
DEMONSTRATIVES AND EXHIBITS.......................................................................... 32

25
26
27
28
i

1

I, Ravin Balakrishnan, Ph.D., do hereby declare as follows:

2

1.

I am a tenured Professor in the Department of Computer Science at the University

3

of Toronto, and have been asked by counsel for Apple Inc. (“Apple”) to provide analysis and

4

expert opinions in the above-captioned case. I understand that in response to Apple’s allegations

5

of patent infringement, Defendants Samsung Electronics Co. Ltd., Samsung Electronics America,

6

Inc., and Samsung Telecommunications America, LLC (collectively “Samsung”) have asserted

7

that United States Patent No. 7,469,381 (“the’381 patent”) is invalid, and submitted in support of

8

its position the Declaration of Andries van Dam, Ph.D. in Support of Samsung’s Motion for

9

Summary Judgment Regarding the Invalidity of U.S. Patent No. 7,469,381 (“Van Dam

10

Declaration”). I have been asked to provide opinions as to whether claim 19 of the ’381 patent is

11

valid and to address the Van Dam Declaration. My opinions are set forth below in this

12

declaration and in the accompanying exhibits.

13

2.

I reserve the right to supplement or amend this declaration if additional data or

14

other information that affects my opinions becomes available. I may testify at a hearing or at trial

15

regarding the matters expressed in this declaration and any supplemental declarations that I may

16

prepare for this litigation. I also may prepare and rely on audiovisual aids to demonstrate various

17

aspects of my testimony at a hearing. I also may testify with respect to any matters addressed by

18

any expert testifying on behalf of Samsung, if asked to do so.

19

3.

I am being compensated at my standard consulting rate of $430 per hour for my

20

work in connection with this action. My compensation is not based in any way on the outcome of

21

the litigation.

22

4.

I hereby incorporate by reference the Rebuttal Expert Report of Ravin

23

Balakrishnan, Ph.D. Regarding Validity of U.S. Patent No. 7,469,381, submitted on April 16,

24

2012.

25

I.

26
27

QUALIFICATIONS
5.

Here, I provide a brief summary of my qualifications. My qualifications are stated

more fully in my curriculum vitae, which is attached to this report as Exhibit 1.

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

1

1

6.

I earned my B.Sc. (1st Class Honours) degree in computer science from the

2

University of New Brunswick, Canada, in May 1993. Subsequently, I received my M.Sc. and

3

Ph.D. degrees in computer science from the University of Toronto, Canada, in January 1997 and

4

February 2001, respectively.

5

7.

As an undergraduate, I worked as a research assistant in the human interface lab,

6

working with different kinds of novel input technologies, including touch input systems for three

7

dimensional data interaction. Since then, I have either trained or worked in the field of human-

8

computer interfaces, including interfaces for touch sensitive input devices, multi degree-of-

9

freedom input devices, two-handed input, multi-touch input, haptic feedback interfaces, tablet-

10
11

based input, large and small scale displays, and interactive 3D graphics.
8.

I have published over one hundred refereed publications in the field of human-

12

computer interaction. I have further presented numerous conference abstracts, posters, talks, and

13

demonstrations in my field. I am a named inventor on fourteen issued patents in my area of work,

14

plus an additional seven pending (though not yet issued) patents.

15

9.

I joined the University of Toronto faculty in July 2001 as an Assistant Professor.

16

In 2006, I was promoted to Associate Professor with tenure, and in 2011 was promoted to full

17

Professor. As a professor, I have taught numerous undergraduate and graduate courses in topics

18

related to human-computer interaction. Ten Ph.D. students and twenty research masters students

19

have completed their degrees and research under my supervision, and seven postdoctoral fellows

20

have completed their research training under my supervision. In addition to these graduate

21

students and postdoctoral fellows, I currently supervise one postdoctoral fellow, six Ph.D.

22

students, and two Masters students. In addition to my professorship, I also hold the Canada

23

Research Chair in Human-Centered Interfaces in the Department of Computer Science, and I co-

24

direct the Dynamic Graphics Project laboratory.

25

10.

My research at The University of Toronto has involved nearly every broad aspect

26

of human-computer interaction and data visualization. For instance, I have done significant work

27

in the areas of input devices, sensing technologies, and interaction techniques, in particular touch

28

and multi-touch interaction, gestural, sketching, and multi degree-of-freedom interaction,
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

2

1

interfaces to small and/or mobile computers, and interfaces to displays of the future. As another

2

example, I have done work in the evaluation of user interfaces, including associated metrics and

3

predictive models of human performance. I have previously served as a visiting researcher at

4

Mitsubishi Electric Research Laboratories. My research program has been funded by leading

5

companies such as Microsoft, IBM, and Hewlett-Packard and also organizations such as the

6

National Sciences and Engineering Research Council of Canada and also the Sloan Foundation.

7

11.

I have also served on the organizing and paper reviewing committees of many

8

leading conferences in my field, and have taken on editorial roles for leading technical journals in

9

fields pertinent to my research. For example, I am currently an Associate Editor of “ACM

10

Transactions on Computer-Human Interfaces” (the premier journal in the field), and until recently

11

was an Associate Editor of the journal “IEEE Transactions on Visualization and Computer

12

Graphics.” Similarly, I have been the Papers Chair for the ACM UIST Symposium on User

13

Interface Software and Technology, and have served multiple times as an Associate Chair for the

14

premier ACM CHI Conference on Human-Computer Interaction.

15

12.

I have also received major awards and honors in my field, including:

16



Alfred P. Sloan Research Fellowship.

17



Nine best paper awards and honorable mentions at the leading conferences
in my field.



Ontario Premier’s Research Excellence Award, which included a $100,000
research grant.



Election to the ACM SIGCHI Academy in 2011, which honors the
principal leaders in the research field of human-computer interaction.

18
19
20
21
22

13.

As set forth in my CV, I have over twenty years of experience studying and

23

teaching computer programming. I have been a professor of computer science for over ten years.

24

I can read and program using both procedural and object-oriented programming languages

25

fluently, including the C, C++, Objective C and Java languages.

26

14.

I have previously testified as an expert during administrative proceedings before

27

the International Trade Commission and by deposition in connection with those same

28

proceedings. I have also submitted reports or testified in connection with proceedings before
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

3

1

United States District Courts and one foreign court. Specifically, I have submitted reports or

2

testified in:

3



4
5

In re Certain Electronic Digital Media Devices and Components Thereof, Inv. No.
337-TA-796, on behalf of complainant Apple.



In re Certain Electronic Devices, Including Wireless Communication Devices,

6

Portable Music and Data Processing Devices, and Tablet Computers, Inv. No. 337-

7

TA-794, on behalf of respondent Apple.

8



9
10

behalf of complainant Apple.


11
12

In re Certain Mobile Devices, and Related Software Thereof, Inv. No. 337-TA-750, on

In re Certain Video Game Machines and Related Three-Dimensional Pointing
Devices, Inv. No. 337-TA-658, on behalf of respondent Nintendo.



In re Certain Electronic Devices With Multi-Touch Enabled Touchpads And

13

Touchscreens, Inv. No. 337-TA-714, on behalf of respondent Apple. During that

14

proceeding, the parties stipulated, and Chief Administrative Law Judge Paul J.

15

Luckern acknowledged, that I was an expert in the field of computer user input

16

devices.

17



18
19

on behalf of complainant Apple.


20
21



26

Apple Inc. v. Samsung Electronics Co. Ltd., et al., 11-cv-1846-LHK (N.D. Cal.) on
behalf of plaintiff Apple.



24
25

Apple Inc. v. Samsung Electronics Co. Ltd., et al., 12-cv-630-LHK (N.D. Cal.) on
behalf of plaintiff Apple.

22
23

In re Certain Portable Electronic Devices and Related Software, Inv. No. 337-TA-797,

Mobilemedia Ideas LLC v. Apple Inc., 10-cv-258 (D. Del.) on behalf of defendant
Apple.



Apple Inc. v. Samsung Electronics Co. Ltd., et al., KG ZA 11-730 and KG ZA 11-731
(District Court of the Hague).

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

4

1

II.

MATERIALS CONSIDERED
15.

2

In forming my opinions and views expressed in this declaration, I reviewed (1) the

3

’381 patent, its prosecution file history, and the file history for Reexamination Application No.

4

90/090,963; (2) the file history of U.S. Provisional Patent Application No. 60/883.801; (3)

5

portions of the deposition transcript of Bas Ording, the named inventor of the ’381 patent, as well

6

as certain exhibits marked during that deposition; (4) the Van Dam Declaration with exhibits; (5)

7

the declarations, with exhibits, of Clifton Forlines and Adam Bogue in support of Samsung’s

8

Motion for Summary Judgment; (6) the deposition transcripts, and certain exhibits marked during

9

those depositions, of Andries Van Dam, Clifton Forlines, and Adam Bogue; (7) Samsung’s Patent

10

Local Rule 3-3 and 3-4 Disclosures; (8) the Expert Report of Jeffrey Johnson, Ph.D. Regarding

11

Non-Infringement of U.S. Patent No. 7,469,381 with exhibits; (9) the Order Construing Disputed

12

Claim Terms of U.S. Patent Nos. 7,698,711; 6,493,002; 7,469,381; 7,663,607; 7,812,828;

13

7,844,915; and 7,853,891 (Dkt. No. 849); and (10) the Order Denying Motion for Preliminary

14

Injunction (Dkt. No. 452).
16.

15

I also reviewed portions of the Mitsubishi production from the hard drive labeled

16

MERL00000001, including source code for the Tablecloth program (“Tablecloth”), as well as a

17

DiamondTouch system running Tablecloth. Attached hereto as Exhibits 2 and 3 are videos

18

demonstrating certain functionality of the Tablecloth program.

19

II.

20

LEGAL PRINCIPLES
17.

I have not been asked to offer an opinion on the law; however, as an expert

21

assisting the Court in determining validity, I understand that I am obliged to follow existing law.

22

Attorneys for Apple have informed me of a number of legal principles, and my opinions in this

23

report take into account my understanding of those principles.

24

18.

I have been informed by counsel regarding the standards for invalidity. I have

25

been informed by counsel that a patent claim is invalid if it is “anticipated” or “obvious” in view

26

of the “prior art.”

27

19.

28

I have been informed by counsel that a patent is presumed valid, and each patent

claim is independently presumed valid, even if other claims within the patent are held invalid. I
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

5

1

have been informed by counsel that the burden of proving invalidity rests on the person

2

challenging the patent, who must demonstrate that it is anticipated or obvious by clear and

3

convincing evidence. I have been informed by counsel that “clear and convincing” evidence is

4

evidence indicating that the thing to be proved is highly probable or reasonably certain.

5

A.

Anticipation

6

20.

I have been informed by counsel that a claimed invention is invalid if it is

7

anticipated by a single prior art reference. I have been informed by counsel that a prior art

8

reference anticipates a patent claim if each and every limitation of that claim is found, either

9

expressly or inherently, in that single prior art reference. I have been informed by counsel that a

10

claim limitation is inherent in the prior art if it is necessarily present in the prior art, not merely

11

probably or possibly present. I have been informed by counsel that, to anticipate, there must be

12

no difference between the claimed invention and the reference disclosure, as viewed by a person

13

of ordinary skill in the field of the invention. I have been informed by counsel that anticipation

14

requires that the disclosure in the prior art reference be sufficient to enable one skilled in the art to

15

carry out the claimed invention.

16

21.

I also understand that a claim is invalid under 35 U.S.C. §102 (a) if the claimed

17

invention was known or used by others in the U.S., or was patented or published anywhere,

18

before the applicant’s invention. I further understand that a claim is invalid under 35 U.S.C. §102

19

(b) if the invention was patented or published anywhere, or was in public use, on sale, or offered

20

for sale in this country, more than one year prior to the filing date of the patent application. And a

21

claim is invalid, as I understand, under 35 U.S.C. §102 (e), if an invention described by that claim

22

was described in a U.S. patent granted on an application for a patent by another that was filed in

23

the U.S. before the date of invention for such a claim. A claim is also invalid, as I understand,

24

under 35 U.S.C. §102 (f) if the invention was invented by another prior to the claimed invention.

25

It is also my understanding that a claim is invalid under 35 U.S.C. §102 (g)(2) if, prior to the date

26

of invention for the claim, the invention was made in the U.S. by another who had not abandoned,

27

suppressed or concealed the invention.

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

6

1

B.

Obviousness

2

22.

I have been informed by counsel that a claimed invention is only unpatentable

3

under 35 U.S.C. § 103 if the differences between the invention and the prior art are such that the

4

subject matter as a whole would have been obvious at the time the invention was made to a

5

person having ordinary skill in the art to which the subject matter pertains.

6

23.

I am informed and understand that certain factors must be evaluated to determine

7

if a patent claim is obvious. These factors include: (1) the scope and content of the prior art; (2)

8

the differences between each claim of the patent and the prior art; (3) the level of ordinary skill in

9

the art at the time the claimed invention was made; and (4) “secondary considerations” of non-

10
11

obviousness.
24.

I understand that a claim of obviousness may be based on one or more references,

12

taken in combination. I understand that a patent composed of several elements is not proved

13

obvious merely by demonstrating that each of its elements was known in the prior art. There

14

must be a reason for combining the elements in the manner claimed. That is, there must be a

15

showing that a person of ordinary skill in the art at the time of the invention would have thought

16

of either combining two or more references or of modifying a reference to achieve the claimed

17

invention. It is not sufficient to show that it was obvious to try a combination.

18

25.

In determining whether an invention is obvious, I understand that it is

19

impermissible to engage in hindsight reconstruction of the claimed invention, using the

20

applicant’s invention as a template and selecting elements from the references to fill the gaps. In

21

order for a combination of multiple references to be obvious, a person of ordinary skill in the art

22

should have some reason to combine the references. When considering a reference for purposes

23

of an obviousness analysis, the reference must be taken for everything it teaches, including

24

information that that diverges from or teaches away from the claimed invention.

25

26.

I also understand that a combination of known elements can be obvious when it

26

does no more than yield predictable results. In other words, where it is obvious to try a particular

27

combination of known elements to solve a problem and there are a finite number of known,

28

predicable solutions, the result is likely the product not of innovation but of ordinary skill and
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

7

1

common sense. At the same time, a finding of obviousness may not be proper where the prior art

2

merely provides a person of ordinary skill in the art a promising field for experimentation. I have

3

further been informed that a proper obviousness analysis focuses on what was known or obvious

4

to a person of ordinary skill in the art, not just to the patentee, at the time of the invention. I also

5

understand that practical and common sense considerations should guide a proper obviousness

6

analysis.

7

27.

I also understand that the law distinguishes between one of ordinary skill in the art

8

and inventors. Under this distinction, one should not go about determining obviousness by

9

inquiring into what patentees or inventors would have known or would likely have done faced

10

with the revelation of references. A person of ordinary skill in the art is one who thinks along the

11

lines of conventional wisdom and is not one who undertakes to innovate.

12

28.

I have been informed by counsel that secondary considerations of non-obviousness

13

should be considered and include: (1) commercial success of the claimed invention; (2) long-felt

14

but previously unsolved needs for the claimed invention; (3) copying of the invention by others in

15

the field; (4) initial expressions of disbelief or skepticism by experts in the field; (5) praise or

16

industry acclamation for the claimed invention; and (6) failure of others to solve the problem that

17

the inventor solved.

18

C.

Invention and Patent Application Dates

19

29.

I understand that there are several significant dates that are relevant to my analysis.

20

The first is the date of conception. Specifically, an invention is complete when the inventor has

21

formed a definite and permanent idea of the complete and operative invention, as it is to be

22

applied in practice. I understand that conception must include every feature or limitation of the

23

claimed invention.

24

30.

A second significant date is that of reduction to practice. I understand that there are

25

two types of reduction to practice. An actual reduction to practice requires that the inventor

26

constructed an embodiment or performed a process that met all the limitations of the claim that

27

would work for its intended purpose. A constructive reduction to practice is the filing of a patent

28

application. I understand that for a patentee to be entitled to rely upon a conception date as of the
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

8

1

date of invention for purposes of a prior art analysis, he or she must have been reasonably diligent

2

from conception through reduction to practice.

3

31.

The filing date of a patent is the date that the application for the patent was filed

4

with the United States Patent and Trademark Office (“PTO”). That date is printed on the first

5

page of the patent. I understand that, to claim the benefit of the date of an earlier patent

6

application, the earlier application must disclose and support the subject matter of the claims.

7

32.

I understand that the “critical date” for a patent is one year before its priority date.

8

A.

Claim Construction

9

33.

I understand that the Court construed the term “edge of [an or the] electronic

10

document” to have its plain and ordinary meaning, and that the term is not limited to “only an

11

external edge,” and “may be internal.” (Dkt. No. 849 at 23.) The Court also declined to adopt

12

“boundary” as a substitute for the word “edge.” (Id. at 20.) I have applied the Court’s

13

construction in coming to my opinions about the validity of claim 19 of the ’381 patent.

14

34.

I also understand that the Court ruled in its order on Apple’s Motion for a

15

Preliminary Injunction that “Claim 1 of the ’381 patent is fatalistic: if a user scrolls past the edge

16

of an electronic document in the first direction, the screen must snap back to that document when

17

the user lifts her finger.” (Dkt. No. 452 at 60.) I have performed my analysis under the Court’s

18

interpretation of claim 1 as well as under an alternative interpretation that claim 19 does not

19

prohibit behavior other than the rubber banding functionality.

20

35.

For those claim terms for which the parties did not dispute their construction, I

21

have interpreted the claims as one of ordinary skill in the art would have at the time the patent

22

was filed in light of the teachings of the patent and its prosecution history, which may limit claim

23

scope, either affirmatively or by implication.

24

III.

DETAILED OPINION

25

A.

Background of the ’381 Patent

26

36.

U.S. Patent no. 7,469,381 (Arnold Decl. Ex. 83) is titled List Scrolling and

27

Document Translation, Scaling and Rotation on a Touch-Screen Display. The filing date of the

28

patent application (Application No. 11/956,969) is December 14, 2007, and its date of issue is
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

9

1

December 23, 2008. There are a number of related patent applications to which the ’381

2

application claims priority, including U.S. Provisional Patent Application No. 60/883,801, filed

3

on January 7, 2007. (Bartlett Decl. Ex. 55.) That application contains either verbatim or near

4

verbatim the same written description, figures, and claim language as the ’381 patent, and

5

accordingly fully supports and enables claim 19 of the ’381 patent.

6
7
8
9

37.

I understand that the critical date for the claims of the ’381 patent is January 7,

2006, one year before the filing date of the first provisional application.
38.

I have reviewed portions of the deposition transcript of Bas Ording, the named

inventor of the ’381 patent, and understand that Mr. Ording conceived of his invention in early

10

February 2005, and reduced it to practice in a prototype by February 11, 2005. (Bartlett Decl. Ex.

11

56 [8/9/11 Ording Dep.] at 126:3 – 130:7.)

12

39.

I understand that on April 28, 2010, a Request for Reexamination was filed at the

13

request of Nokia Corporation (see Dkt. No. 87-40), which was then involved in a patent

14

infringement lawsuit with Apple regarding the ’381 patent, with the Patent Office, asserting that a

15

substantial new question of patentability existed in light of certain patents and printed

16

publications. (Bartlett Decl. Ex. 57.) On July 14, 2010, the Patent Office granted this request for

17

ex parte reexamination. On January 13, 2011, the Patent office issued a Notice of Intent to Issue

18

Ex Parte Reexamination Certificate, and confirmed that the identified patents and printed

19

publications, “either singularly or in combination fail to teach or suggest, ‘in response to

20

detecting that the object is no longer on or near the touch screen display, translating the electronic

21

document in a second direction until the area beyond the edge of the electronic document is no

22

longer displayed to display a fourth portion of the electronic document, wherein the fourth portion

23

is different from the first portion.’” (Bartlett Decl. Ex. 58 [APLPROS0000019626-32].) On

24

April 26, 2011, the Patent Office issued an Ex Parte Reexamination Certificate in which the

25

patentability of all of the claims of the ’381 patent was confirmed. (Bartlett Decl. Ex. 59

26

[SAMNDCA00000030-31].)

27
28

40.

The ’381 patent relates to translation of an electronic document on a touch screen

display in response to a user’s movement of an object, such as the user’s finger, on or near the
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

10

1

touch screen. (See Arnold Decl. Ex. 83 [’381 patent] at Abstract.) The ’381 patent generally

2

claims an innovative method of informing the user of a touch screen mobile device that the edge

3

of an electronic document has been reached by allowing the user to scroll beyond the edge of the

4

document and to view an area beyond the edge of the document for as long as the user keeps his

5

finger in contact with the screen. Once the user’s finger is removed, the ’381 patent describes

6

having the document or image scroll back into place so that the area beyond its edge is no longer

7

shown, and the document or image can be viewed.

8
9

41.

An overview of the invention is depicted in Figures 8A-8D of the patent, which

show the ’381 patent’s “rubber banding” feature in action:

10
11
12
13
14
15
16
17

42.

This invention provides an elegant and appealing form of visual feedback to a user

18

that there is no more of a document to be seen. For example, if a user is zoomed in on one part of

19

a large photo, he may continue to scroll the photo as he looks at other parts of the image. Not

20

knowing exactly where the photo ends, he may continue to scroll in a direction even when there is

21

no more of the photo to display. When this happens, an area beyond the edge of the photo will be

22

displayed, and once the user lifts his finger, the photo will “bounce” or “rubber band” back until

23

the area beyond the edge is no longer visible. This form of visual feedback is readily understood

24

and makes clear to the user that he cannot continue to scroll in that direction.

25

43.

This visual feedback also provides an intuitive solution to a vexing user interface

26

issue: what to do when a user scrolls to the edge of an electronic document. In the prior art, when

27

a user scrolled to the edge of a document, one of two scenarios would play out. Either she would

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

11

1

scroll continuously past the edge of the document into nothingness (i.e. beyond a place where

2

there was any meaningful content), or she would hit a “hard stop” and not be allowed to scroll

3

any further.

4

44.

Each of these scenarios has its own disadvantages. Allowing a user to move

5

through virtual space going absolutely anywhere, including beyond a place that has any

6

meaningful content, can cause the user to become disoriented. (See Bartlett Decl. Ex. 60 [9/17/11

7

Bederson Dep. Ex. 222] at 4; Bartlett Decl. Ex. 61 [9/17/11 Bederson Dep.] at 204:6-24; 205:6-

8

207:5; 213:2-214:1.) In a paper he wrote in 2011, Benjamin Bederson referred to this as the

9

“Desert Fog” phenomenon, citing an earlier article written by Jul and Furnas. (Id.; see also

10

(Bartlett Decl. Ex. 62 [9/14/11 Van Dam Dep.] at 63:3-17 (referring to the empty area as “no

11

man’s land”).) Users who navigate into these empty spaces may get lost and not know how to

12

find their way back. (Id.; see also Van Dam Decl. (Dkt. No. 168) ¶ 144.)

13

45.

Most user interfaces avoided the “Desert Fog” problem by inserting a hard stop at

14

the edge of a document. But that solution has its own disadvantages. If the user does not realize

15

he has hit the edge of a document, he may keep trying to move the document in vain. No matter

16

how hard he tries, however, the device will not allow the document to move. As a result, the user

17

may think his device has frozen or locked up, or that it is otherwise not registering his input. In

18

any case, the user could become frustrated when the scrolling or translating does not reflect his

19

intent. (Arnold Decl. Ex. 83 [’381 Patent] at 2:26-28; see also Van Dam Decl. (Dkt. No. 168) ¶

20

144 (one way to prevent a user from moving an electronic document beyond the edge is “to

21

prevent the document from moving beyond the edge by ignoring further requests for any such

22

movement”).)

23
24
25

46.

I reserve my right to discuss the general background of the technology and user

interfaces that existed at the time of the invention of the ’381 patent.
47.

The inventor of the ’381 patent recognized these disadvantages and created a novel

26

solution to overcome them. By displaying an area beyond the edge of an electronic document,

27

the invention of the ’381 patent provides the user with an instant visual cue informing him that

28

the edge of the document has been reached, and importantly, in an exemplary embodiment shown
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

12

1

in Figures 8A-8D of the patent, this area beyond the edge is displayed adjacent to a portion of the

2

electronic document, enabling the user to maintain context and avoid the “Desert Fog” problem.

3

48.

The elegant solution proposed by the ’381 patent significantly enhances the user’s

4

experience in viewing photos, web pages, lists, and other electronic documents. To my

5

knowledge, touch screen devices prior to the Apple iPhone did not have a visually intuitive way

6

to alert a user when he or she had reached the edge of the document when scrolling or panning.

7

Now, this feature is nearly ubiquitous, including in Samsung’s own devices. Accordingly, the

8

inventions of the ’381 patent make possible a user interface that is more visually appealing and

9

intuitive in its handling of the display of electronic documents.

10

49.

The hardware and structural components on which the instructions for performing

11

the “bounce” or “rubber banding” functionality claimed in the ’381 patent are disclosed, for

12

example, in Figure 17 of the patent, as well as the text in columns 34:47 – 35:19. There, a device

13

with a touch-screen display, a central processing unit, memory, and communication buses is

14

described. Flow charts of exemplary algorithms for performing the aforementioned

15

functionalities can be found, for example, in Figures 5 and 7 of the patent, along with the

16

accompanying text.

17

B.

Person of Ordinary Skill in the Art

18

50.

If called to testify on the topic of the definition of a person of ordinary skill in the

19

art for the ’381 patent, I expect to testify regarding the skill, education, and experience that a

20

person of ordinary skill in the relevant art would have had at the time of the invention of the ’381

21

patent. In my opinion, and as submitted by Apple in a January 19, 2012 Joint Statement (Dkt.

22

No. 650), a person of ordinary skill in the relevant art of the ’381 patent at the time of the

23

invention would have a Bachelor’s degree in computer science or electrical engineering, or the

24

equivalent, and one or more years experience working on designing and/or implementing user

25

interfaces.

26

51.

27

I would have met the criteria for being such a person of ordinary skill in the art at

the time of the invention of the ’381 patent.

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

13

1

C.

Validity of the ’381 Patent

2

52.

The Van Dam Declaration limits itself to a discussion of a single software

3

application named Tablecloth. The Tablecloth program refers to a software program that uses the

4

DTFlash library and runs on a Mitsubishi Electric Research Laboratories (“MERL”)

5

DiamondTouch system. The DiamondTouch system included a number of components including

6

a touch sensing table, an overhead projector, and pads on which users sat. A typical configuration

7

for the system is depicted below.

8
9
10
11
12
13
14

(Bogue Decl. Ex. 1 at SAMNDCA00035802.)
53.

15
16
17
18

researcher at MERL. To my recollection, there was a DiamondTouch system in the lobby of the
research lab, and a key card was necessary to access that area. I was required to sign a nondisclosure agreement before working in the research lab.
54.

19
20
21
22
23
24
25
26

I am familiar with the DiamondTouch system based on my time as a visiting

The designers of Tablecloth created a program that auto-centers on an image

regardless of the distance it is scrolled, and regardless of whether an edge of the document has
been reached. The Tablecloth program was not designed to display an area beyond the edge of a
document. Below are two screen captures from the Tablecloth program in which no additional
scrolling is possible after pulling the image down as far as it will go (the mouse crosshairs
towards the top right in the first image (circled in red) indicates the starting point for the drag
downwards, and the crosshairs towards the bottom right (circled in red) indicate the ending
point):

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

14

1
2
3
4
5
6
7
8
9
Fig. 1

10
11

55.

Fig. 2

In the Tablecloth program, individuals are not allowed to scroll past the borders of

12

the repeated images of the Windows desktop. Attempting to do so will result in a hard stop. In

13

the examples provided in the Van Dam Declaration, a user drags her finger over the Internet

14

Explorer toolbar region and past the Internet Explorer program window, which does not comport

15

with the intended operation of the program. As discussed below, the DiamondTouch system

16

described in the Van Dam Declaration was configured in a particular manner so that the window

17

containing the Tablecloth program was minimized and did not fill the entire DiamondTouch

18

system touch surface.

19

56.

I have reviewed Dr. Van Dam’s discussion of the Tablecloth reference and

20

disagree with his conclusions regarding that program. I also believe that his discussion of

21

Tablecloth’s functionality is incomplete.
1.

22
23

57.

The Tablecloth program is not equivalent to DTFlash

There is considerable ambiguity in the way that the Tablecloth program is

24

described in the Van Dam Declaration. References to the program conflate the single Tablecloth

25

program with DTFlash as “Tablecloth/DT Flash.” (See, e.g., Van Dam Decl. at ¶ 51.) This

26

conflation is misleading to the extent that the Van Dam Declaration sets forth arguments and

27

opinions relating to DTFlash separate from the Tablecloth program, which I understand is the

28

only program written for DTFlash that has been asserted as prior art. Similarly, I note that the
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

15

1

article about DTFlash attached to the Bogue Declaration as Exhibit 2 does not mention

2

Tablecloth.

3

58.

DTFlash is not, as suggested in the Van Dam Declaration, a specific software

4

application. Rather, it is a software library. Therefore, it is inaccurate to say that the DTFlash

5

libraries are a prior art reference.

6

2.

7
8
9

59.

The Van Dam Declaration does not demonstrate that the Tablecloth
program was used in public, publicly known, or commercially sold
before the critical date for the ’381 patent

The Van Dam Declaration states that “the DiamondTouch system was publicly

available running Tablecloth by at least by [sic] January 6, 2006, before the earliest possible

10

critical date of the ’381 patent, and is therefore prior art to the ’381 patent.” (Van Dam Decl. ¶

11

50.) The basis for this assertion is attributed to “the declaration of and phone conversations with

12

Adam Bogue,” yet this date does not appear in the Bogue Declaration. (Id.)

13

3.

14
15

60.

The DiamondTouch system required precise calibration in order to
function as intended

The DiamondTouch system on which the Tablecloth application was operated in

16

the video exhibits accompanying the Bogue Declaration required a very specific configuration,

17

any deviation from which could have led to different results. I note that in the photographs and

18

videos in Dr. Van Dam’s earlier expert report, it was impossible to see the set up of the

19

DiamondTouch system being discussed, and that the setup discussed in the Van Dam Declaration

20

may be different.

21

61.

The Van Dam Declaration depicts the laptop, projector, and touch table

22

components to the DiamondTouch system in paragraph 110. An image from the laptop is

23

projected onto the touch sensing table from a perched projector. If the projector were suspended

24

too far above the table, the projected image would exceed the dimensions of the table. If it were

25

suspended too close to the table, the projected image would be smaller than the dimensions of the

26

table, leaving an empty border region around the projected image. To my understanding, the

27

DiamondTouch was designed to have the projector set at a height where the projected image

28

would fill the available table space.
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

16

1

62.

The DiamondTouch system needed to be calibrated to function properly, and the

2

projected image was “mapped” to the table by pressing on certain highlighted points. In the

3

image below, the green square indicates one of these mapping points.

4
5
6
7
8
9

Fig. 3

10

63.

If the projected image and the table were not properly aligned, the DiamondTouch

11

system would not function as intended. Adam Bogue, a MERL employee, testified that during

12

demonstrations of the DiamondTouch system, one of the “[t]hings that could go wrong with the

13

DiamondTouch” was that the table could be “bumped.” (Bartlett Decl. Ex. 63 [3/9/12 Bogue

14

Dep.] at 104:18-105:10.) If the table were bumped, the solution was to “realign the projected

15

image onto the surface.” (Id.)

16

64.

I personally have had extensive exposure to the DiamondTouch system, both in

17

my time at MERL and in academia. In all that time, I have never seen anyone deliberately

18

calibrate the projected image to be smaller than the touch sensitive area of the DiamondTouch

19

table.

20

4.

21
22

65.

The “electronic document” in the Tablecloth program differs in the
Van Dam Declaration, Dr. Van Dam’s Expert Report, and Samsung’s
Invalidity Contentions

Dr. Van Dam’s declaration does not make clear what the “electronic document” in

23

the Tablecloth program is. I note that Samsung previously represented in its Invalidity

24

Contentions that the electronic document in the Tablecloth program was a picture of a Windows

25

desktop showing a green meadow and blue sky with clouds. (Bartlett Decl. Ex. 64 [Invalidity

26

Contentions Ex. G-7] at 2].) As seen below on the left, the electronic document included the

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

17

1

green “Start” button of the desktop, as well as the grey bar underneath with the word “Done” in it.

2

In the next figure, Samsung omitted the grey bar from the electronic document.

3
4
5
6
7
8
9
10

Fig. 4

66.

Fig. 5

Samsung then went on to represent that the portion of the desktop with the green

11

“Start” button was the edge of the electronic document, and that a second instance of the original

12

image was the “area beyond the edge.” (Id. at 6.)

13
14
15
16
17
18
19
20
21
22

Fig. 6

67.

In his expert report, Dr. Van Dam departed from Samsung’s position and

23

contended that the electronic document extends beyond what is visible on the table when the

24

desktop image is at rest, and actually comprises two identical instances of the same image. (See

25

Bartlett Decl. Ex. 65 [Van Dam Expert Report Ex. 8] at 4.)

26

i.

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

18

1
2
3
4
5
6
7
8
Van Dam Expert Report Ex. 8 at 4

9
10

68.

Finally, the Van Dam Declaration goes beyond the disclosure in Dr. Van Dam’s

11

expert report and claims that the electronic document can be either (1) what is visible when the

12

Tablecloth application is at rest, or (2) two copies of the same image.

13
14
15
16
17
18
19
20
(1) Van Dam Decl. Ex. 4 at 6

21
22
23
24

69.

(2) Van Dam Decl. Ex. 3 at 7

These arbitrary identifications of what constitutes the electronic document in Dr.

Van Dam’s representative examples are inconsistent.
70.

First, I note that Dr. Van Dam’s identification of the electronic document in

25

example (1) is different from Samsung’s identification in Figure 4, above, which included the

26

grey window bar just below the dotted line that Dr. Van Dam uses to demarcate his electronic

27

document.

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

19

1

71.

Though Dr. Van Dam claims to have applied Samsung’s definition of electronic

2

document – “information that is visually represented on a screen that has a defined set of

3

boundaries” (Van Dam Decl. Ex. 4 at 2) – this electronic document includes an arbitrary portion

4

of the second copy of the desktop image.

5

72.

In the figure on the left, one can already see part of the cloud layer from the copy

6

of the document below the central image, and in the figure on the right, one can see part of the

7

green “Start” button from the copy of the document above the central image and under the grey

8

bar with “DTIEFlash” in it.

9
10
Fig. 7

11

Fig. 8

12

In other words, far from having a defined set of boundaries, what Dr. Van Dam claims is the

13

electronic document in this example includes portions of another copy of the desktop image.

14

73.

Second, Dr. Van Dam’s identification of the electronic document in example (2) is

15

also arbitrary. According to Clifton Forlines’ testimony, on which Dr. Van Dam relied, there are

16

two copies of an image that can be displayed in Tablecloth:

17
18

Q. Now, in the TableCloth application, when the user scrolls up, is
there a new copy of the image that was previously not on the
screen?

19

...

20

A. When the user scrolls up, a second copy of the image is
displayed below the original image.

21
22
23

(Bartlett Decl. Ex. 66 [3/8/12 Forlines Dep.] at 109:2-8.)
74.

In other words, the image of the Windows desktop alone depicted in the figures

24

above may qualify as an electronic document, with what appears when a user scrolls this image

25

up or down qualifying as a second electronic document. Dr. Van Dam offered no explanation of

26

why one copy of an image and a second copy of the same image should be considered together as

27

a single electronic document. In the same way that two copies of the same photograph in a photo

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

20

1

gallery would not be considered one electronic document, a person of ordinary skill in the art

2

would not have considered the second copy of the Windows desktop image to be part of the same

3

electronic document with the first copy.

4

75.

Accordingly, neither of the inconsistent theories set forth in either Samsung’s

5

Invalidity Contentions or in the Van Dam Declaration properly identifies an electronic document

6

for purposes of conducting an invalidity analysis.

7
8
9

5.
76.

The DiamondTouch system does not have a touch screen display

A person of ordinary skill in the art would not consider a touch-sensitive table with

an image projected on to it to be a touch screen display, as required by claim 19 of the ’381

10

patent. A touch screen display connotes a display screen such as an LCD or LED that is

11

integrated with the capability of sensing touch input from a user. Simply projecting an image

12

onto a touch sensor does not make it a touch screen display. For example, one of skill in the art

13

would simply not consider the DiamondTouch system with an image projected on it to be a touch

14

screen any more than they would consider a laptop’s touchpad with an image projected onto it to

15

be a touch screen. For this reason alone, it is my opinion that the Tablecloth program does not

16

anticipate or render obvious this claim.

17

77.

The Van Dam Declaration largely glosses over the fact that the DiamondTouch

18

system does not have a touch screen display. Dr. Van Dam states that it would have been obvious

19

to combine the DiamondTouch system with a display device such as an LCD touchscreen, but

20

provides no support for this assertion. (Van Dam Decl. ¶ 65.) Indeed, during his deposition,

21

Clifton Forlines, who was involved with the DiamondTouch system, admitted that he had never

22

seen the implementation of a program utilizing the DTFlash libraries on a touch-sensitive surface

23

overlaying a display. (See Bartlett Decl. Ex.66 [3/8/12 Forlines Dep.] at 51:4-8.)

24
25

6.
78.

Dr. Van Dam’s opinion requires a particular motion

Dr. Van Dam’s analysis requires a user to first scroll the image in a first direction

26

prior to commencing movement in the opposite “first direction” to attempt to meet the limitations

27

of the ’381 patent’s claims. Dr. Van Dam offers no explanation why any user would choose to do

28

this particular movement.
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

21

1

7.

The Tablecloth program does not display an area beyond the edge of
an electronic document in response to an edge being reached

2
3

79.

The Tablecloth program does not display an area beyond the edge of an electronic

4

document in response to an edge being reached. As noted above, when run on a properly

5

calibrated system, the Tablecloth program implements a hard stop that prevents a user from

6

scrolling beyond the edge of the electronic document. (See Ex. 3, attached hereto.)

7

80.

In Dr. Van Dam’s representative example (2) where the electronic document

8

appears to be just the first instance of the Windows desktop image, an area beyond the edge of

9

that image is already displayed before any user interaction.

10
11
Fig. 7

12

Fig. 8

13

In other words, an area beyond the edge of the electronic document is not displayed in response to

14

an edge being reached. Rather, it is already displayed before the user even touches the table.

15

81.

In addition, as noted above, the Van Dam Declaration and Bogue Declaration

16

videos depict very specific calibration and set up of the Tablecloth program. In the Van Dam

17

Declaration, the set up of the DiamondTouch system is depicted and described as being

18

“calibrated properly and . . . behaving in its intended manner.” (Van Dam Decl. ¶ 110.) I note

19

that in Dr. Van Dam’s photo, the Internet Explorer program window is configured in such a way

20

that the Windows desktop image only occupies less than half of the window. This is a different

21

set up from the one that was actually filmed and photographed for the Van Dam and Bogue

22

Declarations.

23
24
25
26
27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

22

1

Annotated Close-Up of Van Dam Declaration ¶ 110

2

Assuming that this configuration shows Tablecloth running on the DiamondTouch when it is

3

“calibrated properly and . . . behaving in its intended manner,” it is clear that the Tablecloth

4

application is drastically different from the ’381 patent, which specifically was invented to

5

prevent this sort of user interface phenomenon.

6

82.

Moreover, the image recenters after any movement by a user, making clear that the

7

display of any other “area beyond the edge” is not in response to an edge being reached, but

8

merely because the image has been moved off center.

9

83.

Accordingly, the Tablecloth program fails to disclose “displaying an area beyond

10

the edge of the document . . . in response to the edge of the electronic document being reached,”

11

as required by claim 19 of the ’381 patent, and so can not anticipate or render obvious that claim.

12

8.

13
14
15

84.

The Tablecloth program does not translate an electronic document in
a second direction until the area beyond the edge of the electronic
document is no longer displayed in response to the edge of the
electronic document being reached

I have reviewed the source code for Tablecloth and have confirmed that there is a

16

function named “snapBack,” and that it auto-centers the initial (or primary) Windows desktop

17

image. In other words, regardless of how little or how far the image is scrolled, or whether

18

anything that the Van Dam Declaration identifies as an “edge” is scrolled beyond, the snapBack

19

function will auto-center on the initial Windows desktop image.

20

85.

In Van Dam Declaration representative example (1), in which he defined the

21

“electronic document” as what is entirely visible while the Tablecloth application is at rest,

22

“snapBack” is not performed in response to the edge of the electronic document being reached –

23

it is triggered if the image is moved off center. In addition, “snapBack” does not translate the

24

electronic document in a second direction until the area beyond the edge is no longer displayed.

25

As can be seen below, the area beyond the edge of the image remains displayed.

26
27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

23

1
2
3

Fig. 7

Fig. 8

4
86.

In Van Dam Declaration representative example (2), in which he defines the

5
“electronic document” as the combined set of two copies of the Windows desktop image,
6
“snapBack” is not performed in response to the edge of the electronic document being reached –
7
it is triggered any time the image is moved off center.
8
87.

In addition, “snapBack” does not translate the electronic document in a second

9
direction “until the area beyond the edge is no longer displayed.” As depicted below, the
10
translation in the second direction does not stop at “until the area beyond the edge of the
11
electronic document is no longer displayed,” which would be from the bottom of the grey space
12
to the green line, but rather continues all the way back to the red line indicating the initial
13
position.
14
15
16
17
18
19
20
21
22
Cropped and Annotated Figures from Van Dam Decl. Ex. 3

23
24

88.

As can be seen from Figure 8C from the ’381 patent, the invention claimed in the

25

‘381 patent is very different from the functionality of Tablecloth. The solution provided by the

26

‘381 patent to a difficult user interface problem was to bounce back until the area beyond the

27

edge of an electronic document was no longer displayed, not to snap all the way back to the

28

center of a document.
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

24

1
2
3
4
5
6
7
8
9
10
11

Annotated Figure 8C from ’381 Patent

89.

The Tablecloth functionality is not the same as that of the ’381 patent, and would

12

not have been an acceptable functionality for photographs, contacts lists, Internet browsers, and

13

productivity documents, as constantly auto-centering after any user contact would have rendered

14

them unusable.

15
16

9.
90.

Tablecloth does not always snap back

I understand that the Court ruled in its order on Apple’s Motion for a Preliminary

17

Injunction that “Claim 1 of the ’381 patent is fatalistic: if a user scrolls past the edge of an

18

electronic document in the first direction, the screen must snap back to that document when the

19

user lifts her finger.” (Dkt. No. 452 at 60.) Tablecloth does not always snap back, and displays

20

two additional behaviors not discussed in the Van Dam Declaration.

21

91.

First, Tablecloth exhibits a “hold still” behavior wherein the image of the

22

Windows desktop holds still and does not move when a user lifts her finger from the

23

DiamondTouch table. Jeffrey Johnson, Samsung’s expert on non-infringement, opined that this

24

behavior does not meet the limitations of the claims of the ’381 patent because there is no

25

translation of the document in a second direction. (Bartlett Decl. Ex. 67 [4/16/12 Johnson

26

Rebuttal Rpt.] ¶ 65.) If Dr. Johnson is correct, then Tablecloth does not meet the limitations of

27

the claims of the ’381 patent. Depicted below is a screen capture from video Exhibit 2 to my

28

declaration, which shows this functionality in action.
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

25

1
2
3
4
5
6
7
8
9
10
11

Fig. 9

(Ex. 2.)
92.

Second, Tablecloth exhibits a “snap forward” behavior wherein the image of the

12

Windows desktop snaps forward in the first direction, rather than in a second direction. This

13

behavior is the same as that seen in the prior art asserted by Samsung in opposing Apple’s

14

preliminary injunction motion, and which the Court analyzed in coming to its decision that those

15

references did not anticipate the claims of the ’381 patent. Depicted below are screen captures

16

from video Exhibit 2 to my declaration, which shows this functionality in action. The Tablecloth

17

application was set up in a minimized window, just as in Dr. Van Dam’s representative examples,

18

rather than in a maximized window that would fill the entire DiamondTouch table.

19
20
21
22
23
24
25
26
27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

26

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Fig. 10

(Id.)

19

10.

20
21

93.

The Tablecloth program does not anticipate or render obvious claim
19 of the ’381 patent

For at least the reasons stated above, claim 19 of the ’381 patent is not anticipated

22

or rendered obvious by the Tablecloth program. In particular, Tablecloth does not include or

23

obviously suggest or teach a combination that would meet the limitations of claim 19. There is

24

nothing in the Van Dam Declaration or exhibits that suggests that there was a reason or

25

suggestion at the time of the invention that would have prompted a person of ordinary skill in the

26

art to combine the elements in the way the claimed invention does.

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

27

1

11.
94.

2

Secondary considerations of non-obviousness

I understand that there are secondary considerations of non-obviousness with

3

respect to the claimed invention that should be considered in determining whether the claimed

4

invention was obvious. These considerations include, among other things: (1) commercial

5

success of the claimed invention; (2) praise or industry acclaim for the claimed invention; (3)

6

initial expressions of disbelief or skepticism by experts in the field; (4) copying; and (5) failure of

7

others.

8

95.

Based on my experience with the general state of the art at the time of the

9

invention of the ’381 patent, I believe that there was nothing like the solution of the ’381 patent

10

before the iPhone. I reserve my right to discuss the general background of the technology and

11

other products in the marketplace at that time.

12

96.

I understand that Terry L. Musika will testify that the claimed inventions of the

13

’381 patent have been commercially successful. In that regard, I have previously set forth in my

14

opening expert report my conclusions that numerous Apple products embody the important

15

inventions of the ’381 patent.

16
17
18
19
20
21
22
23
24
25
26
27
28

97.

I also believe that there has been undisputed praise or industry acclamation for

Apple’s user interface technology as implemented on its iPhone, iPod touch, and iPad products.1
1

Steve Jobs, iPhone Introduction, http://www.youtube.com/watch?v=6uW-E496FXg, at
16:16 – 16:33 (audience reaction and statement “isn’t that cool, do a little rubber-banding up
when I went off the edge?”)
Lev Grossman, “Invention of the Year: The iPhone,” Time, Nov. 1, 2007,
http://www.time.com/time/specials/2007/article/0,28804,1677329_1678542_1677891,00. html;
Engadget, “Ten Gadgets that Defined the Decade,” Dec. 30, 2009,
http://www.engadget.com/2009/12/30/ten-gadgets-that-defined-the-decade/;
Tom Krazit, “Apple’s iPhone Wins Second J.D. Power Award,” April 30, 2009,
http://news.cnet.com/8301-13579_3-10231135-37.html.
David Pogue, “The iPhone Matches Most of Its Hype,” NY Times, June 27, 2007,
http://www.nytimes.com/2007/06/27/technology/circuits/27pogue.html?
pagewanted=1&_r=1&ref=iphone;
Korea JoongAng Daily, “Apple’s iPhone Tops List of Innovative Inventions,” Feb. 18,
2008, http://joongangdaily.joins.com/article/view.asp?aid=2886322; and
(Footnote continues on next page.)
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

28

1

As I discussed in my opening expert report, the inventions of the ’381 patent contributed to the

2

intuitive, elegant user interface that was credited with helping make the iPhone a success.

3

98.

I also believe that there was significant skepticism in the industry that a

4

touchscreen phone without a large number of physical buttons could provide an effective user

5

interface.2 This comes as little surprise given the poor track record of individuals who had

6

previously attempted to solve this problem.

7

99.

For example, the Dr. van Dam previously mentioned in his expert report a number

8

of user interfaces that did not resolve the “Frozen Screen” or “Desert Fog” issues described

9

above. Indeed, Dr. Bederson, one of the developers of the LaunchTile reference, which Samsung

10

has proffered as prior art, even admitted in a 2011 paper that the problems solved by the ’381

11

patent still plagued other user interfaces. (See Bartlett Decl. Ex. 60 [9/17/11 Bederson Dep. Ex.

12

222] at 5 (“It is also clear that the essential problem of getting lost in Desert Fog has not been

13

consistently avoided. Furthermore, it is clear that there is no consistency in the mechanisms that

14

are used to navigate through space”).) He went on to note that LaunchTile, in addition to a

15

number of other user interfaces, did not succeed. (See id. at 3 (“it is fair to say that none of them

16

have been great commercial successes (defined either monetarily or by large numbers of

17

users)”).)

18
19
20
21
22

(Footnote continued from previous page.)

Walter Mossberg & Katherine Boehret, “Testing Out the iPhone, The Wall Street Journal,
June 27, 2007, http://online.wsj.com/articles/SB118289311361649057.html.
2

23
24
25
26
27

Olga Kharif, “Another Music Phone? Yawn . . .”, Bloomberg Businessweek, Oct. 18,
2006, http://www.businessweek.com/technology/content/oct2006/tc20061018_099162.htm
(noting that “Many analysts are skeptical on the appeal of an iPhone”); and
Christopher Meinck, “Palm CEO Remains Skeptical of Apple iPhone”, everythingiCafe,
Feb. 20, 2007, http://www.everythingicafe.com/palm-ceo-remains-skeptical-of-appleiphone/2007/02/20/ (“for businesspeople the touch-sensitive screen without a physical button
keyboard will be a challenge . . . We’ve learned and struggled for a few years here figuring out
how to make a decent phone. PC guys are not going to just figure this out. They’re not going to
just walk in”).

28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

29

1

100.

It is also evident that following Apple’s introduction of the technology of the ’381

2

patent, Samsung quickly saw the merit and commercial value in that technology and planned to

3

and did copy it for use in its products.

4

101.

I have also reviewed a number of documents produced by Samsung in this

5

litigation, including analyses of features in Apple products and email messages. Based on my

6

review of these documents, it appears that Samsung studied a number of Apple products that

7

embody the asserted claims of the ’381 patent, recognized the benefits of the ’381 patent, and

8

implemented the features of the ’381 patent in Samsung products.

9

102.

As just one example, in the document titled “Behold3 Usability Evaluation

10

Results” (Bartlett Decl. Ex. 68 [SAMNDCA00508318 – 508411]), Samsung evaluated its

11

Behold3 phone against Apple’s iPhone. (Id. at SAMNDCA00508331.) This evaluation

12

concluded that Samsung’s “Behold3 [was] shown inferior to Apple’s iPhone in both the task

13

success rate (68.5%) and satisfaction score (86).” (Id. at SAMNDCA00508333.) On a page titled

14

“Aesthetics_Browsing,” the document notes that the iPhone has “a ‘bouncing’ visual effect,”

15

which “generates fun for the user,” while the Behold3 has “no visual effect” when “a web page is

16

dragged to its endpoint.” (Id. at SAMNDCA00508383.) On that page, there is a side by side

17

comparison between the Behold3 and the iPhone, where the rubber-banding feature of the ’381

18

patent is being demonstrated on a web page displayed on the iPhone. (Id.) Specifically, the

19

displayed web page is being pulled to the upper right hand corner, revealing an area beyond the

20

edge of the web page to the left and below. (Id.) The caption notes that “If a web page is

21

dragged to the edge, and the hand is released, a ‘bouncing’ visual effect is provided.” At the

22

bottom of the page, following the column “Direction of Improvement,” is a direction to “Provide

23

a fun visual effect when dragging a web page.” (Id.) Based on the existence of this feature in the

24

Samsung devices I examined, it appears that this instruction was carried out.

25

103.

As another example, in the document titled “P5 Usability Evaluation Results”

26

(Bartlett Decl. Ex. 69 [SAMNDCA00176053 – 176171]), Samsung evaluated a prototype of its

27

“GT-P7300” (the Galaxy Tab 8.9) against Apple’s iPad 2. (Id. at SAMNDCA00176053.) The

28

document notes that the “GUI and Visual Effect are lacking in comparison to iPad 2.” (Id. at
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

30

1

SAMNDCA00176055.) Subsequently, the evaluation notes that when a Browser application

2

window is scrolled to the top or bottom, the P5 “lack[s] bounce effect,” and that the Samsung’s

3

product “Lacks Fun, Wow Effect.” (Id. at SAMNDCA00176071.) This issue appears to have

4

been designated “Critical,” with the direction that the “Bounce effect is scheduled to be

5

reviewed.” (Id.) Later in the evaluation, there is a side by side comparison between the P5 and

6

the iPad 2, where the rubber-banding feature of the ’381 patent is being demonstrated on a web

7

page displayed on the iPad 2. (Id. at SAMNDCA00176125.) Specifically, the displayed web

8

page is being pulled to the lower right hand corner, revealing an area beyond the edge of the web

9

page to the left and above. (Id.) To the right, it states that “In case of iPad 2, there is a fun

10

element from a natural Bounce effect that follows hand gestures.” (Id.) Based on this statement,

11

it appears that Samsung understood at least part of the purpose and value of the rubber banding

12

feature of the ’381 patent, which were to provide a natural, intuitive experience for the user that

13

could also inspire wonder and a sense of delight. On the other hand, the P5 prototype is described

14

as lacking the “Bounce effect.” (Id.) Finally, the evaluation notes as an area for “Proposed

15

Improvement” of the P5 the incorporation of the “Bounce effect” from the iPad 2. (Id.)

16

104.

I have also reviewed a spreadsheet titled “Analysis of Galaxy Tab Operation

17

Speed and Screen Effects” (Bartlett Decl. Ex. 70 [SAMNDCA00201771 – 201780].) This ten-

18

page chart shows a detailed side-by-side comparison and analysis of over 70 features in the iPad

19

and the Galaxy Tab. Samsung concluded that its Memo and Browser applications had “no

20

emotional impact” because they lacked the “Bounce effect” included in the iPad. (Id. at

21

SAMNDCA00201773-6.) As discussed above, the “bouncing” or “bounce” feature described in

22

Samsung’s documents is an example of a feature covered by the ’381 patent. Based on my

23

analysis of the Galaxy Tab and Galaxy Tab 10.1, I conclude that Samsung implemented this

24

feature in the Galaxy Tab and Galaxy Tab 10.1 products as well.

25

105.

I understand that Samsung has identified Mr. Wookyun Kho as a person involved

26

in the implementation of the “bounce” feature in Samsung products. (See Bartlett Decl. Ex. 71

27

[Samsung’s Supplemental Response to Apple’s Interrogatory No. 16].) Based on Samsung

28

emails and the portions of the Wookyun Kho deposition transcript I reviewed, it appears that Mr.
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

31

1

Kho studied the “bounce” or “bouncing” feature in various Apple products including the iPhone

2

3GS, iPhone 4, iPad, and iPad 2 in order to improve Samsung’s products. (See, e.g., Bartlett

3

Decl. Ex. 72 [1/12/12 Kho Dep.] at 40:1-44:11, 48:18-49:21.) In particular, Mr. Kho appears to

4

have worked with an outside contractor company called NemusTech to emulate the features of

5

the iPad and make the “bounce effect” in Samsung’s Galaxy Tab prototype similar to the iPad.

6

(Bartlett Decl. Ex. 73 [SAMNDCA10851706-7]; Bartlett Decl. Ex. 74 [SAMNDCA10850604-

7

6].)

8

106.

That evidence speaks powerfully to the non-obviousness of the inventions of the

9

’381 patent. In short, if the inventions of this patent were as obvious and trivial as Dr. Van Dam

10

claims, it is unclear why no one had previously resolved the user interface issues described above,

11

and why a multinational company like Samsung would abandon its previous user interface

12

solutions (such as a hard stop at the edge of an electronic document) and copy Apple’s

13

functionality. Based on this information, I conclude that the inventions of the ’381 patent were

14

not trivial or obvious.

15

107.

Dr. Van Dam’s opinions on obviousness are limited to general assertions, and

16

though I disagree with Dr. Van Dam’s position, I note that there is nothing specific I can respond

17

to based on the fact that the Van Dam Declaration contains no explanation of the aforementioned

18

motivations to combine. As discussed above, Tablecloth does not anticipate claim 19 of the ’381

19

patent, and Dr. Van Dam’s contention that Tablecloth renders claim 19 obvious suffers from

20

hindsight bias. Tablecloth teaches away from the invention of the ’381 patent by implementing

21

the same traditional user interface features from which the ’381 patent departed.

22

IV.

23

DEMONSTRATIVES AND EXHIBITS
108.

In connection with my anticipated testimony in this action, I may use as exhibits

24

various documents produced in this case that refer or relate to the matters discussed in this

25

declaration. In addition, I may have demonstrative exhibits prepared to assist in the presentation

26

of my testimony and opinions as set forth or cited in my declaration.

27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf- 3150806

32

1

I declare under penalty of perjury under the laws of the United States of America that the

2

foregoing is true and correct and that this Declaration was executed this 31st day of May, 2012, at

3

Washington, D.C.

4
5

___________________________________
Ravin Balakrishnan

6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
DECLARATION OF RAVIN BALAKRISHNAN, PH.D. IN SUPPORT OF APPLE’S OPPOSITION TO MOT.
CASE NO. 11-CV-01846-LHK (PSG)
sf-3150806

33