Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 563

EXHIBITS re #559 Declaration in Support, filed byApple Inc.. (Attachments: #1 Exhibit 5.02, #2 Exhibit 5.03, #3 Exhibit 5.04, #4 Exhibit 5.05, #5 Exhibit 5.06, #6 Exhibit 5.07, #7 Exhibit 5.08, #8 Exhibit 5.09, #9 Exhibit 5.10, #10 Exhibit 5.11, #11 Exhibit 6.1, #12 Exhibit 6.2, #13 Exhibit 6.3, #14 Exhibit 6.4, #15 Exhibit 6.5)(Related document(s) #559 ) (Jacobs, Michael) (Filed on 12/29/2011)

EXHIBIT 5.03

claims 8-14 are directed to statutory subject matter and request the removal of the
rejection of these claims under 35 U.S.C. §101.
Allowable Subiect Matter

The Office Action indicates that claims 1-7 are allowed.
Applicants thank the Examiner for the allowance of claims 1-7. New independent
claim 89 includes similar limitations as independent claim 1. Thus, applicants request

the allowance of independent claim 89 and associated dependent claims 90-95.
Conclusion
Applicant respectfully submits that in view of the amendments and discussion set

forth herein, the applicable rejections have been overcome. Accordingly, the present
and amended claims should be found to be in condition for allowance.
If a telephone interview would expedite the prosecution of this application, the
Examiner is invited to contact the undersigned at [phone],
if there are any additional charges/credits, please charge/credit our deposit
account no. 02-2666.

Respectfully submitted,
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP
Dated:
'

J

R

. Scheller, Jr.

. No. 31,195

Customer No. 45217
1279 Oakmead Parkway
Sunnyvale, CA 94085
[phone]

Ser No.: 11/620,717

Page 9 of 9

Dkt No: 4860P4895

APLNDC00027256

Electronic Acknowledgement Receipt
EFS ID:

6232207

Application Number:

11620717

International Application Number:
Confirmation Number:

9801

Title of Invention:

Application Programming Interfaces for Scrolling Operations

First Named Inventor/Applicant Name:

Andrew Platzer

Customer Number:

45217

Filer:

James Scheller Jr./Connie Thayer

Filer Authorized By:

James Scheller Jr.

Attorney Docket Number:

04860.P4895

Receipt Date:

08-OCT-2009

Filing Date:

07-JAN-2007

Time Stamp:

22:01:41

Application Type:

Utility under 35 USC 111(a)

Payment information:
Submitted w th Payment

no

File Listing:
Document
Number

.
Document Description

i

File Name

File Size(Bytes)/
.
Message Digest

Multi
Pages
Part /.zip (if appl.)

262432

1

04860P4895_amendment.PDF

yes

9

b8175fe7b82267f3bf788f78f30f3d568dec4
3d9

APLNDC00027257

Multipart Description/PDF files in .zip description
Document Description

Start

End

Amendment/Req. Reconsideration-After Non-Final Reject

1

1

Specification

2

2

Claims

3

7

Applicant Arguments/Remarks Made in an Amendment

8

9

Warnings:
Information:
Total Files Size (in bytes)

262432

This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,
characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a
Post Card, as described in MPEP 503.
New Annlications Under 35 U.S.C. 111

If a new application is being filed and the application includes the necessary components for a filing date (see 37 CFR
1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this
Acknowledgement Receipt will establish the filing date of the application.
National Staae of an International Annlication under 35 U.S.C. 371

If a timely submission to enter the national stage of an international application is compliant with the conditions of 35
U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a
national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.
New International Annlication I¯iled with the USPTO as a Receivina Office

If a new international application is being filed and the international application includes the necessary components for
an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number
and of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerning
national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of
the application.

APLNDC00027258

PTO/SBIO6 (07-06)
Approved for use through 1/31/2007. OMB 0651-0032
U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

Under the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless it displavs a valid OMB control number.

PATENT APPLICATION FEE DETERMINATION RECORD

Application or Docket Number

Filing Date

Substitute for Form PTO-875

11/620,717

01/07/2007

APPLICATION AS FILED - PART I

To be Mailed

OTHER THAN

(Column 1)

(Column 2)

NUMBER FILED

NUMBER EXTRA

RATE ($)

N/A

N/A

N/A

N/A

SEARCH FEE

N/A

N/A

N/A

N/A

EXAMINATION FEE

N/A

N/A

N/A

N/A

FOR

BASIC FEE

(37 CFR 1.16 a

b), or (c))

(37 CFR 1.16(k), (i), or (m))

SMALL ENTITY Q

OR

FEE ($)

SMALL ENTITY
RATE ($)

FEE ($)

(37 CFR 1.16(o), (p), or (q))

TOTAL CLAIMS
(37 CFR 1.16(i))

minus 20 =

(37 CFR 1.16(h))

APPLICATION SIZE FEE
(37 CFR 1.16(s))

*

X $

=

minus 3 =

INDEPENDENT CLAIMS

*

X $

OR

=

X $

=

X $

=

If the specification an, drawings exceed 100
sheets of paper, the application size fee due
is $250 ($125 for small entity) for each
additional 50 sheets or fraction thereof. See
35 U.S.C. 41(a 1 G) and 37 CFR 1.16(s)

MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.16(j))
* If the difference in column 1 is less than zero, enter "0" in column 2.

TOTAL

TOTAL

APPLICATION AS AMENDED - PART II
OTHER THAN
(Column 1)

(Column 2)

I CLAIMS
REMAINING
10/08/2009
AFTER
AMENDMENT

(Column 3)

HIGHEST
NUMBER
PREVIOUSLY
PAID FOR

PRESENT
EXTRA

SMALL ENTITY

RATE ($)

OR

ADDITIONAL
FEE ($)

SMALL ENTITY

RATE ($)

x

o=

ADDITIONAL
FEE ($)

0

Application Size Fee (37 CFR 1.16(s))
FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

OR

TOTAL
ADD'L
FEE
(Column 1)

(Column 2)
HIGHEST
NUMBER
PREVIOUSLY
PAID FOR

PRESENT
EXTRA

TOTAL
ADD'L
FEE

0

(Column 3)

CLAIMS
REMAINING
AFTER
AMENDMENT

OR

RATE ($)

ADDITIONAL
FEE ($)

RATE ($)

x$

=

OR

x$

=

x$

=

OR

x$

ADDITIONAL
FEE ($)

=

Application Size Fee (37 CFR 1.16(s))
<

OR

O FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

TOTAL
ADD'L
FEE

* If the entry in column 1 is less than the entry in column 2, write "0" in column 3.

OR

TOTAL
ADD'L
FEE

Legal Instrument Examiner:

** If the "Highest Number Previously Paid For" IN THIS SPACE is less than 20, enter "20".
/PATSY ZIMMERMAN/
*** If the "Highest Number Previously Paid For" IN THIS SPACE is less than 3, enter"3".
The "Highest Number Previously Paid For" (Total or Independent) is the highest number found in the appropriate box in column 1.

This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to
process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete, including gathering,

preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you
require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S.
Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS

ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.

APLNDC00027259

PTO/SBIO6 (07-06)
Approved for use through 1/31/2007. OMB 0651-0032
U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

Under the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless it displavs a valid OMB control number.

PATENT APPLICATION FEE DETERMINATION RECORD

Application or Docket Number

Filing Date

Substitute for Form PTO-875

11/620,717

01/07/2007

APPLICATION AS FILED - PART I

To be Mailed

OTHER THAN

(Column 1)

(Column 2)

NUMBER FILED

NUMBER EXTRA

RATE ($)

N/A

N/A

N/A

N/A

SEARCH FEE

N/A

N/A

N/A

N/A

EXAMINATION FEE

N/A

N/A

N/A

N/A

FOR

BASIC FEE

(37 CFR 1.16 a

b), or (c))

(37 CFR 1.16(k), (i), or (m))

SMALL ENTITY Q

OR

FEE ($)

SMALL ENTITY
RATE ($)

FEE ($)

(37 CFR 1.16(o), (p), or (q))

TOTAL CLAIMS
(37 CFR 1.16(i))

minus 20 =

(37 CFR 1.16(h))

APPLICATION SIZE FEE
(37 CFR 1.16(s))

*

X $

=

minus 3 =

INDEPENDENT CLAIMS

*

X $

OR

=

X $

=

X $

=

If the specification an, drawings exceed 100
sheets of paper, the application size fee due
is $250 ($125 for small entity) for each
additional 50 sheets or fraction thereof. See
35 U.S.C. 41(a 1 G) and 37 CFR 1.16(s)

MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.16(j))
* If the difference in column 1 is less than zero, enter "0" in column 2.

TOTAL

TOTAL

APPLICATION AS AMENDED - PART II
OTHER THAN
(Column 1)

(Column 2)

I CLAIMS
REMAINING
10/08/2009
AFTER
AMENDMENT

(Column 3)

HIGHEST
NUMBER
PREVIOUSLY
PAID FOR

PRESENT
EXTRA

SMALL ENTITY

RATE ($)

OR

ADDITIONAL
FEE ($)

SMALL ENTITY

RATE ($)

x

o=

ADDITIONAL
FEE ($)

0

Application Size Fee (37 CFR 1.16(s))
FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

OR

TOTAL
ADD'L
FEE
(Column 1)

(Column 2)
HIGHEST
NUMBER
PREVIOUSLY
PAID FOR

PRESENT
EXTRA

TOTAL
ADD'L
FEE

584

(Column 3)

CLAIMS
REMAINING
AFTER
AMENDMENT

OR

RATE ($)

ADDITIONAL
FEE ($)

RATE ($)

x$

=

OR

x$

=

x$

=

OR

x$

ADDITIONAL
FEE ($)

=

Application Size Fee (37 CFR 1.16(s))
<

OR

O FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

TOTAL
ADD'L
FEE

* If the entry in column 1 is less than the entry in column 2, write "0" in column 3.

OR

TOTAL
ADD'L
FEE

Legal Instrument Examiner:

** If the "Highest Number Previously Paid For" IN THIS SPACE is less than 20, enter "20".
/ANNETTE SMITH/
*** If the "Highest Number Previously Paid For" IN THIS SPACE is less than 3, enter"3".
The "Highest Number Previously Paid For" (Total or Independent) is the highest number found in the appropriate box in column 1.

This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to
process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete, including gathering,

preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you
require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S.
Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS

ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.

APLNDC00027260

Document code: WFEE
United States Patent and Trademark Office
Sales Receipt for Accounting Date: 04/20/2010

ASMITH3

SALE #00000001
Mailroom Dt: 10/08/2009
022666
01
FC : 1201
220.00 DA
02
FC : 1202
364.00 DA

11620717

APLNDC00027261

Attorney's Docket No. 4860P4895

PATENT

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
In Re Application of:
Andrew Platzer

Examiner: Bautista, Xiomara L

Application No.: 11/620,717

Art Unit:

Filed:

Confirmation No.: 9801

January 7, 2007

2179

For: APPLICATION PROGRAMMING
INTERFACES FOR SCROLLING
OPERATIONS
Mail Stop Amendment

Commissioner for Patents
P.O. Box 1450
Alexandria, Virginia 22313-1450
INFORMATION DISCLOSURE STATEMENT

Sir:

Enclosed is a copy of Information Disclosure Citation Form PTO-1449 or
PTO/SB/08 together with copies of the documents cited on that form, except for

copies not required to be submitted (e.g., copies of U.S. patents and U.S.
published patent applications need not be enclosed for applications filed after
June 30, 2003). It is respectfully requested that the cited documents be

considered and that the enclosed copy of Information Disclosure Citation Form
PTO-1449 or PTO/SB/08 be initialed by the Examiner to indicate such

consideration and a copy thereof returned to applicant(s).
I hereby certify that this correspondence is being submitted electronically via EFS Web to the

United States Patent and Trademark Office on the date shown below.
(Date of Deposit)
Jeremy A Schwegert
(Typed or printed name of person mailing correspondence)
/JeremyA. Schweig_ert/
(Signature of person mailing correspondence)

Application No.: 11/620,717

- 1 -

10-13-2009
(Date)

Docket No.: 4860P4895

APLNDC00027262

Pursuant to 37 C.F.R. § 1.97, the submission of this Information

Disclosure Statement is not to be construed as a representation that a search
has been made and is not to be construed as an admission that the information
cited in this statement is material to patentability.
Pursuant to 37 C.F.R. § 1.97, this Information Disclosure Statement is
being submitted under one of the following (as indicated by an "X" to the left of

the appropriate paragraph):
37 C.F.R. §1.97(b).

X

37 C.F.R. §1.97(c). If so, then enclosed with this Information

Disclosure Statement is _o_n_e of the following:
A statement pursuant to 37 C.F.R. §1.97(e) Œ
X

The Director is Authorized to charge our Deposit Acct. No. 022666 in the amount of $180.00 for the fee under 37 C.F.R. §
1.17(p).

37 C.F.R. §1.97(d). If so, then enclosed with this Information

Disclosure Statement are the following:
(1)

A statement pursuant to 37 C.F.R. §1.97(e); and

(2)

Authorization to charge our Deposit Acct. No. 02-2666 in the
amount of $180.00 for the fee under 37 C.F.R. §1.17(p) for
submission of the Information Disclosure Statement.

If there are any additional charges, please charge Deposit Account No.
02-2666.

Respectfully submitted,
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP

Dated: nos 12 onn

Jeremy A. Schwegrt
Jeremy A. Schweigert

Reg. No. 56,244
1279 Oakmead Parkway
Sunnyvale, CA 94085-4040
[phone]

Application No.: 11/620,717

-2-

Docket No.: 4860P4895

APLNDC00027263

t
Europäisches Patentamt
¿

(19)

European Patent Office

Office européen des brevets

(12)

(11)

EP 1 517 228 A2

EUROPEAN PATENT APPLICATION

(43) Date of pubücation:
23.03.2005 Buitetin 2005/12

(51) int Cli: GO6F 3/033

(21) Appucation number: 04255568.0

(22) Date of finng: 15.09.2004

(84) Designated Contracting States:
AT BE BG CH CY CZ DE DK EE ES FI FR GB GR
HU IE IT Li LU MC NL PL PT RO SE Si SK TR
Designated Extension States:
AL HR LT LV MK

• Morrison, Gerald D.
Calgary, Alberta T3G 4T6 (CA)

(30) Priority: 16.09.2003 US 662813

(74) Representative: Naismith, Robert Stewart et at
Marks & Clerk Scotland
19 Royal Exchange Square

(71} Applicant: Smart Technologies, Inc.
Calgary, Alberta T3C OMS (CA)

(54)

(72) Inventors:
• Hill, Douglas B.
Calgary, Alberta T2K 1Z2 (CA)

Glasgow, G1 3AE Scotland (GB)

Gesture recognition method and touch system incorporating the same

(57)
A gesture recognition method includes detecting multiple pointers in close proximity to a touch surface
to determine if the multiple pointers are being used to

are being used to perform a known gesture, acommand
associated with the gesture is executed. A touch system
incorporating the gesture recognition method is also

perform a known gesture, When the multiple pointers

provided,

60

FlG. 4a

co

CL

LLI

Panted ey Jowe

soo ens em

(Cont next page)

APLNDC00027264

EP 1 517 228 A2

FIG 4b

FIG. 4c

o

60

FIG. 4e

2
)OCID:<EP

(Cont. next page)

1517228A2 1 >

APLNDC00027265

EP 1 517 228 A2

FIG. 4f

3

APLNDC00027266

1

EP 1 517 228 A2

Description

sociated with the digital cameras to determine if a pointer exists in the captured image data. When it is determined that a pointer exists in the captured image data,

Field of the invention

[0001] The present invention relates generally to
touch systems and in particular to a gesture recognition

s

method and touch system incorporating the same.
Background of the Invention
10

[0002) Touch systems are well known in the art and
typically include a touch screen having a touch surface
on which contacts are made using a pointer in order to
generate user input. Pointer contacts with the touch surface are detected and are used to generate correspond-

ing output depending on areas of the contact surface
where the contacts are made. There are basicany two
general types of touch systems available and they can
be broadly classified as "active" touch systems and
"passive" touch systems,
[0003) Active touch systems allow a user to generate
user input by contacting the touch surface with a special

2

15

the digital signal processors generate pointer information packets (PIPs) and convey the PIPs to a mâster
controffer. Each PiP includes a header portion, a data
portion and a checksum. The data portion includes a
pointer ID field that stores a pointer identif
a allow
multiple pointers to be tracked The data p
n also
includes a pointer location parameter that
dies a
pointer x-position and a pointer tip parameter that ídentifies a pointer z-position. A contact state field stores a
value indicating whetherthe pointer is in or out of contact
with the touch sudace allowing pointer hover to be detected.
[0007] Upon receipt of the PIPs, the master controßer
processes the PIPs using triangulation to determine the
location of each pointer in the captured images relative

power source, typically batteries. The special pointer
emits signals such as infrared light, visible light, ultra-

to the touch surface in (x,y) coordinates, in this manner,
20 as PIPs are generated in response to captured images,
the position and movement of pointers over the touch
surface can be tracked. The pointer location data generated by the master controller is conveyed to a computer executing one or more application programs. The
25 computer uses the pointer location data to update the

sonic frequencies, electromagnetic frequencies, etc.

computer-generated image that is presented on the

that activate the touch surface.
[0004) Passive touch systems allow a user to generate user input by contacting the touch surface with a

touch surface. Pointer contacts on and pointer movement over the touch surface can therefore be recorded
as writing or drawing or used to control execution of ap30 plication programs executed by the computer.

pointer that usually requires some form of on-board

passive pointer and do not require the use of a special
pointer in orderto activate the touch surface. The pointer

(0008]

can be a finger, a cylinder of some material, or any suitable object that can be used to contact some predetermined area of interest on the touch surface.
[DOOS) Passive touch systems provide advantages
over active touch systems in that any suitable pointing
device, including a user's finger, can be used as a point-

As will be appreciated, since digital cameras

at the comers of the bezeln are used to capture image
data, the touch system is saa to determine when mul-

tiple pointers contact and move across the touch sur35

face. This of course provides for enhanced functionality
as compared to analog resistive touch systems that are
only able to track a single pointer. Although enhanced
functionality is provided by the above-described camera-based touch system, to-date, this enhanced functionally has not been fully exploited. It is therefore an
object of the present invention to provide a novel gesture
recognition method and touch system incorporating the

er to contact the touch surface. As a result, user input
can easily be generated, Also. since special active
pointers are not necessary in passive touch systems, 40
battery power levels and/or pointer damage, theft, or
misplacement are of no concern to users.
[0006] For example, U.S. Patent Application No.
same.
09/610481 filed on July 5, 2000 and Intemational PCT
Application No. PCT/CA01/00980 filed on July 5, 2001 45 Summarv of the Invention
and published under No, WO 02/03316 on January 10,
2002, assigned to SMART Technologies Inc., assignee
[0009] According to one aspect of the present invenof the present invention, disclose a camera-based touch
tion there is provided a gesture recognition method comsystem comprising a touch screen that includes a pasprising the steps of:
sive touch surface on which a computer-generated im- 50
age is presented. A rectangular bezei or frame surdisplaying an image on a touch surface;
rounds the touch surface and supports digital cameras
detecting pointer contacts on said touch surface
at its comers. The digital cameras have overlapping
and examining said pointer contacts to recognize
fields of view that encomcass and look across the touch
multiple pointer contacts representing a gesture;
surface. The digital cameras acquire images looking 55
and
across the touch surface from different locations and
when multiple pointer contacts representing a gesgenerate image data, image data acquired by the digital
ture occur updating the displayed image in accordcameras is processed by digital signal processors asance with said gesture.

4

APLNDC00027267

3
(0010)

EP 1 517 228 A2

4

Multiple pointer contacts representing a ges-

at least one processor communicating with said at

ture include multiple finger contacts on the touch surface, a fingercontact on the touch surface and an object

least one imaging device and analyzing images acquired by said at least one imaging device to deter-

contact on the touch surface and multiple object con-

minethelocationonsaidtouchsurfacewherepoint-

tacts on the touch sudace,
(0011] in one aspect, the gesture is a right-click event

5

and is represented by a first pointer contact on a displayed applicatiort and a subsequent second pointer
contact that occurs within athreshold distance of the first
pointer contact and while the first pointer contact is
maintained. In another aspect, the gesture is a scroll
event and is represented by simultaneous pointer contacts on the touch surface.
(0012) According to another aspect of the present invention there is provided a gesture recognition method
comprising the steps of:

to

is

20

ciated with said gesture.

[0013] Preferably, during the detecting, pointer contacts with or close pointer hovers overthe touch surface

amining said multiple pointer contacts to determine
if said multiple poinîer contacts represent a gesture
and when said multiple pointer contacts represent
agesture,saidprocessorexecutingacommandassociated with said gesture
[0016] According to still yet another aspect of the
present invention there is provided an interactive input
system comorising:

at least one imaging device having an input region
within its field of view into which one or more pointers is moved to generate user input; and

detecting multiple pointers in close proximity to a
touch surface to determine if said multiple pointers

are being used to perform a known gesture; and
when said multiple pointers are being used to perform a known gesture, executing a command asso-

er contacts are made. when said touch surface is
contacted by multiple pointers, said processor ex-

at least one processor cornrnunicatirig with said at
least one imaging device and analyzing each image
acquired by said at least one imaging demce to determine the action of pointers in said input region,

25

said at least one processor determining when multiple pointer actions within said input region represent a gesture, when said multiple pointer actions

are detected to determine if a known gesture is being
represent a gesture, said at least one processor experformed and specifically if one of a number of known
ecuting a command corresponding to said gesture.
gestures is being performed, each known gesture being
associated with a different command, in a preferred em- 30 [0017) According to still yet another aspect of the
bodiment, the movement of the multiple pointers relative
present invention there is provided in an interactive
to the touch sudace and/or the pointer type determines
touch system, a method of simulating a right-click
the gesture being performed.
mouse event comprising the steps of:
[0014] According to yet another aspect of the present
invention there is provided an input detection method in
an interactive system capable of detecting movement of
multiple pointers generally simultaneously within an input region, said method comprising the steps of:

35

capturing images looking generally across said input region;
analyzing said images to detect multiple pointers
within said input region;
when multiple pointers are detected, examining dala associated with said multiple pointers to determine if the data represents an input gesture; and

40

45

when the data represents an input gesture, executing acommand corresponding to the recognized in-

detecting a first pointer contact on a touch surface
over a displayed application that represents a leftclick mouse event:
detecting a second pointer contact on said touch
surface that occurs within a threshold period of time
fonowing said first pointer contact and within a
threshold distance of said first pointer contact; and
generating a right-click mouse event in response to
said detected second pointer contact,
(0018] The present invention provides advantages in
that since gestures representedbymultiplepointercon-

tacts on and/or movement over the touch surface are
detected and corresponding commands generated, en-

put gesture.

hanced touch system functionality is provided.
50

[0015) According to still yet another aspect of the
present invention there is provided a touch system compasing
a touch surface to be contacted by at îeast one
pointer;
at least one imaging device having a field of view
looking generany across said touch surface: and

Brief Description of the Drawings

SS

[0019] Embodiments of the present invention will now
be described more fully with reference to the accompanying drawings in which:
Figure 1 is a schematicdiagram of acamera-based
touch system in accordance with the present inven-

5

APLNDC00027268

5

EP 1 517 228 A2

6

essors of the digital cameras 70 generate pointer information packets(PIPs) and convey the PIPs to the digital
signal processor (DSP) based master controller 54.

tion:
Figure 2 is a front elevation view of a touch screen
forming part of the touch system of Figure 1;
Figures 3a to 3d are front perspective views of the
touch screen of Figure 2 showing an input right-click
gesture; and
Figures 4a to 4f are front perspective views of the
touch screen of Figure 2 showing input up/down
and left/right scroli gestures.

5

to

Detaned Descriotion of the Preferred Embodiments

Each PIP includes a header portion, a data portion and
a checksum. The data portion includes a pointer (D field
that stores a pointer identifier to anow multiple pointers
lo be tracked. The data porbon also includes a pointer
location parameter that identifies a pointer x-position
and a pointer tip parameter that identifies a pointer zpositîon. A contact state field stores a value indicating
whether the pointer is in ðr out of contact with the touch

surface 60 allowing pointer hover to be detected.
(0020]

Turning now to Figure i, a camera-based

touch system in accordance with the present invention
is shown and is generally identifËed by reference numeral 50. Camera-based touch system 50 is similar to that
disclosed in International PCT Application Serial No.
WO 02/03316, assigned to SMART Technologies Inc.
assignee of the present invention, the contents of which

is

[0023] Upon receipt of the PîPs, the master controller
54 processes the PIPs using triangulation to determine
the location of each pointer in the captured images relative to the touch surface 60 in (x,y) coordinates. In this
manner, as PfPs are generated in response to captured

images, the position and movement of pointers over the
touch surface 60 can be tracked. Since image data is

are incorporated herein by reference. As can be seen, 20 processed to detect the existence of one or more point.
touch system 50 includes a touch screen 52 coupled to
ers, the pointers may take any suitable form such as for
a digitai signal processor (DSP) based master controller
example, a user's finger, a cylinder of material, a passive
54. Master controller 54 is also coupied to a computer
or active pen tool or erase tool or other appropriate ob56. Computer 56 executes one or more application project. Specifics of the marmer by which the image data is
grams and provides computer-generated image output 2s acquired by the digital cameras 70 and processed by
that is displayed on the touch screen 52. The coordinate
the master controller 54 are described in Intemational
systern of the touch system 52 is mapped to the coordiPCT Application No. PCT/CA01/00980 fued on July 5,
nate system of the computer. The touch screen 52, mas2001 and published under No, WO 02/03316 on Januter controller 54 and computer 56 form a closeddoop so
ary 10, 2002, the contents of which are incorporated
that pointer hover or contacts with and pointer move- so herein by reference. Accordingly, such specifies will not
ment overor above the touch screen 52 can be recorded
be described further herein.
as writing or drawing or used to control execution of ap[0024] The master controller 54 outputs generated
pHcation programs executed by the computer 56.
pointer data to the computer 56 that identifies the loca[002¶ Figure 2 better illustrates the touch screen 52.
tion of each pointer relative to the touch surface as each
Touch screen 52 in the present embodiment includes a 35 pointer approaches and/or contacts and moves over the
high-resolution display device such as a plasma display
touch surface 60, A driver loaded on the computer 56
58, the front sudace of which defines a touch surface
60. The touch surface 60 is bordered by an inuminated

receives the pointer data and examines the pointer data
to determine if the pointer data has been generated in

bezel or frame 62 coupled to the display device, illuminated bezel 62 is of the type disciosed in U,S. Patent
Application No. 10/354,168 filed on January 30, 2003
and includes elongate side frame assemblies 64 that are
coupled to the sides of the plasma display 58. Each side
frame assembly 64 accommodates a light source (not
shown) that projects infrared backlighting across the

touch surface 60. The ends of the side irame assemblies
64 are joinedbycomerpieces 68 that house DSP-based
CMOS digital cameras 70. Each digital camera 70 Ís
mounted within its res:
comer piece 68 so that its

field of view encompa

ad looks across the entire

touch surface 60,

[0022] During operato
.ne digital cameras 70 acquire images of the touch surface 60 and generate image data. The acquired image data is processed by digital signal processors associated with the digital cameras 70 to determine if a pointer exists in the captured images. When it is determined that one or more pointers
exist in the acquired irnage data, the digital signal proc-

response to a known input gesture stored in a gesture
40

45

library. Specifically, the driver examines the pointer data
to detect the existence of multiple pointers in captured
images and then examines the nature of the multiple
pointers to determine if a known gesture has been performed such as forexample a right-click gesture, a scroll
gesture, a rotate gesture etc. When a gesture has been

performed, the driver generates a comrnand (event) that
is associated with the determined gesture and conveys
the command to the active application program being
executed by the computer 56,
so [0025] Turning now to Figures 3a to 4f, examples of
gestures that can be recognized by the touch system
andthe resulting actions that are performed in response
to the recognized gestures are shown.
56

intuitive Right-Click Gesture
[0026]

Figures 3a to 3d illustrate an intuitive right-elick

gesture. When a user contacts the touch surface 60 with

6
)O

<EP

151722BA2 i >

APLNDC00027269

7

EP i 517 228 A2

5

closely and generally horizontally spaced, the driver recognizesthesimultaneousfingercontactsasapagegesture and injects a page event into the application. Pointer position data conveyed to the application by the driver
in response to subsequent vertical movement of the fin-

10

gers is interpreted by theapplication as page up or page
down commands depending on the direction of the verBeat movement. 10 response to the page up or page
down commands, the application moves information
displayed within the window in the appropriate direction.

a finger over an application displayed on the touch surface, the driver recognizes the contact as a left-click
mouse event and injects the left-click mouse event into
tne appHcation. If the user subsequently contacts the
touch surface 60 with another finger while rnaintaining
the contacl with the one finger and the subsequent con-

tact is to the right. of and close to the initial contact, the
driver recognizes the second touch surface contact as
a right-click gesture and injects a right-click event into
the application. In response to the right-cGck event. the
application opens and displays a drop down menu (not
shown). As will be appreciated, this enables a user to
invoke a right-click action using a hand gesture that is
similar to the action performed when invoking a rightclick action using a mouse. Although, Figures 3a to 3d
show the irituitive right-click gesture being performed
using two fingers on the same hand, it will be appreci-

Rotate Gesture

is

(0027] Figures 4a to 4e illustrate up/down and left/
right scroli gestures. If the user contacts the touch sur-

face 60 with a pair of fingers simultaneously over an application window displayed on the touch surface and the
fingers are closely and generally horizontally spaced,
the driver recognizes the simultaneous finger contacts
as a scroll gesture and injects a scroll event into the application. Pointer position data conveyed to the applica-

tion by the driver in response to subsequent vertical
movement of the fingers is interpreted by the application
either as scroll up or scron down commands, in response to the scron up or down comrnands, the appilcation moves informatíon presented within the application window in the direction of the vedical .movement.
Pointer position data conveyed to the application by the
driver in response to subsequent horizontal movement
of the fingers is interpreted by the application as scroll
to side commands. In response to the scroll to side commands, the application moves infonnation displayed
within the application window to the side corresponding

to the direction of the horizontal movement, Although
Figures 4a to 4f show the scroli gestures being performed using two fingers on the same hand. it will be
appreciated that the scroll gestures can be perfonmed
using a finger on different hands.
[0028) Although not illustrated, anumberof other gestures can be recognized by the driver and used to generate commands to control an application being executed by the computer56, Examples of such other gestures
will now be described.

Page Up/Down Gesture

(0029) if the user contacts the touch surface 60 with
three fingers simultaneously overan application window
dísplayed on the touch surface and the threetingers are

(0030) If the user contacts the touch surface 60 over
an object displayed within an application window with
one finger and then subsequently contacts the touch

surface with another finger and moves that other finger
in an arc whue maintaining the touch surface contact

ated that the right-click gesture can be performed using
a finger on different hands
Scroll Un/Down and Left/Riciht Gesture

8

with the one finger. the driver recognizes the arcuate

20 movement of the second finger as a rotate gesture. The
driver in turn injects a rotate command into the application causing the application to rotate the object about
the contact point defined by the first finger in the direction of the arc and by an amount equivalent to the path
25 of the arc.

Zoom Gesture
(0031]

If the user contacts the touch surface 60 with

so a pair of closely spaced fingers simultaneously over an
application window and expands the distance between
the fingers in a generally horizontal direction, the driver
recognizes the finger movement as a zoom-out gesture,
The driver in tum injects a zoom-out command into the
35

application causing the application to expand the size
of the information presented in the application window.
If the user contacts the touch surface 60 with a pair of

spaced fingers simultaneously over an application window and moves the fingers in a generally horizontal di4o

rection towards one another, the driver recognizes the

finger movement as a zoom-in gesture. The driver in
turn injects a zoom-in command into the application
causing the application to reduce the size of the information presented in the application window.
45 [0032] Afternatively, the zoom-out and zoom-in commands may be represented by other gestures. For example, if the user contacts the touch surface 60 with a
clawed hand having its fingers bunched together over
an appücation window and expands the hand by extendso ing the fingers outwardly. the driver recognizes the finger movement as the zoom-out esture.
[0033] If the user contacts the touch surface 60 with
a generally flat hand having its fingers extended over an
application window and contracts the hand by clawing
55 the fingers inwardly to bunch them together the driver
recognizes the finger movement as the zoom-in gesture.

7

APLNDC00027270

9

EP 1 517 228 A2

Expand Gesture

10

scrolled. Pointers contacting the touch surface 60 at a

steep angle may represent a slow scroll rate whereas
[0034] If the user contacts the touch surface 60 with
a pair of closely spaced fingers simultaneously over an
application wîndow and expands the distance between

pointers contacting the touch surface 60 at a shattow

angle may represent a fast scroll rate.
5

(0040) if the touch system is able to differentiate between the type of pointers used to contact the touch surface 60 as is described in co-pending U.S. Patent Application Serial No. 10/384,783 filed on March 11, 2003
and/or is able to determine pointer characteristics as is
described in co-pending U.S. Patent Application Serial
No. 10/294,917, fited on November 15, 2002, the contents of which are incorporated herein by reference, different functionalíty can be assigned to similar gestures
that are performed using different pointers. For exampie, in the case of the rotate gesture described above.
If the same gesture is carried out using a fingerto initially

the fingers in a generally diagonal direction, the driver

recognizes the fînger rnovement as an increase window
size gesture. The driver in turn injects an increase window size command into the application causing the application to expand the size of the application window,
if the user contacts the touch surface 60 with a pair of
spacod fingers simultaneously over an application window and moves the fingers in a generally diagonal direction towards one another, the driver recognizes the
finger movement as a decrease window size gesture,
The driver in turn injects a decrease window size command into the application causing the application to re-

10

15

contact an object within the application window and a
pen tool to describe the are, the driver recognizes the
finger contact and pen movement as a pattern fill gesIcon Select and Open Gesture
20 ture rather than a rotate gesture. A finger contact and
subsequent closely spaced pen tool contact may repre[0035] If the user contacts the touch surface 60 with
sent a draw circle gesture rather than a scroli gesture
two closely spaced fingers simultaneously over an icon,
and a finger contact and subsequent closely spaced
the driver recognizes the finger contact as a double-click
eraser contact may represent an erase page gesture,
gesture. The driver in turn generates an open applica- 25 As will be appreciated, being able to differentiate betion command causing the desktop application running
tween multiple pointers brought into proximity with the
on the computer 56 to open the selected application.
touch surface 60 significantly increases the functions
that may be invoked by performing the same gestures
Obiect/Window Move Gesture
simply by using discrete pointers that can be differenti30 ated.
[0036] If the user moves a pair of closely spaced fin(0041] Although the driver is described as examming
gers above the touch surface 60 and over an object disthe pointer data to determine if the pointer data is genplayed within an application window, the driver recogerated in response to a known gesture, it will be apprenizes the hovering finger movement as a translate obciated by those of skill in the art that if the active appliject gesture. The driver in tum injects a translate object as cation being executed by the computer has the capabilcommand into the application causing the application to
ity of recognizing gestures, the pointer data may be contranslate the displayed object in the direction of and by
veyed to the active application for gesture recognition,
an amount equal to the distance the fingers are moved.
[0042] If desired the touch surface 60 may be partä
[0037] ff the user moves three closely spaced fingers
tioned into multiple regions to enable multiple users to
above the touch surface 60 and over an application win- 40 interact with the touch surface simultaneously without
dow, the driver recognizes the hovering finger moveambiguity between user input. In this case multiple conment as a translate window gesture. The driver in turn
tacts on or hovers over the touch surface that are beduce the size of the application window,

generates a translate window command causing the
desktop application running on the computer 56 to trans-

late the application window in the direction of and by an
amount equal to the distance the fingers are moved,

(0038]

45

Although the above gestures are described as

being recognized in responseto multiple finger contacts

or hovers, the same gestures can be recognized if other
objects are used to perform the gestures. For example, 50
multiple pen tools can be used to pedorm the gestures
or alternatively a finger and a pen tool can be used to
perform the gestures.
[OC 9] Also, recognized gestures may be enhanced
U p different pointer characteristics. For e×ample, in 55
the case of scroll gestures, the angle at which the point-

yond a threshold distance are treated as multiple user
inputs. Multiple contacts on or hovers over the touch surface that are within the threshold distance are treated

as multiple contacts made by a single user and are examined to determine if the multiple contacts represent
a gesture.
[0043] Although preferred embodiments of the
present invention have been described, those of skill in
the art will appreciate that variations and modifications
may be made without departing from the spirit and
scope thereof as defined by the appended claims.

Claims

ers contact the touch surface 60 may be used to deter-

mine the rate at which the displayed inforrnation is

i

A gesture recognition method comprising the steps

8
CO <EP

1517228A2 i >

APLNDC00027271

11

EP 1 517 228 A2

of:

12

on said touch surface determining the direction of
scroll

displaying an image on a touch surface:
detecting pointer contacts on said touch surface and examining said pointer contacts to
recognize multiple pointer contacts represent-

5

10

ing a gesture; and
when multiple pointer contacts representing a
gesture occur. updating the displayed image in
accordance with said gesture.

12. Agesturerecognitionmethodcomprisingthesteps
of:
detecting multiple pointers in close proximity to
a touch surface to determine if said multiple
pointers are being used to perform a known
gesture; and

when said multiple pointers are being used to
2.

The method of claim 1 wherein multiple pointer con-

perform a known gesture, executing a com-

tacts representing a gesture include multiple finger

mand associated with said gesture,

contacts on said touch surface.
15
3.

The method of claim 1 wherein multiple pointer contacts representinga gesture include a finger contact

on said touch surface and an object contact on said
touch surface

if a known gesture is being performed,
20

4.

The method of claim 1 wherein multiple pointer contacts representing a gesture include multiple object

contacts on said touch surface.
5.

The methodof claim 1 wherein multiple pointer contacts representing a gesture include multiple finger
contacts on said touch surface, a finger contact on
said touch surface and an object contact on said
touch surface, and/or multiple object contacts on
said touch surface.

13. The method of claim 12 wherein duríng said detecting, pointer contacts with or close pointer hovers
over said touch surface are detected to determine

14. The method of claim 13 wherein said muîtÍple pointers include multiple fingers, at least one finger and
at least one object, and multiple objects in close

proximity to said touch surface.
25

15. The method of claim 13 wherein during said detecting the multiple pointers are examined to determine
if one of a number of known gestures is being performed, each known gesture being associated with
a different command,

30
16. The method of claim 15 wherein the movement cf

6. The method of claim 1 wherein said gesture represents a right-click event.
7.

the multiple pointers relative to the touch surface
determines the gesture being performed.

Themethodolciaim6whereinsaidright-clickevent .35
is represented by a first pointer contact on a dis-

17. The method of claim 15 wherein the pointer type
determines the gesture being performed.

played application, and a subsequent second pointer contact adjacent said first pointer contact.

8.

The methodof claim 6 wherein said right-click event
is represented by a first pointer contact on a dis-

18. The method of claim 15 wherein the movement of
the multiple pointers relative to the touch surface

40

pîayed application, and a subsequent second pointer contact that occurs within a threshold distance of
said first pointer contact and whBe said first pointer

contact is maintained.
9.

19. An input detection method in an interactive system
capable of detecting movement of multiple pointers

45

The method of claim i wherein said gesture is represented by simultaneous pointer contacts on said
touch surface,

generaHy simultaneously within an input region,
said method comprising the steps of:
capturing images looking generally across said
input region;

50
10. The method of claim 9 wherein said simultaneous
pointer contacts are simultaneous finger contacts
on said touch surface.

11. The method of claim 9 wherein said simultaneous
pointer contacts on said touch surface represent a
scroll event, the direcdon of movement of the pointers over said touch surface subsequent to-contact

and the pointer type determines the gesture being
performed.

55

anaîyzing said images to detect multiple pointers within said input region;
when multiple pointers are detected, exarnining
data associated with said multiple pointers to
determine if the data represents an input gesture; and
when the data represents an input gesture, executing a command corresponding to the recognized input gesture,

9

APLNDC00027272

13

EP 1 517 228 A2

14

lating a right-click mouse event comprising the
steps of:

20. A touch system comprising:
a touch surface to be contacted by at least one
pointer;
at least one imaging device having a field of
view looking generally across said touch surface; and

5

at least one processorcommunicating with said

detecting a first pointer contact on a touch surface that represents a left-click mouse event;
deteclog when a subsequent second pointer
contact on said touch surface occurs within a
threshold distance of said first pointer contact:

at least one imaging device and analyzing images acquired by said at least one imaging device to determine the location on said touch sur-

70

face where pointer contacts are made, when
said touch surface is contacted by multiple
pointers, said processor examining said multi-

and
generating a right-clîck rnouse event in response to said detected second pointer contact-

pie pointercontacts to determine if said multiple
pointer contacts represent a gesture and when

is

said multiple pointer contacts represent a ges-

28. The method of claim 27 wherein said second pointer contact must also occur during said first pointer
contact in order for said right-click mouse event to
be generated.

ture, said processor executing a command associated with said gesture.
20

21 A touch system according to claim 20 wherein a first
pointer contact followed by a subsequent second
pointer contact adjacent said first pointer contact
represents a gesture.

29. The method of claim 28 wherein said second pointer contact must also occur to the right of said first
poínter contact in order for said right-click mouse
event to be generated.

25

22. A touch system according to claim 20 wherein at
least two simultaneous pointer contacts represents
a gesture.
23. A touch system according to claim 20 wherein each
gesture is represented by specified multiple pointer
actions.
24. A touch system according to claim 23 wherein each
gesture is further represented by pointer type.

30

35

25. An interactive input system comprising:
at least one imaging device having an input region within its field of view into which one or
more pointers is moved to generate user input;

4

and
at least one processor communicating.with said
at least one imaging device and analyzing each

image acquired by said at least one imaging device to determine the action of pointers in said

45

input region, said at least one processor determiningwhenmultiplepointeractionswithinsaid
input region represent a gesture, when said
multiple pointer actions represent a gesture,

50

said at least one processor executing a command corresponding to said gesture.
26. An interactive input system according to claim 25
wherein each gesture is further represented by
pointer type.

55

21 in an interachve touch system, a method of simo

10
XJCID <EP

TS1722BA2 E >

APLNDC00027273

EP 1 517 228 A2

52

CAMERA 1

70
CAMERA 2

119
CAMERA 0

70

70

58
CAMERA 3
70

54
MASTER DSP

COMPUTER

se

FIG. 1
52

sa

84

68

88

.

64

FIG. 2

11
EP

1517228A2

APLNDC00027274

EP 1 517 228 A2

62

60

60

FIG. 3a

FIG. 3b

60

FIG. 3c

FIG. 3d

12
30010: <EP

1517228A

>

APLNDC00027275

EP 1 517 228 A2

so

FIG. 4a

FIG. 4b

FIG. 4c

FIG. 4¢

©o

FIG. 4e

FIG. 4f

13

APLNDC00027276

APLNDC00027277

lHI lililliMIllIII
(11)
(12)

EP 1 517 228 A3

EUROPEAN PATENT APPLICATION

(88) Date of publication A3:
02.04.2008 Bulletin 2008/14

(51) Int CL:
GD6F3/033f2°°6 11

(43) Date of publication A2:
23.03.2005 Bultetin 2005/12
(21) Application number: 04255568.0
(22) Date of filing: 15.09.2004

(84) Designated Contracting States:
AT BE BG CH CY CZ DE DK EE ES FI FR GB GR
HU IE IT Li LU MC NL PL PT RO SE Si SK TR
Designated Extension States:
AL HR LT LV MK

• Morrison, Gerald D.
Calgary, Alberta T3G 4T6 (CA)

(30) Priority: 16.09.2003 US 662813

(74) Representative: Naismith, Robert Stewart et al
Marks & Clerk Scotland
19 Royal Exchange Squate
Glasgow, G1 3AE
Scotland (GB)

(71) Applicant: Smart Technologies, Inc.
Calgary, Alberte T3c OMS (CA)

(54)

(72) inventors:
• Hiti, Douglas B.
Calgary, Alberta T2K 1Z2 (CA)

Gesture recognition method and touch system incorporating the same

(57)
A gesture recognition method includes detecting
multiple pointers in close proximity to a touch surface to

being used to perform a known gesture, a command associated with the gesture is executed. A touch system

determine if the multiple pointers are being used to perform a known gesture. When the multiple pointers are

incorporating the gesture recognition method is also provided.

60

FIG. 4a
co
CM
CM

LU
F teny somessoo: PAnis (m

(Cont next page)

APLNDC00027278

EP 1 517 228 A3

60

F

/

60

FIG. 4c

60

FIG. 4d

FIG. 4e

2
YMPr <F

1FØ799RA3

(Cont. next page)

>

APLNDC00027279

EP 1 517 228 A3

F G. 4f

3

APLNDC00027280

EP 1 517228A3

European Patent

EUROPEAN SEARCH REPORT

Appucanonnemee

onde

EP 04 25 5568

DOCUMENTS CONSIDERED TO BE RELEVANT
cate

citation of clocument with ndication, where appropriate,

Aeievant

CLAsswicATioNOF THE

tocoxim

ofrelevantpassages

APrucATsoN UPci

X

US 4 746 770 A (MCAVINNEY PAUL [US])
24 May 1988 (1988-65-24)
* column 3, line 15 - column 4. line 49;
figure 1 *
* column 7, line 17 - column 8, line 18;
figures 10-16 *

1-29

X

WO 98/07112 A (SYMBIOS LOGIC INC [US];
GILL DAVID ALAN [GB) SYMBIOS LOGIC INC

INV.
GO6F3/033

1-29

[US); G) 19 February 1998 (1998-02-19)

* page 4, lines 1-9 *
* page 6, line 22 - page 9, line 28 *
X

WO 99/28812 A (NORTHERN TELECOM LTD (CA])
10 June 1999 (1999-06-10)
* the whole document *

1-29

X

WO 02/08881 A (QINETIQ LTD [GB];
WORTHINGTON CHRISTOPHER [GB]; ANDREW IAN
DUNCAN CAMP) 31 January 2002 (2002-01-31)
* page 13, line 11 - page 14, line 9;

1-29

WO 03/054681 A (NOKIA CORP [FI]; PIHLAJA

11

TECHNICALFlEUX
seaneneo nec
606F

figures 4,5 *
A

A

PEKKA [FI)) 3 July 2003 (2003-07-03)
* the whole document *

US 6 278 443 B1 (AMRO HATIM YOUSEF [US) ET 24,26
AL) 21 August 2001 (2001-08-21)
* page 10, last paragraph - page 11,
paragraph 1; figures 13,14 *

Berlin
CATEGORY OF OfTED DOCUMENTS
X : particulady relevant 4 taken alone
Y : particulariy redevant à combined with another
document of the same category
A : techociogicalinsch:grotmd
O: non-writien disodesure
P ntermediate document

25 February 2008

Mouton, Benjamin

T : tfory or pencipfe underlying the invemon
E : aarlier patent document, but publehedon, or
after the filing date
O doeurnenteiteg n the applionijon

L : document cited for other reasons

& : member of the sono patent family, correspondag
dCCUmenT

4

APLNDC00027281

EPT 517228A3

ANNEX TO THE EUROPEAN SEARCH REPORT
ON EUROPEAN PATENT APPLIGATION NO.

EP 04 25 5566

This annex fets the patent famdy members twielmg to the patent documents cited in the acove-mentioned European searcluegott

The members are as contened in the Eurooean Patent Office EDP fWe an
The Europes.n Patent Office is in no wav liable for mese particulars which are merely given for the purpose cHnformation.

25-02-2008
Patent document

Publication

aned in search leport

date

US 4746770

A

A

19-02-1998

Paterd family

AU

Pub¾eation

memimris)

24-05-1988

40 9807112

.

date

603643 B2

22-11-1990

AU
EP
JP

1173288 A
0279652 A2
63223819 A

18-08- 988
24-08- 988
19-09- 988

AU
EP
US

3857697 A
0919023 A2

06-03-1998
02-06-1999

6208329 81

27-03-2001

40 9928812

A

10-06-1999

US

6310610 B1

30-10-2001

WO 0208881

A

31-01-2002

AU
AU

7080801 A
2001270808 A1

05-02-2002
05-02-2002

40 03054681

A

03-07-2003

AU
EP
FI

2002366877 A1
1456740 Al
20012540 A

09-07-2003
15-09-2004
21-06-2003

US
US

2003146905 Al
2006267956 A1

07-08-2003
30-11-2006

US 6278443

B1

21-08-2001

NONE

o
o
16 For more details abord this anner : see Official Joumal of the European Patent Office, No 12/82

5

APLNDC00027282

L

APLNDC00027283

PATENT COOPERATION TREATY

PCT

From the INTERNATIONAL SEARCHING AUTHORITY
To:
BLAKELY Attn.

SOKOLOFF -

Scheller,

NOTIFICATION OF TRANSMITTAL OF
THE INTERNATIONAL SEARCH REPORT AND
THE WRITTEN OPINION OF THE INTERNATIONAL
SEARCHING AUTHORITY, OR THE DECLARATION

TAYLOR & 2APMAN LLP

Jr James C.

1279 Oakmead Parkwav

Sunnyvale, California 94085 4040
ETATS-UNIS D'AM2RIQUE

(PCT Rule 44.1)
Dâte of mailing
(day/rnonthlyear)

, , ,

Applicants or agents file reference
4860P4895PCT
intemational application No·

JUN 0 9 200Û

FOR FURTHER ACTION

ELAK

PCT/US2008/000089

O 4 / O 6 / 2 OO 8

See paragraphe t and 4 below

intemational filing date

e

(day/month/year)

04/01/2008

Applicant
APPLE INC.

L

The applicant is hereby notified that the international search report and the written opinion of the Intemational Searching
Authority have been established and are transmitted herewith,
Filing of amendments and statement under Article 19:
The applicant is entitled, if he so wishes, to amend the claims of the Intemational Application (see Rule 46):
When? The time Nmu for filing such amendments is normauy two months from the date of transmittal of the
intemational Search Report
Where? Directly to the intemational Bureau of WIPO, 34 chemin des Colombettes
1211 Geneva 20. Switzerland, Fascimile No.: (41-22) 338.82.70
For more detailed instructions, see the notes on the accompanying sheet.

2.

The applicant is hereby notified that no intemational search report will be established and that the declaration under
Article 17(2)(a) to that effect and the written opinion of the Intemational Searching Authority are transmitted herewith.

3.

With regard to the protest against payment of (an) additional fee(s) under Rule 402, the applicant is notified that:
O the protest together with the decision thereon has been transtnitted to the Intemational Bureau together with the
applicanfs request to forward the texts of both the protest and the decision thereon to the designated Offices,
no decision has been made yet on the protest; the applicant will be notified as soon as a decision is made.

4. Reminders
Shortly after the expiration of 18 months from the priority date, the international appilcation will be published by the
international Bureau. If the applicant wishes to avoid or postpone publication, a notice of withdrawal of the international
application, or of the priority claim, must reach the international Bureau as provided in Rules 90bis.1 and 90bis.3, respectively,
before the completîon of the technicâl preparations for intemational publication.
The applicant may submit comments on an informal basis on the written opinion of the international Searching Authority to the
intemational Bureau. The intemational Bureau will send a copy of such comments to all designated Offices unless an
intemational preliminary examination report has been or is to be established, These comments would also be made avaNable to
the public but not before the expiration of 30 months from the priority date.

Within 19 months from the priority dates but only in respect of some designated Offices, a demand for international preliminary

examination must be fited if the applicant wishes to postpone the entry into the national phase until 30 months from the priority
date (in some Offices even later); otherwise, the applicant must, within 20 months from the priority date, perform the prescribed

acts for entry into the national phase before those designated Offices.

In respect of other designated Offices, the time limit of 30 months (or later) will apply even if no demand is filed within 19

months.

See the Annex to Form PCT/tB/301 and, for details about the applicable time limits, Office by Office, see the PCT Applicants
Guide, Volume 11, National Chapters and the WIPO Internet site.

Name and mening address of the internationai Searching Authority

Authodzed officet

European Patent Office. PB 5818 Patentiaan 2
NL-2280 HV R4ew k
Tet ([phone], Tx, 31 651 epo nL

Pax: (÷31-7ö} 340-3016
Form PCT/ISA/220 (October 2005)

.
Date in to SV F
Ma rj a B romdSW4b d by: W

-

Reviewed by:
Sant MthyJW

Ry

00

rtir

(See notes on accompanying speet)

APLNDC00027284

NOTES TO FORM PCTIIS"'
These Notes are intended to give the basic instructions conceming the filing of amendments under article 19. The
Notes are based on the requirements of the Patent Cooperation Treaty, the Regulations and the Administrative instructions

under that Treaty. In case of discrepancy between these Notes and those requirements, the latter are apolicable. For more

detailed information, see also the PCT Applicants Guide, a publication of WIPO.

In these Notes, "Article", "Rule", and "Section" refer to the provisions of the PCT, the PCT Regulations and the PCT

Administrative instructions, respectively.

INSTRUCTIONS CONCERNING AMENDMENTS UNDER ARTICLE 19
The applicant has, after having received the intemational search report and the written opinion of the intemationaî

Searching Authority, one opportunity to amend the claims of the intemational application, it should however be emphasized

that, since a!! parts of the intemational application (cíaims.description and drawings) may be amended during the
intemationaî prenminary examination procedure, there is usually no need to file amendments of the claims under Article 19
except where, e.g. the applicant wants the latter to be published for the purposes of provisional protection or has another

reason for amending the claims before intemational publication. Furthermore, it should be emphasized that provisional
protection is available in some States only (see PCT Applicants Guide, Volume VA, Annexes BT and B2)

The âttention of the applicant is drawn to the fact that amendments to the claims under Article 19 are not allowed where
the International Searching Authority has declared, under Article 17(2), that no intemational search report would be

established (see PCTApplicant's Guide, Volume t/A, paragraph 296).
What parts of the international application may be amended?
Under Article 19, only the claims may be amended.

During the international phase, the claims may also be amended (or further amended) under Article 34 before the

Irtternational Preliminary Examining Authority. The description and drawings may only be amended under
Article 34 before the International Examining Authority.

Upon entry into the national phase, all parts of the intemational application may be amended under Article 28 or,

where applicable, Article 4L

When?

Within 2 months from the date of transmittal of the intemational search report or 16 months from the priority date,
whichever time limit expires later. It should be noted, however, that the amendments will be considered as having

been received on time if they are received by the laternational Bureau after the e×piration of the applicable time

limit but before the completion of the technical preparations for intemational publication (Aule 46.1).

Where not to fue the amendments?
The amendments may only be filed with the fritemational Bureau and not with the receiving Office or the
Intemationâl Searching Authority (Rule 462),

Where a demand for înternational preliminary e×amination has beenlis filed, see below.
How?

Either by cancelling one or more entire claims, by adding one or more new claims or by amending the text of one
or more of the claims as filed.

A replacement sheet must be submitted for each sheet of the claims which, on account of an amendment or

amendments, differs from the sheet originally filed.

All the claims appearing on a replacement sheet must be numbered in Arabic numerals. Where a claim is
cancelled, no renumbering of the other claims is required. In all cases where claims are renumbered, they must be
renumbered consecutively (Section 205(b)).
The amendments must be made in the language in which the international application is to be published.
What documents must/may accompany the amendments?
Letter (Section 205(b)):
The amendments must be submitted with a letter.
The letter will not be published with the international application and the amended claims. It should not be
confused with the "Statement under Article 19(1)" (see below, under "Statement under Article 19(t)").
The letter must be in English or French, at the choice of the applicant However, if the language of the
international applicatiore is Enghsh, the letter raust be in EngHah; if the language of the international appiscation
is French, the letter must be in French.

Notes to Form PCT/ISA/220 (first sheet) (October 2005)

APLNDC00027285

PATENT COOPERATION TREATY

PCT
INTERNATIONAL SEARCH REPORT
(PCT Article 18 and Rules 43 and 44)

Applicânt's or agenfs file reference
4 860 P4 8 95 PCT

FOR FURTHER

see Form PCT/ISA/220

ACTION

înternational application No-

as wen as, where appucable, item 5 below.

Intemational f¾ing date (day/month/year}

PCT/Us2008/000089
Applicant

\

(Earliest) PriorÑ|ÕÄÏ

04/01/2008

ÑÃy-

07/01/2007

APPLE INC.

This intemational search report has been prepared by this latemational Searching Authority and is transmitted to the applicant
according to Article 18. A copy is being transmitted to the intemational Bureau..
This international search report consists of a total of

3

sheets.

It is also accompanied by a copy of each prior art document cited in this report,

1. Basis of the report
a, With regard to the language, the international search was carried out on the basis of:
the intemational application in the language in which it was filed
O
a imosiason ome satsmasona, avereason into
. walors me lanossos
of a translation fumished for the purposes of intemational search (Rules 12.3(a) and 23.1(b))
b.

This intemationaî search report has been established taking into account the rectification of an obvious mistake
authorized by or notified to this Authority under Rule 91 (Rule 43.6bis(â)}.

c.

With regard to any nucleotide and/or amino acid sequence discîosed in the intemational application, see Box No. L

2.

Conan clans were found unseardable (See Box No. 0)

3.

Unity of invention is lacking (see Bo× No 10)

4.

With regard to the title,

O

5

the text is approved as submitted by the applicant
the text has been established by this Authority to read as follows:

With regard to the abstract,
the text is approved as submitted by the applicant

the text has been established, according to Rule 38.2(b), by thie Authority as it appears in Box No. IV. The applicant
may, within one month from the date of maliing of this intemational search report, submit comments to this Authority
6.

With regard to the drawings,
a. the figure of the drawings to be published with the abstract is Figure No. 4
as suggested by the app¾cant
as selected by this Authority, becâuse the appticant faned to suggest a figure

b.

as selected by this Authority. because this figure better characterizes the invemion
none of the figures is to be published with the abstract

Form PCT/ISA/210 (first sheet) (April2007)

APLNDC00027286

INTERNATIONAL SEARCH REPORT

international app¾cation No

PCT/Us2008/000089
A. CLASSIFICATI

INV.

SUBAGT MATTER

GO6F3 048

According to intomational Patent Classification (IPC) of to both national classification and IPC
B. FlELDS SEARCHED
Minimum documentation searched (classitication System followed by classification symbols)

GO6F
Documentanon searched other than rmnimum documentation to the extent that such documents are included in the fields searched

Electronic data base consulted danng the miemationai search (name of data base and. where practical search terms used)

EPO-Internal

C. DOCUMENTS CONSIDERED TO BE RELEVANT

Category'

Cilation of document with indicatiort, where appropriate, of the relevant passages

X

EP 1 517 228 A (SMART TECHNOLOGIES INC
[CA)) 23 March 2005 (2005-03-23)
paragraphs [0009],
[0031]

[0024],

Relevant to claim No

1-51,
74-77,
80-88

[0027],

X

US 2003/174149 Al (FUDISAKI HITOMI [JP] ET
AL) 18 September 2003 (2003-09-18)
paragraphs [0001] - [0017]

52-73,
78,79

A

US 6 958 749 B1 (MATSUSHITA NOBUYUKI (JP]
ET AL) 25 October 2005 (2005-10-25)

1-88

O Further documents are listed in the continuation of Box C

Special categones of cited documents :
*A" document defining the general state of the art which is not

considered to be of particular relevance

'E' earlier document but published on or after the intemational
nRag date

'L' document whîch my throw doubts ora priority claim(s} or

which is cited to estaMsh the publication date of another

citation or other special reason (as specified)

*O' document referring to an oral disclosure, use, exhibdion or
other means
"P' document published prior to the intomational filing date but
:aler than the priority date claimed

Date of the actual completion of the international search

27 May 2008

'T' later document published after the international fding date
or priority date and not in conflict with the application but
cited to understand the principle or theory underlying the

mvention
•X' document of particular relevance; the claimed invention
cannot be considered novel or cannot be considered to
involve an inventive step when the document is taken alone
'Y' document of parlicular relevance; the claimed invention
canoc4 be considered to involve an inventive step when the
document is combîned with one or more other such documents, such combina1îcn being obvious to a person skilled

in the art
*&' document member of the same patent family

Date of mailing of the intemational search repon

04/06/2008

Nam and rradirra arkirm of me ISA

Europm Palent

See patent family annex.

Auth<:>rized officer

k:B, PB 5618 Paîentiaan 2

NL - 2¾ HV R4ewgk
Tel [phone], TX 37 651 epo nL
Fax: (+3 70}340-N16

Davenport, Kevin

Fomi PCTllSAl210 (second shee0 (ApriE 2004)

APLNDC00027287

INTERNATIONAL SEARCH REPORT
information on patent family members

Patent document
cited in search report

PuNication
date

intemational application No

PCT/Us2008/ooooa9
Patent family
member(s)

PubHcation
date

EP 1517228

A

23-03-2005

CA
JP
US

2481396 Al
2005108211 A
2005057524 Al

16-03-2005
21-04-2005
17-03-2005

US 2003174149

Al

18-09-2003

JP
JP

3951727 B2
2003233455 A

01-08-2007
22-08-2003

US 6958749

B1

25-10-2005

JP

2001134382 A

18-05-2001

i

Form PCT/ISAl210 (patent family annex) (AprB2005)

APLNDC00027288

PATENT COOPERATION TREATY
From the

1NTERNATIONAL SEARCHiNG AUTHORlTY

To:

PCT
WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING AUTHORITY
(PCT Rule 43bis.1)

see form PCTASA220

Date of manmg
any tronth yea i
Applicanfs or agents file reference
see form PCTASA220

see form PCTASA210 (second sheet)

FOR FURTHER ACTION
See paragraph ? below

intemational application No

nÑnational filing date (daymonthyear)

Priority date (dayänonthyear)

PCTAJS2OO8DOOOS9

04.OL2008

07.002007

intemational Pateni Classification (IPC}or both national classification and IPC

INV, GO6F3D48
Applicant

APPLE INC.
L

This opinion contains indications relating to the following items:
Bo×
O Box
O Box
O Box
Box

O Box No. Vl
O Box No. Vil

Basis of the opinion
Priority
Non-establishment of opinion with regard to novelty, inventive step and industrial applicability
Lack of unity of invention
Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrial
applicability; citations and explanations supporting such statement
Certain documents cited
Certain defects in the international application

O Box No. Vill

2.

No.
No.
No.
No.
No.

I
Il
til
IV
V

Certain observations on the international application

FURTHER ACTION
If a demand for international preliminary examination is made, this opinion will usually be considered to be a
written opinion of the International Preliminary Examining Authority ("lPEA ) except that this does not apply where
the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notifed the
Intemational Bureau under Rule 66.1bis(b) that written opinions of this International Searching Authority
will not be so considered,
if this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to

submit to the IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months
from the date of mailing of Form PCTASA220 or before the expiration of 22 months from the priority date,
whichever expires later.
For further options, see Form PCTASA/220.
! 3.

For further details, see notes to Form PCTASA220,

- n

Date of completion of

Authorized Officer

this opinion

European Patent O¾iee - P,B.. 5818 Patent! a6Ñe form
NL-2280 HV R swijk - Pays Bas
PCTASARTO
TeE ÷31 70 340 - 2O4O TX 31 651 apo ni
Fax: ÷31 70 340 - 3016

Telephone No [phone]

Form (PCTASA237) (Cover Sheet} (April 2005)

APLNDC00027289

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING AUTHORITY

Box No. I

International application No.
PCT/US2008DOOOS9

Basis of the opinion

1. With regard to the language, this opinion has been established on the basis of:
the international application in the language in which it was filed

O

a translation of the intemational application into , which is the language of a transtation tumished for the
purposes of intemational search (Rules 12.3(a) and 23.1 (b))

2. O

This opinion has been established taking into account the rectification of an obvious mistake authorized
by or notified to this Authority under Rule 91 (Rule 43bis.1(a))

3. With regard to any nucleotide andhr amino acid sequence disclosed in the international application and
necessary to the claimed invention, this opinion has been established on the basis of:
a. type of material:
O

a sequence listing

O

table(s) related to the sequence listing

b. format of material:

O

on paper

O

in electronic form

c. time of filing/fumishing:
O

O

filed together with the international application in electronic form.

O
4. O

contained in the intemational application as filed.

furnished subsequently to this Authority for the purposes of search.

in addition, in the case that more than one version or copy of a sequence listing and/or table relating thereto
has been filed or fumished, the required statements that the information in the subsequent or additional
copies is identical to that in the application as filed or does not go beyond the application as filed, as
appropriate, were fumished.

5. Additional comments:

Form PCTASA/237 (April 2007)

APLNDC00027290

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING AUTHORITY

intemational application No
PCT/US2008DOOO89

Box No. V Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or
industrial appHcability citations and explanations supporting such statement

1. Statement
Novelty (N}

Yes: Clâlms
No: Claims

imi 7L77 anon
1-14529178Je

Inventive step (IS)

Yes: Claims
No: Claims

1-88

Yes: Claims
No: Claims

1-88

Industrial applicability (IA}

2. Citations and explanations

s_ee yearate

eet

Form PCTASA/237 (April 2007)

APLNDC00027291

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING
AUTHORITY (SEPARATE SHEET)

Intemational apptication No,
PCT/US2008/000089

Ra Itam V

Reasoned statement with regard to novelty, inventivo step or industrial applicability;
citations and explanations supporting such statement
1

Reference is made to the following documents:
D1: EP-A-1 517 228 (SMART TECHNOLOGIES INC (CA]) 23 March 2005 (200503-23)
D2: US 2003/174149 A1 (FUJlSAKI HITOMI (JP] ET AL) 18 September 2003 (200309-18)
D3: US-81-6 958 749 (MATSUSHITA NOBUYUKI [JP] ET AL) 25 October 2005
(2005-10-25)

2

2.1

The present application does not meet the criteria of Article 33(1) PCT, because the
subject-matter of claims 1,52 is not new in the sense of Article 33(2) PCT.
Regarding claim 1, the document D1 discloses a machine implemented method
for scrolling on a display of a portable data processing device comprising:
receiving a user input (§9);

creating an event object in response to the user input (the very nature of data
processing devices means that this will always occur);

determining whether the event object invokes a scroll or gesture operation (§24
- Di discloses that both scrolling and scaling are two different types of gesture;
while claim 1 does not define a scroll call as being initiated by a gesture, it may
be seen from the description that a scroll call is initiated by a movement of a
finger across the input device, which indeed constitutes a gesture; what is
significant is that there is a differentiation between two different gestures;);

issuing at least one scroll or gesture call based on invoking the scroll or gesture
operation (the very nature of data processing devices means that this will
always occur);

Form PCT/ISA/237 (Separate SheetBSheet i HER

Aprú 2005)

APLNDC00027292

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING
AUTHORITY (SEPARATE SHEET)

intemational application No.
PCT/US2008/000089

responding to at least one scroll call, if issued, by scrolling a window having a
view associated with the event object based on an amount of a scroll with the
scrofi stopped at a predetermined position in relation to the user input (§27 - " If
the user contacts the touch surface 60 with a pair of fíngers simultaneously over
an application window displayed on the touch surface and the fingers are
closely and generally horizontally spaced, the driver recognizes the
sirnultaneous finger contacts as a scroll gesture and injects a scroll event into
the application."); and
responding to at least one gesture call, if issued, by scaling the view associated
with the event object based on receiving a plurality of input points in the form of
the user input (§31- "If the usercontacts the touch surface 60 with a pair of
closely spaced fingers simultaneously over an application window and expands
the distance between the fingers in a generally horizontal direction, the driver
recognizes the finger rnovernent as a zoorn-out gesture.").

The subject-matter of claim 1 is therefore not new.
The same reasoning applies, mutatis mutandis, to the subject-matter of the
corresponding independent claim 8, which therefore is also considered not new.
2.2

2.2.1

2.3

Regarding claim 52, D2 discloses a method for operating through an application
programming interface (API) in an environment with user interface software
interacting with a software application and a user input contacting a view of a
display of a device, comprising transferring a directional scroll call to determine
if directional scrolling is enabled. See D2, §11-§17.

With regard to the method of operating through an API, the use of an
application programming interface is considered to be implicitly disclosed
in the closest prior art for each case because it is standard practice to use
an APt to interface between and input device and the software which it is
manipulating. See e.g. D3, §7-§13, §33.

The same reasoning applies, mutatis mutandis, to the subject-matter of the

Form PCT/ISAl237 (SeparateShæîMSheet 2) (EP

Aprä 2005)

APLNDC00027293

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING
AUTHORITY (SEPARATE SHEET)

intemational application No.
PCT/US2008/000089

corresponding independent claims 58,64,65,69,73,78 which therefore are also
considered not new.

3

3.1

The present application does not meet the criteria of Article 33(1) PCT, because the
subject-matter of claim 15 does not involve an inventive step in the sense of Article
33(3) PCT.
The document D1 is regarded as being the closest prior art to the subject-matter
of claim 15, and discloses a method for operating through an application
programming interface in an environment with user interface software
interacting with a software application (§9). See also §2.2.1 supra.
The subject-matter of claim 15 therefore differs herefrom in that a set bounce
call is transferred.

3.2

The problem to be solved by the present invention may therefore be rece. ded
as how to present a more visually intuitive graphical user interface.
However, this problem is merely a problem of visual design nature, and thus

cannot be used in the assessment of inventive step. The result achieved by
solving this problem, namely the simulation of the natural phenomenon of
bouncing of one object against another before eventually coming to rest has no

technical effect on the usability of the interface by the user, and does not lead to
an improved user - machine interaction. The claims merely define a solution
which does not lead to technical effects once they are applied to a real world
situation.
3.3

The ISA does not dispute the fact that technical means may be involved in the

implementation of a more visually intuitive graphical user interface. However, a
technical problem solved by this combination of technical means is not
apparent.

Therefore, the technical problem has to be seen as how to implement a method

Forrn PCTASAl237 (Separate Sheet) (Sheet 3) (EPC>Apúl 2005)

APLNDC00027294

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING
AUTHORITY (SEPARATE SHEET)

Intemational application No.
PCT/US2008/000089

of presenting a more visually intuitive graphical user interface.
The inventive activity is then assessed using the general technical knowledge of
the skilled person, in the sense of Article 33(3) PCT, which, in the present case
is a programmer of graphical user interfaces, acting on instructions (see §2.8
supra) received from the person in charge of defining the visual design of the
interface.
Starting from this knowledge, the skilled person, aware of the design
requirements of the person in charge of defining the visual design of the
interlace, would arrive at the proposed solution without the exercise of an

inventive step. The implementation details described in the application, are well
known algorithms used in the field of programming for looping through
sequential lists. In general, said implementation does not require any technical
considerations that go beyond the skilled person's normal technical abilities.

The same reasoning applies, mutatis mutandis, to the subject-matter of the
corresponding independent claims 20,25,26,29,32,74,80,86 which therefore are

also considered not inventive.
4

The document D1 is regarded as being the closest prior art to the subject-matter of
claim 33, and discloses a method for operating through an application programming
interface in an environment with user interface kit interacting with a software
application (§9). See also §2.2.1 supra.

4.1

The subject-matter of claim 33 therefore differs herefrom in that a rubberband
call to cause a scrolled region displayed within a region of a device to
rubberband is transferred.

4.2

The problem to be solved by the present invention may therefore be regarded
as how to present a more visually intuitive graphical user interface.
However, this problem is merely a problem of visual design nature, and thus

cannot be used in the assessment of inventive step. The result achieved by

APLNDC00027295

WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING
AUTHORITY (SEPARATE SHEET)

International application No.

PCT/US2008/000089

solving this problem, namely the simulation of the natural phenomenon of the
behaviour of two objects elastically bound to each other has no technical effect
on the usability of the interface by the user, and does not lead to an improved

user - machine interaction. The claims merely define a solution which does not
lead to technical effects once they are applied to a reat world situation.
4.3

A similar argumentation to that of §3.3 supra applies.
The same reasoning applies, mutatis mutandis, to the subject-matter of the
corresponding independent claims 39,44,45,48,51,76 which therefore are also
considered not inventive.

5

The dependent claims do not contain any features which, in combination with the

features of any claim to which they refer, meet the requirements of the PCT in
respect of novelty and/or inventive step. See documents D1-D3 and the
corresponding passages cited in the search report.

Form PCTilSA/237 Separate Sheet} (Sheef 5) (EP

AprR 2005)

APLNDC00027296

Possible steps after receipt of the inte<national search report (ISR) and
written opinion of the International Searching Authority (WO-ISA)
General

For all international applications filed on or after 01/01/2004 the competent

information

ISA will establish an ISR It is accompanied by the WO-ISA. Línlike the
former written opinion of the IPEA (Rule 66.2 PCT), the WO-ISA is not
meant to be responded to, but to be taken into consideration for further
procedural steps. This document explains about the possibilities.

Amending claims Within 2 months after the date of mailing of the ISA and the WO-ISA the
under
Art. 19 PCT

applicant may file amended claims under Art. 19 PCT directly with the
I nternational Bureau of WI PO. The PCT reform of 2004 did not change
this procedure. For further information please see Rule 46 PCT as well as
form PCT/ISN220 and the corresponding Notes to form PCT/tSN220.

Filing a demand
for international
preliminary

in principle, the WO-ISA will be considered as the written opinion of the
iPEA. This should, in many cases, make it unnecessary to file a demand for
international preliminary examination. If the applicant nevertheless wishes

examination

to file a demand this must be done before expiry of 3 months after the
date of mailing of the ISR/WO-ISA or 22 months after priority
date, whichever expires later (Rule 54bis PCT). Amendments under Art.
34 PCT can be filed with the IPEA as before, normally at the same time as
filing the demand (Rule 66.1 (b) FCT).
If a demand for international preliminary examination is filed and no

comments/amendments have been received the WO-ISA will be transformed
by the IPEA into an IPRP (International Preliminary Report on Patentability)

which would merely reflect the content of the WO-ISA. The demand can still
be withdrawn (Art. 37 PCT).
Filing informal
comments

After receipt of the ISFVWO-ISA the applicant may file informal comments
on the WO-ISA directly with the I nternational Bureau of WI PO. These
will be communicated to the designated Offices together with the I PRP

(international Preliminary Report on Patentability) at 30 months from the
priority date. Rease also refer to the next box.

End of the
international
phase

Relevant PCT
Ru les and more
inf orm at son

1000

At the end of the international phase the International Bureau of WIPO will
transform the WO-fŠA or, if a demand was filed, the written opinion of the
IPEA into the IPRP, which will then be transmitted together with possible
informal comments to the designated Offices. The IPRP replaces the former
IPER (international preliminary examination report).
Rule 43 PCT, Rule 43bis PCT, Rule 44 PCT, Rule 44bis PCT, PCT Newsletter
12/2003, Œ11/2003, Œ12/2003

<XS

2U70&M

K i >

APLNDC00027297

PATENT COOPERATION TREATY

From the INTERNATIONAL SEARCHING AUTHOAITY
To:

NOTIFICATION OF TRANSMITTAL OF
BLAKELY,

SOK

DFF,

TATL R &

ZAFMAN

THE INTERNATIONAL SEARCH REPORT AND

Acca, schener, or James
tro oakmaae Parkway
3untry,7ale,

Ca.1.ifornia

THE WAtTTEN OPINiON OF THE!NTEANATIONAL
SEARCHING AUTHORITY, OR THE DECLARA.TION

94085 4040

(PCT Rule 44.1)
Date of mainng

Applicants or agents file reference
4860 P5040PCT

'
FOR FURTHER ACTION

intemational application No.

PCT/US2008/000060

See paragraphs 1 and 4 below

inter ational fliing date

.

.

(day/month eâr)

03/01

2008

Applicant

APPLE INO.

L

The applicant is hereby notified that the intemational search report and the written opinion of the intemational Searching
Authoríty have been established and are transmitted herewith.
Filing of amendments and statement under Article 19:
The applicant is entuied, if he so wishes, to amend the claims of the intemational Application (see Rule 46);
When? The time limit for filing such amendments is normally two months íïom the date of transmittal of the
Intemational Search Report
Where? Directly to the Intemational Bureau of WIPO, 34 chemin des Colombettes
1211 Geneva 20, Switzerland, Fascimile No.: (41-22) 338.82.70
For more detailed instructions, see the notes on the accompanying sheet.

2.

The applicant is hereby notified that no internatonal search report will be established and that the declaration under
Article 17(2)(a) to that effect and the written opinion of the Intemational Searching Authority are transmitted herewith.

3.

With regard to the protest against payment of (an) additional fee(s) under Rule 40.2, the applicant is notified that
the protest together with the decision thereon has been transmitted to the international Bureau together with the
applicanfs request to forward the texts of both the protest and the decision thereon to the designated Offices.

no decision has been made yet on the protest; the applicant will be notified as soon as a decision is made,
4. Aeminders
Shortly after the expiration of 18 months from the priority date, the intemational application witi be published by the
Intemationai Bureau if the applicant wishes to avoid or postpone publication, a notice of withdrawal of the íntemâtional
applicatiore or of the priority claim, must reach the international Bureau as provided in Aules 90Dis.i and 90bis 3, respectively,
before the completion of the technical preparations for intemâtfonal publicatiort

The appheant may submit comments on an informal basis on the written opinion of the intemational Searching Authority to the
international Bureau The intemational Bureau will send a copy of such comments to âll designated Offices unless an
international preliminary examination report has been of is to be estab¾shed. These comments would also be made avahabte to
the pubha but not before the expiration of 30 months from the priority date.
Within 19 months from the priority date, but only in respect of some designated Offices. a demand for intemational preliminary
exarnination must be filed if the applicant wishes to postpone the entry into the national phase until 30 months from the priority
date On some Offices even later); otherwise, the appúcant must, within 20 months from the priority date, perform the prescribed
acts for entry into tne national phase befoæ those designated Ottices.

In respect of other designated Offices, the time ilmit of 30 months for later) win apply even if no demand is TRed within 19
months.
See the Annex to Form PCT/IS/301 and, for details about the applicable time umits, Office by Office, see the ROT Applicants
Gui<ie. Volume IL National Chapters and the WIPO intemet site.

Form PCT/ISAl220 (October 2005)

APLNDC00027298

NOTES TO FORM PCTIISA/220
These Notes are interided to give the basic instructions conceming the filing of amendments under article 19. The
Notes are based on the requirements of the Patent Cooperation Treaty, the Reguíations and the Administrative lastructions
under that Treaty. in case of discrepancy between these Notes and those requirements, the latter are aopticable. For more
detailed information. see also the PCT Applicants Guide, a pubücation of WOO.

In these Notes. "Article". "Rule", and "Section" refer to the provisions of the PCT. the PCT Regulations and the PCT
Administrative instructions, respectively.

INSTRUCTIONS CONCERNING AMENDMENTS UNDER ARTICLE 19
The appBcant has, after having received the intemational search report and the written opinion of the intematíonal

Searching Authority, one opportunity to amend the claims of the international application, it shouid however be emphasized
that since an parts of the international application (clairns,description and drawings) may be amended durireg the
international preliminary examination procedure, there is usually no need to file amendments of the claims under Article 19
except where, e.g. the applicant wants the latter to be pubüsted for the purposes of provisional protection or has another
reason for amending the claims before international pubücation Furthermore, it should be emphasized that provisional
protection is availabîe in some States only (see PCT AppRcanfs Guíde, Volume I/A, Annexes 81 and 82).
The attention of the applicant is drawn to the fact that ameridments to the claims under Article 19 are not allowed where
the International Searching Authority has declared, under Article 17(2), that no internâtional Search report would be

estâbiished (see POT Applicanfs Guide, Volume I/A, paragraph 29ô).
What parts of the international application may be amended?

Under Article 19, only the claims may be amended.
During the intemational phase, the claims may also be amertded (or further amended) under Article 34 before the
international Preliminary Examining Authority. The description and drawings may only be amended under
Article 34 before the International Examirung Authority.
Upon entry into the national phase, ad parts of the intamational application may be amended under Article 28 or,

where applicable, Article 41.
When?

Within 2 months from the date of transmittai of the intemational search report or 16 months from the priority date,
whichever time Mmit expires later. It should be noted, however, that the amendments win be considered as having
been received on tirne if they are received by the Intemational Bureau after the expiration of the applicable time
limit but before the completion of the technical preparations for intemational publication (Rute 46-1¾

Where not to file the amendments?
The amendments may only be filed with the totemational Bureau and not with the receiving Office or the
intemational Searching Authority (Rule 46 2)
Where a demand for international preliminary examination has beenlis filed. see below.

How?

Either by cancelling one or more entire claims, by adding one or more new claims or by amending the text of one
or more of the claims âs filed.
A replacement sheet must be submitted for each sheet of the claims which, on account of an amendment or
amendments, differs from the sheet originally filed,

All the claims appearing on a replacement sheet must be numbered in Arabic numerals. Where a claim is
canceled, no renumberíng of the other claims is required, in aM cases where claims are renumbered. they must be
renumbered consecutively (Section 205(b)).
The amendments must be made in the language in which the international application is to be pubHshed.
What documents must/may accompany the amendments?
Letter (Section 205(b)):
The amendments must be submítted with a letter.
The tetter wili not be pubBshed with the international application and the amended claims. It should not be
confused with the "Statement under Article 19(1 (see below, under "Statement under Article 19(if)

The letter must be in EngHsh or French, at the choice of the appücant However, if the language of the
international appdcation is English, the letter must be in Engnan; if the language of the intamational appneation
is French, the letter must be in French.

Notes to Form PCT/ISA/220 (first sheet) (October 2005)

APLNDC00027299

PATENT COOPERATION TREATY

PCT
INTERNATIONAL SEARCH REPORT
(PCT Adicle 18 and Rules 43 and 44)
AppäCanfSof agenfs fue reference

FOR FURTHER

486OP504 OPCT

see Form PCT/ISA/220

ACTION

Intemational appucation No.

as wen as. where appucaele, item a below.

Intemâtional finng date Iday/month/year)

ecT/us2ooa/ooooso

(Eartiest) Priority Date (day/month/pea:)

03/01/2008

07/01/2007

AppWeant

APPLE INC.

This intemational search report has been prepared by this intemational Searching Authority and is transrnitted to the appleant
according to Article 18. A copy is being transmitted to the latemational Bureau.
This intemational search report consists of a total of

L
. sheets.
t is also accompanied by a copy of each prior art document cited in this report

1.

Basis of the report
a. With regard to the language, the internâtional search was carried out on the basis of:
the intemational applícation in the îanguage in which it was filed
a translation of the intemational app¾catíon into
, which is the language
of a translation furnished for the purposes of international search (Rules 12.3(a) and 23.1(b))
b.

This internationaf search report has been estabushed taking into account the rectification of an obvious mistake
authorized by or notified to this Authority under Rule 91 (Rule 43.6bis(a)).

c.

With regard to any nucleotide and/or amino acid sequence disclosed in the intomational application, see Box No. L

2,

Certain claims were found unsearchable (See Box No. 11)

3.

Unity of invention is lacking (see Box No !!!}

4_

With regard to the title,

the text is approved as submitted by the appitcant
the text has been established by this Authority to read as follows:

5.. With regard to the abstract,
the text is approved as submRied by the applicant

the text has been established, according to Rule 38,2(b), by tNs Authority as it appears in Box No. IV. The appleant
may, wdhin one month from the date of mening of this intemational search report, submit comments to this Authority
a 9

cy *
-1:

* - drawmgs

-:

'odrawmgsi

a

---,

s -t,

,-

eN

4

,tion

APLNDC00027300

INTERNATIONAL SEARCH REPORT

toternational application No

PCT/US2008/000060
A. CLASSfFICATION

INV.

F SUBJECT MATTER

GO6F3 048.

According lo înternational Patent Glassnication (IPCs or to tWXh national classificadon and IPC
B. FIELDS SEA ACHED

Minimum documentation searched (ciassWicadon system tonowed by classifica¾on symbols)

GO6F

Eîectronic data base consuned during the intemational search (name of data base and, where pract cat search terras used}

EPO-Internal

C. DOCUMENTS CONSIDERED TO BE RELEVANT

Category*

Citation of document with indication, where appropriate. onhe relevant passages

X

Reîevanno claim No

US 6 958 749 B1 (MATSUSHITA NOBUYUKI [JP]

1-33,
35-96

US 5 534 893 A (HANSEN JR DANIEL J LUS] ET
AL) 9 July 1996 (1996-07-09)
abstract
column 2, line 33 - line 59
column 5, line 1 - column 9, line 46;
figures 1-4

1-33,
35-96

ET AL) 25 October 2005 (2005-10-25)
the whole document
X

O Fuither documents are foted in the continuation of Box C
Special categories of cited documents :

*A' document defining the general state of the art «Noi, is not
considered to be of parucular relevance

'E' earher document out published on or after the irdemahonal
filing date

*U documed which may throw doubtson priority claim(s) or
wMoh is cited to establish the pub¾eation date of another
citation or other special reason las specified)

'O' document referNog to an oral disclosure. Use. exhibdion or

other means

"P' document published prior to the intemahond filing date but
later than the priorny date claimed

Date of the actual completion of the internahonal search

See patent family annex.
T' later document published after the internabonai tWing date
or priority date and not in conflict with the appúcanort but
cited to Emderstand the principle of theory underlying the
invention

X' document of pardeular relevanæ: the cialmed invenhon

cannot be considered novel or cannot be considered to

involve an inventive step when the documenOs taken alone
Y' document of padicular relevance; the claimed invention

cannot be considered to involve an invenhve step when the
documenns combinerJ WRh one or rnere other such documents. such combinahan being obvious to a. person skmed
in the att

&* document member of the same patent famdy

Date of malfing of the intemaßenal search report

22/04/2008
Authorized omcer

2

Vieira,
L
Form PCTñSA/2 iD isecond sneeg (Apr 2005)

APLNDC00027301

INTERNATIONAL SEARCH REPORT
information on patent famNy members

International appheation No

PcT/us2008/000060

Patent document

Publication

Patent famuy

Pubi!Cation

cited in search report

date

memberís)

date

US 6958749

B1

25-10-2005

JP

US 5534893

A

09-07-1996

2001134382 A

18-05-2001

NONE

APLNDC00027302

PATENT COOPERATION TREATY
From the

INTERNATIONAL SEARCHING AUTHORITY

PCT
WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING AUTHORITY
(PCT Rule 43bis.1)

see form PCTilSA;220

Appucants or agent's file referene"

FOR FURTHER ACTION

see form PCTASA/220
international application No.

PCTAJS2008:000060

: see paragraph 2 oelow
întemational filing date (daymonthyear)

i 03,01 2008

Priority date idayhonthgear)

07.01 2007

intemational Patent Classification (IPC) or both national classification and IPC

INV. GO6F3/048
Applicant

APPLE INC.
L

This opinion contains indications relating to the following items:
Box No. I

Basis of the opinion

O Box No. Il

Priority

Box No. til
O Box No, IV

Non-establishment of opinion with regard to novelty, inventive step and industrial applicability
Lack of unity of invention

Box No. V
Bo× No. Vi
Box No. Vil
O Box No. VUI

2,

Reasoned statement under Ruie 43bis..1(a)(i) with regard to novelty, inventive step or índustrial
applicability; citations and explanations supporting such statement
certain documents cited
Certain defects in the intemational application
Certain observations on the international application

FURTHER ACTION
If a demand for intemational preliminary e×amination is made, this opinion will usually be considered to be a
written opinion of the intemational Preliminary Examining Authority ("iPEA") except that this does not apply where
the applicant chooses an Authority other than this one to be the IPEA and the onosen IPEA has notifed the
Intemational Bureau under Rule 66 bis(b) that written opinions of this intemational Searching Authority
will not be so considered.
If this opinion is, as provided above, considered to be a written opinion of the iPEA, the applicant is invitaci to
submit to the iPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months
from the date of mailing of Form PCT/ISA220 or before the expiration of 22 months from the priority date,
whichever expires later
For further options. see Form POTASA220

3.

For further detals, see notes to Form PCTASA220.

Name and mailing address of the ISA

APLNDC00027303

WRITTEN OPINION OF THE

intemational application No.

INTERNATIONAL SEARCHING AUTHORITY

PCT/US2008;DOOO60

Box No, I

Basis of the opinion

1. With regard to the language, this opinion has been established on the basis of:

O

the intemational application in the language in which it was filed

O

a translation of the intemational application into , which is the language of a translation fumished for the

2. O

purposes of international search (Rules 12,3(a) and 23.1 (b)).

This opinion has been established taking into account the rectification of an obvious mistake authorized
by or notified to this Authority under Rule 91 (Rule 43bis.1(a))

3. With regard to any nucleotide and/or amino acid sequence disclosed in the intemational application and
necessary to the claimed inventiort this opinion has been established on the basis of:

a. type of material:
O

a sequence listing

O

table(s) related to the sequence listing

b. format of material:
O

on paper

O

in electronic form

c, time of filingßumishing:
O
O

filed together with the intemational application in electronic form.

O
4. O

contained in the intemational application as fited.

fumished subsequently to this Authority for the purposes of search.

In addition, in the case that more than one version or copy of a sequence listing ander table relating thereto
has been filed or furnished, the required statements that the information in the subsequent or additional
copies is identical to that in the application as filed or does not go beyond the application as filed, as
appropriate, were furnished.

5. Addítional comments:

i orm PCTASA/237 (April 2007)

APLNDC00027304