Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 462

Declaration of DEOK KEUN Matthew Ahn IN SUPPORT OF #461 APPLES OPENING CLAIM CONSTRUCTION BRIEF PURSUANT TO PATENT L.R. 4-5 filed by Apple Inc.(a California corporation). (Attachments: #1 Exhibit A, #2 Exhibit B Part 1, #3 Exhibit B Part 2, #4 Exhibit C Part 1, #5 Exhibit C Part 2, #6 Exhibit D Part 1, #7 Exhibit D Part 2, #8 Exhibit D Part 3, #9 Exhibit D Part 4, #10 Exhibit E Part 1, #11 Exhibit E Part 2, #12 Exhibit F, #13 Exhibit G, #14 Exhibit H, #15 Exhibit I, #16 Exhibit J, #17 Exhibit K, #18 Exhibit L, #19 Exhibit M Part 1, #20 Exhibit M Part 2, #21 Exhibit N, #22 Exhibit O, #23 Exhibit P, #24 Exhibit Q)(Jacobs, Michael) (Filed on 12/8/2011) Modified on 12/9/2011 linking entry to document #461 (dhm, COURT STAFF).

U.S. Patent

Dec. 23, 2008

(

r

Optical

Sheet 27 of 38

US 7,469,381 B2

) / Proximity '

i Sensor1M J « Sensor M,

To Figure 15B

1506

( Microphone )

Home

Accelerometer(s)

Figure 15A

sustas E---.

-

A

I El Add

APLNDC00022496

U.S. Patent

Dec. 23, 2008

Speaker 111

Sheet 28 of 38

'Optical Sensor
igg

iMicrophone 113
\

Home

US 7,469,381 B2

Proximity '
Sensor 1§û ,

Accelerometer(s)

Figure 15B

ranu nrnvidad hv USPTO from the PIRS Imaae Database on 04/25/2011

APLNDC00022497

U.S. Patent

Dec. 23, 2008

/
'
\ Speaker 111

'

Sheet 29 of 38

Optical
Sensor 16_4

US 7,469,381 B2

Proximity )
Sensor 1ûû )

"---......-/

"---.. ....

To Figure 15D

1508

Microphone '

Home

Accelerometer(s)

Figure 15C

n-.. .--..:.s-i L., I senwa 8..... eks DIDO innassa nainheen man n

95/9011

APLNDC00022498

U.S. Patent

Dec. 23, 2008

Optical '
Sensor j.§

Sheet 30 of 38

US 7,469,381 B2

/ Proximity '
k Sensor i

1502

To Figure 15E

1508

( Microphone )

Home

Accelerometer(s)

Figure 15D

APLNDC00022499

U.S. Patent

Dec. 23, 2008

Speaker 111

Sheet 31 of 38

/ Optical '
s Sensor 16&

US 7,469,381 B2

Proximity '
Sensor lâ.û,

1502

Microphone '

Home

Accelerometer(s)

Figure 15E

m.mas .awanaisimmi kaa i i

DTT% Swam tha DIDO imana

ara

APLNDC00022500

U.S. Patent

1604-L

Dec. 23, 2008

Sheet 32 of 38

(speaker111)

US 7,469,381 B2

() (si,

1606-L

To Figure 16B

1606-R

Microphone '

Home

Accelerometer(s)

1604-R

Figure 16A

APLNDC00022501

U.S. Patent

Dec. 23, 2008

Sheet 33 of 38

US 7,469,381 B2

1604-R

1608-R

1604-L

' \

1512

1608-L

Figure 16B

nanu nmvidad hu usPTO from the PIRS Immae Database on 04/25/2011

APLNDC00022502

I

U.S. Patent

Dec. 23, 2008

*

Sheet 34 of 38

'°a.

i
i

US 7,469,381 B2

1606-L

1
f

1608-L '

If
/

i

-

Figure 16C

pa--.. ......ia-A km. El

DTtn Smaasm tha DIDO ismassa Aminknen an AAN¾

Aid

APLNDC00022503

U.S. Patent

Dec. 23, 2008

Sheet 35 of 38

US 7,469,381 B2

1604-L
(( optical
)
Speaker111
Sensor jgg

{ Froxmty )
k Sensor 166 J

1606-L

To Figure 16E

1606-R

mammassame

Mico one

ccele

e

eter(8

1604-R

Figure 16D

- natenson

APLNDC00022504

I

U.S. Patent

Dec. 23, 2008

1604-L

Sheet 36 of 38

\

US 7,469,381 B2

) ( Optical ) ( Proximity )
Speaker11_1 i Sensor 164 A L Sensor 166 1

'-.

··¾

1606-L

To Figure 16F

1606-R

( Microphone '

m

,

Home

20&

IAccelemmeter(s)

\

g

1604-R

Figure 16E

ta-.... -.-asisia.i has I IODTf3 heam tha DIRO Imana Antakaan nn OA

E

011

APLNDC00022505

U.S. Patent

Dec. 23, 2008

Sheet 37 of 38

Optical Sensor'

ea er

igi

'

US 7,469,381 B2

Proximity

'

Sensor 1û$

1502

Microphone '

Home

Accelerometer(s)

Figure 16F

APLNDC00022506

U.S. Patent

Dec. 23, 2008

Sheet 38 of 38

US 7,469,381 B2

Memory 1770
Operating System
Ôommunication Module
Contact/Motion Module

Graphics Module
Text Input Module
A¡ pitcations
Contacts Module

Device 11¾

/ 126
/ 128
; 130
132

137

Telephone Module
Video Conference Module
E-mail Client Module

140
141

Instant Messaging Module
Blogging Module

1710' CPU(s)

39

43

Camera Module
17
1730
User interface

Touch-Screen
Display

1740

| Keyboard/Mouse |

1750

Network
1760 Communications
Interface

Image Management Module
Video Player Module
Music Player Module
Browsing Module
Calendar Module
Widget Modules
Weather Widget
Stocks Widget
Calculator Widget
Alarm Clock Widget
Dictionary Widget
.

User-Created Widget(s)
Ñdget Creator Module

146
147
148

11 9-1
149-2
149-3
149-4
149-5
149-6
150
151

Search Module

Figure 17

Pann nravidad hv IIRÞT

from the PIRS Imaae Database on 04/25/2011

APLNDC00022507

US 7,469,381 B2
1

2

LIST - - ·LLING AND DOCUMENT
· ·--LATION, SCALING,AND - -- - - -ON
ON A TOUCH-*• «m == DISPLAY

to access, store and manipulate data. These conventional user
interfaces often result in complicated key seq.._.. and
menu hierarchies that must be
Ized by the user.
Many conventional user interfaces, such as those that
include physical pushbuttons, are also inflexible. This is
unfortunate because it may prevent a user interface from
being configured and/oradaptedby either an application running on the portable device or by users. When coupled with
the time consuming requirement to
multiple key
sequences and menu hierarchies, and the difficulty in activating a desired pushbutton, such inflexibility is frustrating to
most users.
As a result ofthe small size of display ....-- on portable
electronic devices and the potentially large size of electronic
files, frequently only a portion of a list or of an electronic
document ofinterest to a user can be displayed on the screen
at a given time. Users thus will frequently need to scroll
displayedlists orto translate displayed electronic documents.
Users also will need to rotate and to scale (i.e., magnify or
de-magnify) displayed electronic documents. However, the
limitations of conventional user interfaces can cause these
actions to be awkward to perform.
Furthermore, scrolling displayed lists and translating electronic documents can be awkward on both portable and nonportableelectroniedeviceswithtouch ......displays.Auser

•< a i a i e I = APPLICATIONS
This application claims priority to U.S. Provisional Patent
Application Nos. 60/937,993, R.LLle Multifunction
Device," filed Jun. 29, 2007; 60/946,971, "List Scrolling and
Document Translation, Scaling, and Rotation on a TouchScreen Display," filed Jun. 28, 2007; 60/945,858, "List
Scrolling and Document Translation on a Touch-Screen Display," filed Jun. 22, 2007; 60/879,469, "Portable Multifunction Device," filed Jan. 8, 2007; 60/883,801, "List Scrolling
and Document Translation on a Touch-Screen Display," filed
Jan. 7, 2007; and 60/879,253, "Portable Multifunction
Device," filed Jan. 7, 2007. All of these applications are
incorporated by reference herein in their entirety.
This application is related to the following applications: (1)
U.S. patent application Ser. No. 10/188,182, "Touch Pad For
Handheld Device," filed on Jul. 1, 2002; (2) U.S. patent application Ser. No. 10/722,948, "Touch Pad For Handheld
Device," filed on Nov. 25, 2003; (3) U.S. patent application
Ser. No. 10/643,256, "Movable Touch Pad WithAdded Functionality," filed on Aug. 18, 2003; (4) U.S. patent application
Ser. No. 10/654,108, "Ambidextrous Mouse," filed on Sep. 2,
2003; (5) U.S. patent application Ser. No. 10/840,862, "Multipoint Touchscreen," filed on May 6, 2004; (6) U.S. patent
application Ser. No. 10/903,964, "Gestures For Touch Sensitive Input Devices," filed on Jul. 30, 2004; (7) U.S. patent
application Ser. No. 11/038,590, "Mode-Based Graphical
User Interfaces For Touch Sensitive Input Devices" filed on
Jan. 18, 2005; (8) U.S. patent application Ser. No. 11/057,
050, "Display Actuator," filed on Feb. 11, 2005; (9) U.S.
Provisional Patent Application No. 60/658,777, "MultiFunctional Hand-Held Device," filed Mar. 4, 2005; (10) U.S.
patent application Ser. No. 11/367,749, "Multi-Functional
Hand-Held Device," filed Mar. 3, 2006; and (11) U.S. Provisional Patent Application No. 60/824,769, "Portable Multifunction Device," filed Sep. 6, 2006. All ofthese applications
are incorporated by reference herein in their entirety.

*

10

15

20

25

may become frustrated ifthe scrolling or translation does not

reflect the user's intent. Similarly, a user may become frustrated ifrotationand scaling ofelectronic documents does not
30 reflect the user's intent.
Accordingly, there is a need for devices with touch-screen
displays with more transparent and intuitive user interfaces
for scrolling lists of items and for translating, rotating, and
scaling electronic documents that are easy to use, configure,
and/or adapt.
SUMMARY
40

The above deficiencies and otherproblems associated with
user interfaces for portable devices and devices with touchsensitive displays are reduced or eliminated by the disclosed
device. In some embodiments, the device has a touch-sensi- CAL FIELD
tive display (also knownas a "touch screen") with a graphical
The disclosed embodiments relate generally to devices 45 user interface (GUI), one or more processors, memory and
one or more modules, programs or sets of instructions stored
with touch-screen displays, and more particularly to scrolling
in the memory for performing multiple functions. In some
lists and to translating, rotating, and scaling electronic docuembodiments, the user interacts with the GUI primarily
ments on devices with touch-screen displays.

through finger contacts and gestures on the touch-sensitive

BACKGROUND

so display. In some embodiments, the functions may include
telephoning, video conferencing, e-mailing, instant messaging, blogging, digital photographing, digital videoing, web
browsing, digital music playing, and/or digital video playing.
Instructions for performing these functions may be included
ss in a computer program product configured for execution by
one or more processors.

As portable electronic devices become more compact, and
the number of functions performed by a given device
increases, it has become a significant challenge to design a
user interface that allows users to easily interact with a multifunction device. This challenge is particularlysignificant for
Inaccordancewith some embodiments, a computer-implehandheldportable devices, which have much smaller screens
mented method for use in conjunction with a device with a
than desktop or laptop computers. This situation is unfortutouch screen display is disclosed. In the method, a movement
nate because the user interface is the gateway through which
users receive not only content but also responses to user 60 ofan object on ornearthe touch screen display is detected. In
response to detecting the - .
t, an electronic document
actions or behaviors, including user attempts to access a
displayed on the touch screen display is translated in a first
device's features, tools, and functions. Some portable communication devices (e.g., mobile telephones, sometimes

direction. If an edge of the electronic document is reached

while translatingthe electronic document in the first direction
called mobile phones, cell phones, cellular telephones, and
the like) have resorted to adding more pushbuttons, increas- 65 while the object is still detected on or near the touch screen
ing the density of push buttons, overloading the functions of

display, an areabeyondthe edgeofthe documentis displayed.

pushbuttons, or using complex menu systems to allow a user

After the object is no longer detected on or near the touch

Copy provided by USPTO from the PIRS Image Database on 04/25/20t1

APLNDC00022508

3

4

US 7,469,381 B2
screen display, the document is translated in a second direcarea beyondthe edge ofthe document is no longer displayed,
tion until the area beyond the edge of the document is no
after the object is no longer detected on or near the touch
longer displayed.
screen display.
In accordance with some embodiments, a graphical user
Inaccordancewithsomeembodiments,acomputer-impleinterface on a device with a touch screen display is disclosed, 5 mented method for use in conjunction with a device with a
comprising a portion of an electronic document displayed on
touch screen display is disclosed. Inthe method, a movement
the touch screen display and an area beyond an edge of the
ofan object on or near the touch screen display is detected. In
document. In the graphical user interface, in response to
responsetodetectingthe ,-ent,alistofitemsdisplayed
detecting a movement ofan object on or nearthe touch screen
on the touch screen display is scrolled in a first direction. If a
display, the electronic document is translated in a first direc- 10 terminus of the list is reached while scrolling the list in the
tion. If the edge of the electronic document is reached while
first direction while the object is still detected on or near the
translating the electronic document in the first directionwhile
touch screendisplay, an area beyondthe terminus ofthe list is
the object is still detected on or near the touch screen display,
displayed.Afterthe object is no longer detected on or nearthe
the area beyond the edge of the document is displayed.After
touch screen display, the list is scrolled in a second direction
the object is no longer detected on or near the touch screen 15 opposite the first direction until the area beyond the t .--s
display, the document is translated in a second direction until
ofthe list is no longer displayed.
the area beyond the edge of the document is no longer disIn accordance with some embodiments, a graphical user
played.
interface on a device with a touch screen display is disclosed,
In accordance with some embodiments, a device is discomprising a portionof a list of items displayed on the touch
closed, comprising a touch screen display, one or more pro- 20 screen display and an area beyond a terminus of the list. In
cessors, memory, and one or more programs. The one ormore
response to detecting a movement of an object on or near the
programs are stored in the memory and configured to be
touch screen display, the list is scrolled in a first direction. If
executed by the one or more processors. The one or more
theterminusofthelistisreachedwhilescrollingthelistinthe
programs include instructions for detecting a movement ofan
first direction while the object is still detected on or near the
object on or near the touch screen display and instructions for 25 touch screen display, the area beyond the terminus of the list
translating an electronic document displayed on the touch
is displayed. After the object is no longer detected on or near
screen display in a first direction, in response to detecting the
the touch screen display, the list is scrolled in a second direcmovement. The one or more programs also include instruction opposite the first direction until the area beyond the
tions for displaying an area beyond an edge ofthe electronic
terminus of the list is no longer displayed.
document if the edge of the electronic document is reached 30 In accordance with some embodiments, a device is diswhile translating the electronic documentinthefirstdirection
closed, comprising a touch screen display, one or more prowhile the object is still detected on or near the touch screen
cessors, memory, and one ormore programs. The one ormore
display. The one or more programs further include instrucprograms are stored in the memory and configured to be
tions for translating the document in a second direction until
executed by the one or more processors. The one or more
the area beyond the edge of the document is no longer dis- as programs include instructions for detecting a movement ofan
played, after the object is no longer detected on or near the
object on or nearthe touch screen display and instructions for
touch screen display.
scrolling a list ofitems displayed on the touch screen display
In accordance with some embodiments, a computer-proin a first direction in responseto detecting the movement. The
gram product is disclosed, comprising a computer readable
one ormore programs also include instructions for displaying
storage medium and a computer program mechanism embed- 4o an area beyond a terminus ofthe list ifthe terminus ofthe list
ded therein. The computer program mechanism comprises
is reached while scrolling the list in the first direction while
instructions, which when executed by a device with a touch
the object is still detected on or near the touch screen display.
screen display, cause the device to detect a movement of an
The one or more programs further include instructions for
object on or near the touch screen display and to translate an
scrolling the list in a second direction opposite the first direcelectronic document displayed on the touch screen display in 45 tion until the area beyond the terminus of the list is no longer
a first direction, in response to detecting the movement. The
displayed, after the object is no longer detected on or near the
instructions also cause the device to displayanarea beyondan
touch screen display.
edge of the electronic document if the edge of the electronic
In accordance with some embodiments, a computer-prodocument is reached while translating the electronic docugram product is disclosed, comprising a computer readable
ment in the first direction while the object is still detected on 50 storage medium and a computerprogram mechanism embedor near the touch screen display. The instructions further
ded therein. The computer program mechanism comprises
cause the device to translate the document in a second direcinstructions, which when executed by a device with a touch
tion until the area beyond the edge of the document is no
screen display, cause the device to detect a movement of an
longer displayed, after the object is no longer detected on or
object onornearthe touch screendisplay and to scroll a list of
near the touch screen display.
55 items displayedon thetouch screen display in a first direction
In accordance with some embodiments, a device with a
in response to detecting the movement. The instructions also
touch screen display is disclosed. The device comprises
cause the device to display an area beyond a t ...i.._ of the
means for detecting a movement of an object on or near the
list ifthe terminus ofthe list is reached while scrolling the list
touch screen display and means for translating an electronic
inthe first directionwhilethe object is still detected on or near
document displayed on the touch screen display in a first 60 the touch screen display. The instructions further cause the
direction, in response to detecting the movement. The device

device to scroll the list in a second direction opposite the first

also comprises means for displaying an area beyond an edge
direction until the area beyond the terminus of the list is no
ofthe electronic document if the edge of the electronic doculonger displayed, after the object is no longer detected on or
ment is reached while translating the electronic document in
near the touch screen display.
the first direction while the object is still detected on or near 65
In accordance with some embodiments, a device with a
the touch screen display. The device further comprises means
touch screen display is disclosed. The device comprises
for translating the document in a second direction until the
means for detecting a movement of an object on or near the

A-.... --..:J.A ku iIODTT4 ha... Ihm DIDO innamen naimiamen an AA

E

Al

APLNDC00022509

5

6

US 7,469,381 B2
touch screen display and means for scrolling a list of items
Inaccordancewithsomeembodiments, acomputer-impledisplayed on the touch screen display in a first direction in
mented method of displaying an electronic document having
response to detecting the movement. The device also coma document length and a document width, for use at a device
prises means for displaying an area beyond a terminus ofthe
withatouch screendisplay, includes displaymg the electronic
list ifthe terminus ofthe list is reached while scrolling the list s document ata firstmagnificationanddetecting a gesture on or
in the first direction while the object is still detected on ornear
nearthe touch screen display corresponding to a command to
the touch screen display. The device furthercomprises means
zoom outbya user-specifiedamount. Inresponseto detecting
for scrolling the list in a second direction opposite the first
the gesture, the electronic document is displayed at a magmdirection until the area beyond the terminus of the list is no
fication less than the first magnification. If the document
longer displayed, after the object is no longer detected on or lo length or document width is entirely displayed while the
near the touch screen display.

gesture is still detected onornearthe touch screen display, the

electronic document is displayed at a magnification wherein
In accordancewith some embodiments, a computer-impleareas beyond opposite edges of the electronic document are
mented method for use at a device with a touch screen display
displayed, and upon detecting termination ofthe gesture, the
includes detecting a multifinger twisting gesture on or near 15 electronic document is displayedat a magnification wherein
the touch screen display. The multifinger twistinggesture has
the areas beyond opposite edges of the electronic document
a corresponding degree of rotation. If the corresponding
a no longer displayed.
degree of rotation exceeds a predefined degree of rotation, a
In accordance with some embodiments, a graphical user
90° screen rotation command is executed. If the correspondaterface on a device with a touch screen display includes an
ing degree of rotation is less than the predefined degree of 2e electronic document having a document length and a docurotation, a screen rotation command with an acute angle of
ment width, to be displayed on the touch screen display at
rotation is executed and, upon ceasing to detect the multifinmultiple magnifications including a first magnification, and
ger twisting gesture, a screen rotation commandwith anangle
areas beyond opposite edges of the electronic document. In
of rotation opposite to the acute angle is executed.
response to detecting a gesture on or near the touch screen
In accordancewith some embodiments, a device includes a 25 display corresponding to a command to zoom out by a usertouch screen display, one or more processors, memory, and
specified amount, wherein the gesture is detected while disone or more programs. The one or more programs are stored
playingthe electronic documentatthe firstmagnification, the
in the memory and configured to be executed by the one or
electronic document is displayed at a magnification less than
more processors. The one or more programs include: instruc¯
the first magnification. If the document length or document
tions for detecting a multifinger twisting gesture on or near 3o width is entirely displayed while the gesture is still detected
the touch screen display, wherein the multifinger twisting
onornearthetouchscreendisplay,theelectroniedocumentis
gesture has a corresponding degree of rotation; instruction
displayed at a magnification wherein the areas beyond oppofor executing a 90° screen rotation command, if the corre¯
site edges ofthe electronic document are displayed, andupon
sponding degree of rotation exceeds a predefmed degree of
detecting termination ofthe gesture, the electronic document
rotation;andinstructionsforexecutingascreenrotationcom¯ 35 is displayed at a magnification wherein the areas beyond
mand with an acute angle of rotation and for executing, upon
opposite edges of the electronic document are no longer disceasing to detect the multifmger twisting gesture, a screen
played.
rotation command with an angle of rotation opposite to the
In accordancewith some embodiments, a device includes a
acute angle, if the corresponding degree of rotation is less
touch screen display, one or more processors, memory, and
than the predefined degree of rotation'
40 one or more programs. The one or more programs are stored
In accordance with some embodiments, a computer-proin the memory and configured to be executed by the one or
more processors. The one or more programs include: instrucgram product includes a computer readable storage medium
tions for displaying an electronic document at a first magniand a computer program mechanism embedded therein. The
fication; instructions for detecting a gesture on or near the
computer program mechanism includes instructions, which
when executed by a device with a touch screen display, cause 45 touch screen display corresponding to a command to zoom
the device to: detect a multifinger twisting gesture on or near
outbya user-specified---t; instructions for displayingthe
electronic document at a magnification less than the first
the touch screen display, wherein the multifmger twisting
magnification, in response to detecting the gesture; instrucgesture has a corresponding degree ofrotation; execute a 90°
tions for displaying the electronic document at a magnificascreen rotation command, if the corresponding degree of
rotation exceeds a predefined degree ofrotation; and execute so tion wherein areas beyond opposite edges of the electronic
document are displayed, ifa document length or a document
a screen rotation command with an acute angle of rotation
width is entirely displayed while the gesture is still detected
and, upon ceasing to detect the multifinger twisting gesture,
execute a screen rotation command with an angle of rotation
on or near the touch screen display; and instructions for
displaying the electronic document at a magnification
opposite to the acute angle, if the corresponding degree of
rotation is less than the predefined degree of rotation.
ss wherein the areas beyond opposite edges of the electronic
document are no longer displayed, upon detecting terminaIn accordance with some embodiments, a device with a
tion of the gesture.
touch screen display includes: means for detecting a multiIn accordance with some embodiments, a computer-profinger twisting gesture on or near the touch screen display,
gram product includes a computer readable storage medium
wherein the multifinger twisting gesture has a corresponding
degree of rotation; means for executing a 90° screen rotation 60 and a computer program mechanism embedded therein. The
computer program mechanism includes instructions, which
command, if the corresponding degree of rotation exceeds a
when executed by a device with a touch screen display, cause
predefined degree of rotation; and means for executing a
the device to: display an electronic document at a first magscreen rotation command with an acute angle ofrotation and,
nification; detect a gesture on or near the touch screen display
upon ceasing to detect the multifinger twisting gesture, for
executing a screen rotation command with an angle of rota- 65 corresponding to a command to zoom out by a user-specified
amount; display the electronic document at a magnification
tion opposite to the acute angle, if the corresponding degree
less than the first magnification, in response to detecting the
of rotation is less than the predefined degree ofrotation.

. --..--a., a sen·rn on- ska ome. Imana nntahnan on 04/25/20Ti

APLNDC00022510

US 7,469,381 B2
7

8

gesture; display the electronic document at a magnification

In accordance with some embodiments, a computer-program product includes a computer readable storage medium
and a computer program mechanism embedded therein. The
computer program mechanism includes instructions, which
when executed by a device with a touch screen display, cause
the device to: display at least a first portion of an electronic
document at a first magnification; detect a gesture on or near
the touch screen display corresponding to a command to
zoom in by a user-specified amount; display decreasing portions ofthe electronic documentat increasing magnifications,
in response to detecting the gesture; and display a respective
portion of the electronic document at a predefined magnification if, upon detecting termination of the gesture, the magnification exceeds a predefmed magnification.
In accordance with some embodiments, a device with a
touch screen display includes means for displaying at least a
firstportionofanelectronicdocumentatafirstmagnification;
means for detecting a gesture on or near the touch screen
display corresponding to a command to zoom in by a userspecified amount; means for displaying decreasing portions
of the electronic document at increasing magnifications, in
response to detecting the gesture; and means for displaying a
respective portion ofthe electronic document at a predefmed
magnification if, upon detecting termination of the gesture,
the magnification exceeds the predefmed magnification.
The disclosed embodiments provide for easy and intuitive
scrolling oflists and translating ofelectronic documents on a
device with a touch screen display, and for easy and intuitive
rotationand scaling ofelectronic documents on a device with
a touch screen display.

wherein areas beyond opposite edges ofthe electonic document are displayed, ifa document length or a document width
is entirely displayed while the gesture is still detected on or
near the touch screen display; and display the electronic
document at a magnification wherein the areas beyond opposite edges ofthe electronic document are no longer displayed,
upon detecting termination of the gesture.
In accordance with some embodiments, a device with a
touch screen display includes: means for displaying an electronic document at a first magailication; means for detecting
a gesture on ornear the touch screen display correspondingto
a command to zoom out by a user-specified amount; means
for displaying the electronic document at a magnification less
than the first magnification, in response to detecting the gesture; means for displaying the electronic document at a magnification wherein areas beyond opposite edges of the electronic document are displayed, if a document length or a
document width is entirely displayed while the gesture is still
detected on or near the touch screen display; and means for
displaying the electronic document at a magnification
wherein the areas beyond opposite edges of the electronic
document are no longer displayed, upon detecting termination of the gesture.
In accordancewith some embodiments, a computer-implemented method ofdisplaying an electronic document, foruse
at a device with a touch screen display, includes displaying at
least a first portion of the electronic document at a first magnification and detecting a gesture on or near the touch screen
display corresponding to a command to mom in by a user-

5

10

15

2

specified amount. In response to detecting the gesture,

decreasing portions ofthe electronic document are displayed
at increasing magnifications. Upon detecting termination of
the gesture, ifthe magnification exceeds a predefined magnification, a respective portion of the electronic document is
displayed at the predefined magnification.
In accordance with some embodiments, a graphical user
interface on a device with a touch screen display includes
decreasing portions of an electronic document, to be displayed on the touch screen display at increasing magnifications. The decreasing portions of the electonic document
include a first portion. In response to detecting a gesture on or
near the touch screen display corresponding to a command to
zoom in by a user-specified amount, wherein the gesture is
detected while displaying at least the first portion of an electronic document at a first magnification, the decreasing portions ofthe electronic document are displayed at the 122-ing magnifications. Upon detecting termination of the
gesture, ifthe magnification exceeds a predefined magnification, a respective portion of the electronic document is displayed at the predefined magnification.
In accordancewith some embodiments, a device includes a
touch screen display, one or more processors, memory, and
one or moæ programs. The one or more programs are stored
in the memory and configured to be executed by the one or
more processors. The one or more programs include instructions for displaying at least a first portion of an electronic

-- DESCRIPTION OF THE DRAWINGS

40

45

so

55

For a better understanding of the aforementioned embodiments of the invention as well as additional embodiments
thereof, reference should be made to the Description of
Embodiments below, in conjunctionwiththe following drawings in which like reference numerals refer to corresponding
parts throughout the figures.
FIG.1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with
some embodiments.
FIG. 2 illustrates a portable multifunction device having a
touch screen in accordance with some embodiments.
FIG. 3 illustrates an exemplaryuser interface for unlocking
aportableelectronic device in accordancewith some embodiments.
FIG. 4 illustrates an exemplaryuser interface for a menu of
applications ona portablemultifunction device in accordance
with some embodiments.
FIG. 5 is a flow diagram illustrating a method of scrolling
through a list in accordance with some embodiments.
FIGS. 6A-6D illustrate an exemplary user interface for
managing an inbox in accordance with some embodiments.
FIG.7is a flow diagram illustrating a method oftranslating
an electronic document in accordance with some embodi-

document at a first magnification; instructions for detecting a

ments.

cation if, upon detecting termination ofthe gesture, the mag-

ment at multiple magnifications in accordance with some
embodiments.

gesture on or near the touch screen display corresponding to 6o FIGS. 8A-8D illustrate an exemplary user interface for a
browser in accordance with some embodiments.
a command to zoom in by a user-specified amount; instrucFIG.9 is a flow diagram illustrating a process ofdisplaying
tions for displaying decreasing portions of the electronic
an electronic document at multiple magnifications in accordocument at ° --ing magnifications, inresponseto detectdance with some embodiments.
ing the gesture; and instructions for displaying a respective
portion of the electronic document at a predefined magnifi- 65 FIGS.10A-10Cillustratethedisplayofanelectronicdocunification exceeds the predefined magnification.

Go y rovided by usPTO trom the PlHS Ima e Database on 04/25/2011

APLNDC00022511

US 7,469,381 B2
9

10

FIG. 11 is a flow diagram illustrating a process ofdisplaydevice, such as thetouch screen. One ormore functions ofthe
ing an electronic document at multiple magnifications in
touch screen as well as corresponding information displayed
accordance with some embodiments.
on the device may be adjusted and/or varied from one appliFIGS.12A-12C illustrate the displayofan electronic docucation to the next and/or within a respective application. In
ment at multiple magnifications in accordance with some 5 this way, a ....-- physical architecture (such as the touch
embodiments.
screen) of the device may support the variety of applications
FIGS. 13A-13C illustrate the display ofan electronic docuwith user interfaces that are intuitive and transparent.
ment at multiple magnifications in accordance with some
The user interfaces may include one or more soft keyboard
embodiments.
embodiments. The soft keyboard embodiments may include
FIG.14 is a flow diagram illustrating a process ofexecuting 10 standard (QWERTY) and/or non-standard configurations of
a screen rotation command in acconlance with some embodisymbols onthe displayed icons ofthe keyboard, such as those
ments.
described in U.S. patent applications Ser. No. 11/459,606,
FIGS. 15A-15E illustrate rotating the display of an elec"Keyboards For Portable Electronic Devices," filed Jul. 24,
tronic document or other digital object in accordance with
2006, and Ser. No. 11/459,615, "Touch Screen Keyboards For
some embodiments.
15 Portable Electronic Devices," filed Jul. 24, 2006, the contents
FIGS. 16A-16F illustrate an exemplary screen rotation
ofwhich are hereby incorporated by reference herein in their
gesture in accordance with some embodiments.
entirety. The keyboard embodiments may include a reduced
FIG. 17 is a block diagram illustrating a device with a
number of icons (or soft keys) relative to the number ofkeys
touch ...._ display in accordance with some embodiments.
in existing physical keyboards, such as that for a typewriter.
20 This may make it easier for users to select one or more icons
DES -- ONOFEMBODu -ss
in the keyboard, and thus, one or more corresponding symbols. The keyboard embodiments may be adaptive. For
Reference will now be made in detail to embodiments,
example, displayed icons may be modified in accordance
examples ofwhich are illustrated in the accompanying drawings. In the following detailed description, wm om specific 25 with user actions, such as selecting one or more icons and/or
one or more corresponding symbols. One or more applicadetails are set forth in orderto provide a thoroughunderstandtions on the portable device may utilize e-un and/or difing of the present invention. However, it will be apparent to
ferent keyboard embodiments. Thus, the keyboard embodione of ordinary skill in the art that the present invention may
mentused may be tailoredto at least some ofthe applications.
be practiced without these specific details. In other instances,
well-known methods, procedures, components, circuits, and so In some embodiments, one or more keyboard embodiments
may be tailored to a respective user. For example, based on a
networks have not been described in detail so as not to unnecword usage history (lexicography, slang, individual usage) of
essarily obscure aspects of the embodiments.
the respective user. Some ofthe keyboard embodiments may
Embodiments of a portable multifunction device, user
be adjusted to reduce a probability of a user error when
interfaces for such devices, and associated processes for
selecting one or more icons, and thus one or
using such devices are described. In some embodiments, the as when using the soft keyboard embodiments. more symbols,
device is a portable c -acations device such as a mobile
Attention is now directed towards embodiments of the
telephone that also contains other functions, such as PDA
device. FIG. 1is a block diagram illustrating a portable muland/or music player functions.
tifunction device 100 with a touch-sensitive display 112 in
The user interface may include a physical click wheel in
addition to a touch screen or a virtual click wheel displayedon 4o accordancewith some embodiments. The touch-sensitivedisplay 112 is sometimes called a "touch screen" for convethe touch -. . A click wheel is a user-interface device that
The device 100 may include a memory 102 (which
may provide navigation commands based on an angular dismay include one or more computer readable storage mediplacement ofthe wheel or a point ofcontact with the wheel by
ums), a memory controller 122, one ormore processing units
a userofthe device. A click wheel may also be usedto provide
a user command corresponding to selection of one or more 4s (CPU's) 120, a peripherals interface 118, RF circuitry 108,
audio circuitry 110, a speaker 111, a microphone 113, an
items, for example, when the user ofthe device presses down
on at least a portion of the wheel or the center of11le wheel.
input/output (1/O) subsystem 106, other input or control
devices 116, and an external port 124. The device 100 may
Alternatively, breaking contact with a click wheel image on a
melude one or more optical woua 164. These components
touch screen surface may indicate a user command corresponding to selection. For simplicity, in the discussion that so may commumcate over one or more communication buses or
signal lines 103.
follows, a portablemultifunction device that includes a touch
screen is used as an exemplary embodiment. It should be
It should be appreciated that the device 100 is only one
understood, however, that some of the user interfaces and
example ofa portable multifunction device 100, and that the
associated processes may be applied to other devices, such as
device 100 may have more or fewer components than shown,
personal computers and laptop computers, that may include ss may combine two or more components, or a may have a
one or more other physical user-interface devices, such as a
different configuration or arrangement of the components.
physical click wheel, a physical keyboard, a mouse and/or a
The various components shown in FIG. 1 may be implejoystick.
mented in hardware, software or a combination of both hardThe device supports a variety of applications, such as a
ware and software, including one or more signal processing
telephone application, a video conferencing application, an 60 and/or application specific integrated circuits.
e-mail application, an instant messaging application, a blogMemory 102 may include high-speed random access
ging application, a digital --- . application, a digital video
memory and may also include non-volatile memory, such as
application, a web browsing application, a digital
one or more magnetic disk storage devices, flash memory
music player application, and/or a digital video player applidevices, or other non-volatile solid-state memory devices.
CallOR.
65
Access to memory 102 by other components of the device
The various applications that may be executed on the
100, such as the CPU 120 and the peripherals interface 118,
device may use at least one common physical user-interface
may be controlled by the memory controller 122.

APLNDC00022512

US 7,469,381 B2
11

12

dials, slider switches,joysticks, click wheels, and so forth. In
The peripherals interface 118 couples the input and output
some alternate embodiments, input controller(s) 160 may be
peripherals of the device to the CPU 120 and memory 102.
coupled to any (or none) of the following: a keyboard, infraThe one or more processors 120 run or execute various softred port, USB port, and a pointer device such as a -. The
ware programs and/or sets of instructions stored in memory
102 to perform various functions for the device 100 and to 5 one or more buttons (e.g., 208, FIG. 2) may include an
up/down button for volume control of the speaker 111 and/or
process data.
the miemphone 113. The one or more buttons may include a
In some embodiments, the peripherals interface 118, the
push button (e.g., 206, FIG. 2). A quick press of the push
CPU 120, and the memory controller 122 may be implebutton may disengage a lock ofthe touch screen 112 or begin
mented on a single chip, such as a chip 104. In some other
embodiments, they may be implemented on separate chips. to a process that uses gestures onthe touch screen to unlock the
device, as described in U.S. patent application Ser. No.
The RF (radio frequency) circuitry 108....... and sends
11/322,549, "Unlocking a Device by Performing Gestures on
RF signals, also called electromagnetic signals. The RF ciran Unlock Image" filed Dec. 23, 2005, which is hereby
cuitry 108 converts electrical signals tolfrom electromagnetic
incorporatedbyreferenceherein in its entirety.A longerpress
signals and communicates with communications networks
and other communications devices via the electromagnetic 15 of the push button (e.g., 206) may turn power to the device
100 on or off. The user may be able to customize a functionsignals. The RF circuitry 108 may include well-known cirality of one or more of the buttons. The touch screen 112 is
cuitry for performing these functions, including but not limusedto implement virtual or soft buttons and one or more soft
ited to an antenna system, an RF transceiver, one or more
amplifiers, a tuner, one or more oscillators, a digital signal
keyboards.
processor, a CODEC chipset, a subscriber identity module 2o The touch-sensitive display system 112 provides an input
interface and an output interface between the device and a
(SIM) card, memory, and so forth. The RF circuitry 108 may
user. The display controller 156 ....: - and/or sends eleccommunicate with networks, such as the Internet, also
trical signals from/to the display system 112. The display
referred to as the World Wide Web (WWW), an intranet
system 112 displays visual output to the user. The visual
and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metro- 25 output may include graphics, text, icons, video, and any combination thereof (collectively termed "graphics"). In some
politan area network (MAN), and other devices by wireless
embodiments, some or all of the visual output may corre--. ' .tion. The wireless co-acationmayuse any of
spond to user-interface objects, further details of which are
a plurality of co
cations standards, protocols and techdescribed below.
nologies, including but not limited to Global System for
Mobile Communications (GSM), EnhancedData GSM Envi- 30 A touch screen in display system 112 is a touch-sensitive
surfacethatacceptsinputfromtheuserbasedonhapticand/or
ronment (EDGE), wideband code division multiple access
tactile contact. The display system 112 and the display con(W-CDMA), code division multiple access (CDMA), time
troller 156 (along with any associated modules and/or sets of
division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g
instructions in memory 102) detect contact (and any moveand/or IEEE 802.11n), voice over Intemet Protocol (VoIP), 35 ment or breaking of the contact) on the display system 112
and converts the detected contact into interaction with userWi-MAX, a protocol for email, instant messaging, and/or
interface objects (e.g., one or more soft keys, icons, web
Short Message Service (SMS)), or any other suitable compages or images) that are displayed on the touch screen. In an
munication protocol, including co
° - - protocols not
exemplary embodiment, a point of contact between a touch
yet developed as of the filing date of this document.
The audio circuitry 110, the speaker 111, and the micro- 40 screen in the display system 112 and the user corresponds to
a finger of the user.
phone 113 provide an audio interface between a user and the
The touch screen in the display system 112 may use LCD
device 100. The audio circuitry 110 ....I... audio data from
(liquid crystal display) technology, or LPD (light emitting
the peripherals interface 118, converts the audio data to an
polymer display) technology, although other display techelectrical signal, and transmits the electrical signal to the
speaker 111. The speaker 111 converts the electrical signal to 45 nologies may be used in other embodiments. The touch
screen in the display system 112 and the display controller
human-audible sound waves. The audio circuitry 110 also
156 may detect contact and any movement or breaking
receives electrical signals converted by the microphone 113
thereofusing any ofa plurality oftouch sensing technologies
from sound waves. The audio circuitry 110 converts the elecnow known or later developed, including but not limited to
trical signal to audio data and transmits the audio data to the
peripherals interface 118 for processing. Audio data may be so capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other
retrieved from and/or transmitted to memory 102 and/or the
elements for determining one or more points of contact with
RF circuitry 108 by the peripherals interface 118. In some
a touch screen in the display system 112. A touch-sensitive
embodiments, the audio circuitry 110 also includes a headset
display in some embodiments ofthe display system 112 may
jack (not shown). The headset jack provides an interface
between the audio circuitry 110 and --..ble audio input/ 55 be analogous to the multi-touch sensitive tablets described in
the following U.S. Pat. Nos. 6,323,846 (Westerman et al.),
output peripherals, such as output-onlyheadphonesorahead6,570,557 (Westermanet al.), and/or 6,677,932 (Westerman),
set with both output (e.g., a headphone for one or both ears)
and/or U.S. Patent Publication "^^*/^^15024A1, each of
and input (e.g., a microphone).
which is hereby incorporated by reference herein in their
The I/O subsystem106 couples input/outputperipheralson
the device 100, such as the display system 112 and other 60 entirety. However, a touch screen in the display system 112
input/control devices 116, to the peripherals interface 118.

displays visual output from the portable device 100, whereas

touch sensitive tablets do notprovidevisual output. The touch
The I/O subsystem 106 may include a display controller 156
screen in the display system 112 may have a resolution in
and one or more input controllers 160 for other input or
excess of 100 dpi. In an exemplary embodiment, the touch
control devices. The one or more input controllers 160
receivelsend electrical signals from/to other input or control 65 screeninthe display systemhasaresolutionofapproximately
168 dpi. The user may make contact with the touch screen in
devices 116. The other input/control devices 116 may include
the display system 112 using any suitable object or appendphysical buttons (e.g., push buttons, rocker buttons, etc.),

Copy provided by USPTO from the PIRS Imaae Database on 04/25/2011

APLNDC00022513

US 7,469,381 B2
13

14

(e.g., battery, alternating current (AC)), a recharging
system, apowerfailure detection circuit, a powerconverteror
inverter, a power status indicator (e.g., a light-emitting diode
(LED)) and any other components associated with the gen5 eration, management and distribution of power in portable
devices.
The device 100 may also include one or more optical
sensors 164. FIG. 1 shows an optical sensor coupled to an
optical sensor controller 158 in I/O subsystem 106. The opti10 cal sensor 164 may include charge-coupled device (CCD) or
complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensor 164 ....I... light from the
environment, projected through one or more lens, and converts the light to data representing an image. In conjunction
15 with an imaging module 143, the optical sensor 164 may
capture still images or video. In some embodiments, an optical sensor is located on the back of the device 100, opposite
the touch screen display 112 on the front ofthe device, so that
the touch screen display may be used as a viewfinder for
20 either still and/or video image acquisition. In some embodiments, anoptical sensoris located onthe front ofthe device so
that the user's image may be obtained for videoconferencing
while the user views the other video conference participants
on the touch screen display. In some embodiments, the posi25 tion ofthe optical sensor 164 can be changed by the user (e.g.,
by rotating the lens and the sensor in the device housing) so
that a single optical sensor 164 may be used along with the
touch screen display for both video conferencing and still
and/or video image acquisition.
30 The device 100 may also include one or more proximity
sensors 166. FIG. 1 shows a proximity sensor 166 coupled to
the peripherals interface 118. Alternately, the proximity senIn some embodiments, in addition to the touch screen, the
sor 166 may be coupled to an input controller 160 in the I/O
device 100 may include a touchpad (not shown) for activating
subsystem 106. The proximity sensor 166 may perform as
or deactivating particular functions. In some embodiments,
the touchpad is a touch-sensitive area of the device that, 35 described in U.S. patent application Ser. Nos. 11/241,839,
"Proximity Detector In Handheld Device," filed Sep. 30,
unlike the touch screen, does not display visual output. The
2005, and 11/240,788, "Proximity Detector In Handheld
touchpad may be a touch-sensitive surface that is separate
Device;' filed Sep. 30, 2005, which are hereby incorporated
from the touch screen in the display system 112 or an extenby reference herein in their entirety. In some embodiments,
sion of the touch-sensitive surface formed by the touch
4o the proximity sensor turns off and disables the touch screen
screen.
112 when the multifunction device is placed near the user's
In some embodiments, the device 100 may include a physiear (e.g., when the user is making a phone call). In some
cal or virtual click wheel as an input control device 116. A
embodiments, theproximitysensorkeeps the screen ofIwhen
user may navigate among and interact with one or more
the device is in the user's pocket, purse, or other dark area to
graphical objects (henceforth referred to as icons) displayed
in the display system 112 by rotating the click wheel or by 45 prevent ----.sary battery drainage when the device is a
locked state.
moving a point ofcontact withthe click wheel (e.g., wherethe
In some embodiments, the software components stored in
amountofmovement ofthepoint ofcontact is measuredby its
memory 102 may include an operating system 126, a comangular displacement with respect to a center point of the
munication module (or set of instructions) 128, a contact/
click wheel). The click wheel may also be used to select one
or more of the displayed icons. For example, the user may so motionmodule (orset ofinstructions) 130, a graphics module
(or set of instructions) 132, a text input module (or set of
press down on at least a portion of the click wheel or an
instructions) 134, a Global Positioning System (GPS) module
associatedbutton. Usercommands and navigationcommands
(or set ofinstructions) 135, and applications (or set ofinstrucprovided by the user via the click wheel may be processed by
tions) 136.
an input controller 160 as well as one or more ofthe modules
and/or sets of instructions in memory 102. For a virtual click 55 The operating system 126 (e.g., Darwin, RTXC, LINUX,
UNIX, OSX, -- WS, or an embedded operating system
wheel, the click wheel and click wheel controllermay be part
such as VxWorks) includes various software components
of the display system 112 and the display controller 156,
and/or drivers for controlling and managing general system
respectively. For a virtual click wheel, the click wheel may be
tasks (e.g., memory management, storage device control,
either an opaque or semitransparent object that appears and
cation
disappears on the touch screen display in response to user 60 power management, etc.) and facilitates co
between various hardware and software components.
interaction with the device. In some embodiments, a virtual
The
,tion module 128 facilitates
' click wheel is displayed on the touch screen of a portable
tion with other devices over one or more external ports 124
multifunction device and operated by user contact with the
and also includes -i- software components for handling
touch device 100 also includes a power system162 for pow- 65 data received by the RF circuitry 108 and/or the external port
.......
The
124. The external port 124 (e.g., Universal Serial Bus (USB),
ering the various components. The power system 162 may
FIREWIRE, etc.) is adapted for coupling directly to other
include a power management system, one or more power

age, such as a stylus, a finger, and so forth. In some embodiments, the user interface is designed to work primarily with
fmger-based contacts and gestures, which are much less precise than stylus-based input due to the larger area ofcontact of
a finger onthe touch . . In some embodiments, the device
translates the rough finger-based input into a precise pointer/
cursor position or command for performing the actions
desired by the user.
A touch-sensitive display in some embodiments of the
display system 112 may be as described in the following
applications: (1) U.S. patent application Ser. No. 11/381,313,
"Multipoint Touch Surface Controller," filed on May 2, 2006;
(2) U.S. patent application Ser. No. 10/840,862, "Multipoint
Touchscreen," filed on May 6, 2004; (3) U.S. patent application Ser. No. 10/903,964, "Gestures For Touch Sensitive
Input Devices," filed on Jul. 30, 2004; (4) U.S. patent application Ser. No. 11/048,264, "Gestures For Touch Sensitive
Input Devices," filed on Jan. 31, 2005; (5) U.S. patent application Ser. No. 11/038,590, "Mad--Based Graphical User
Interfaces For Touch Sensitive Input Devices;' filed on Jan.
18, 2005; (6) U.S. patent application Ser. No. 11/228,758,
"Virtual Input Device Placement On A Touch Screen User
Interface," filed on Sep. 16, 2005; (7) U.S. patent application
Ser. No. 11/228,700, "Operation Of A Computer With A
Touch Screen Interface," filed on Sep. 16, 2005; (8) U.S.
patent application Ser. No. 11/228,737, "Activating Virtual
Keys OfA Touch-ScreenVirtual Keyboard," filed on Sep. 16,
2005; and (9) U.S. patent application Ser. No. 11/367,749,
"Multi-Functional Hand-Held Device;' filed on Mar. 3, 2006.
All ofthese applications are incorporated by reference herein
in their entirety.

APLNDC00022514

US 7,469,381 B2

15
devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a
multi-pin (e.g., 30-pin) connector that is the same as, or
similar to and/or compatible with the 30-pin connector used
on iPod (trademark ofApple Computer, Inc.) devices.
The contact/motion module 130 may detect contact with
the touch screen in the display system 112 (in conjunction
with the display controller 156) and other touch sensitive
devices (e.g., a touchpad or physical click wheel). The contact/motion module 130 includes various software components for performing -i- operations related to detection
ofcontact, such as determining ifcontact has occurred, determining if there is movement of the contact and tracking the
movement across the touch screen in the display system 112,
and determining if the contact has been broken (i.e., if the
contact has ceased). Determining movement of the point of
contact may include determining speed (magnitude), velocity
(magnitude and direction), and/or an acceleration (a change
in magnitude and/or direction) of the point ofcontact. These
operations may be applied to single contacts (e.g., one finger
contacts) or to multiple simultaneous contacts (e.g., "multitouch"Imultiple finger contacts). In some embodiments, the
contact/motion module 130 and the display controller 156
also detects contact on a touchpad. In some embodiments, the
contact/motion module 130 detects movement ofone ormore
objects on or near the touch screen and/or the touchpad. In
some embodiments, the contact/motion module 130 and the
controller 160 detects contact on a click wheel 116.
The graphics module 132 includes
s,-known software
components for rendering and displaying graphics on the
display system 112, including components for changing the
intensity of graphics that are displayed. As used herein, the

16

alarm clock widget 149-4, dictionary widget 149-5, and
other widgets obtained by the user, as well as usercreated widgets 149-6;

widget creator module 150 for making user-created wid5
gets 149-6; and/or search module 151.
Examples of other applications 136 that may be stored in
memory 102 include memo pad and other word processing
applications, JAVA-enabled applications, encryption, digital
rights management, voice recognition, and voice replication.
10 In conjunction with display system 112, display controller
156, contact module130, graphics module132, andtext input
module 134, the contacts module137may be used to manage
an address book or contact list, including: adding name(s) to
the address book; deleting name(s) from the address book;
15 associating telephone number(s), e-mail address(es), physical address(es) or other information with a name; associating
an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and/
orfacilitate communications by telephone 138, video confer20 ence 139, e-mail 140, or IM 141; and so forth.
In conjunction with RF circuitry 108, audio circuitry 110,
speaker 111, microphone 113, display system 112, display
controller 156, contact module 130, graphics module 132,
and text input module 134, the telephone module 138 may be
25 used to enter a sequence of characters corresponding to a
telephone number, access one or more telephone numbers in
the address book 137, modify a telephone number that has
been entered, dial a respective telephone number, conduct a
tion and disconnect or hang up when the conversa30 tion is completed. As noted above, the wireless communication may use any ofa plurality of----cations standards,
protocols and technologies.

term "graphics" includes any object that can be displayed to
In conjunction with RF circuitry 108, audio circuitry 110,
a user, including without limitation text, web pages, icons
speaker 111, microphone 113, display system 112, display
(such as user-interface objects including soft keys), digital 2 controller 156, optical sensor 164, optical sensor controller
images, videos, animations and the like.
158, contact module 130, graphics module 132, text input
The text input module 134, which may be a component of
module 134, contact list 137, and telephone module 138, the
graphics module 132, provides soft keyboards for entering
videoconferencing module 139 may be used to initiate, context invarious applications (e.g., contacts 137, e-mail140, IM
duct, and terminate a video conference between a user and
141, blogging 142, browser 147, and any other application * one or more other participants.
that needs text input).
In conjunction with RF circuitry 108, display system 112,
The GPS module 135 determines the location ofthe device
display controller 156, contact module 130, graphics module
and provides this information for use in ..;- applications
132, and text input module 134, the e-mail client module 140
(e.g., to telephone 138 for use in location-based dialing, to 45 may be used to create, send, receive, and manage e-mail. In
- 143 and/or blogger 142 as picture/video metadata,
conjunctionwith image management module 144, the e-mail
and to applications that provide location-based services such
module140 makes it veryeasyto create and send e-mails with
as weather widgets, local yellow page widgets, and map/
still or video images taken with
module 143.
navigation widgets).
In conjunction with RF circuitry 108, display system 112,
The applications 136 may include the following modules
display controller 156, contact module 130, graphics module
(or sets of instructions), or a subset or superset thereof
132, andtext inputmodule 134, theinstant messaging module
a contacts module 137 (sometimes called an address book
141 may be used to enter a sequence of characters correor contact list);
sponding to an instant message, to modify previously entered
characters, to transmit a respective instant message (for
a telephone module 138;
a video conferencing module 139;
ss example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol), to receive instant
an e-mail client module 140;
messages and to view received instant messages. In some
an instant messaging (IM) module 141;
embodiments, transmitted and/or received instant messages
a blogging module 142;
may include graphics, photos, audio files, video files and/or
a
module 143 for still and/or video images;
other attachments as are supported in a MMS and/or an
an image management module 144;
6° Enhanced Messaging Service (EMS).
a video player module 145;
In conjunction with RF circuitry 108, display system 112,
a music player module 146;
display controller 156, contact module 130, graphics module
a browser module 147;
132, text input module 134, image management module 144,
a calendar module 148;
ss and browsing module 147, the blogging module 142 may be
used to send text, still images, video, and/orother graphics to
widget modules 149, which may include weather widget
149-1, stocks widget 149-2, calculator widget 149-3,
a blog (e.g., the user's blog).

APLNDC00022515

US 7,469,381 B2
17

18

Each ofthe above identified modules and applications corIn conjunction with display system 112, display controller
respond to a set of instructions for performing one or more
156, optical sensor(s) 164, optical sensor controller158, confunctions described above. These modules (i.e., sets of
tact module 130, graphics module 132, and image manageinstructions) need not be implemented as separate software
ment module 144, the ._... module 143 may be used to
capture still images or video (including a video stream) and 5 programs, procedures ormodules, andthus various subsets of
these modules may be combined or otherwise re-arranged in
store them into memory 102, modify characteristics of a still
various embodiments. In some embodiments, memory 102
image or video, or delete a still image or video from memory
may store a subset ofthe modules and data structures identi102.
fled above. Furthermore, memory 102 may store additional
In conjunction with display system 112, display controller
156, contact module 130, graphics module 132, text input 10 modules and data structures not described above.
In some embodiments, the device 100 is a device where
module 134, and camera module 143, the image management
operation of a predefined set of functions on the device is
module 144 may be used to arrange, modify or otherwise
performed exclusively through a touch screen in the display
manipulate, label, delete, present (e.g., in a digital slide show
system112 and/ora touchpad. By using a touch screen and/or
or album), and store still and/or video images.
In conjunctionwith display system 112, display controller 15 a touchpad as the primary input/control device for operation
of the device 100, the number of physical input/control
156, contact module 130, graphics module 132, audio cirdevices (such as push buttons, dials, and the like) on the
cuitry 110, and speaker 111, the video playermodule145 may
device 100 may be reduced.
be usedto display, present or otherwise play backvideos (e.g.,
The predefmed set of functions that may be performed
on the touch screen or on an external, connected display via
20 exclusively through a touch screen and/or a touchpad
external port 124).
includes navigationbetweenuser interfaces. In some embodiIn conjunction with display system 112, display system
ments, the touchpad, whentouched by the user, navigates the
controller 156, contact module 130, graphics module 132,
device 100 to a main, home, or root menu from any user
audio circuitry 110, speaker 111, RF circuitry 108, and
interface that may be displayed on the device 100. In such
browsermodule 147, the music playermodule 146 allows the
user to download and play back recorded music and other 25 embodiments, the touchpad may be referred to as a "menu
button."In some other embodiments, the menu button may be
sound files stored in one or more file formats, such as MP3 or
a physical push button or other physical input/control device
AAC files. In some embodiments, the device100 may include
the functionality of an MP3 player, such as an iPod (tradeinstead of a touchpad.
mark ofApple Computer, Inc.).
FIG. 2 illustrates a portable multifunction device 100 havIn conjunction with RF circuitry 108, display system 112, 30 ing a touch screen 112 in accordance with some embodidisplay system controller 156, contact module 130, graphics
ments. Thetouch screenmay display one or more graphics. In
module 132, and text input module 134, the browser module
this embodiment, as well as others described below, a user
147 may be used to browse the Internet, including searching,
may select one or more ofthe graphics by making contact or
linking to, receiving, and displaying web pages or portions
touching the graphics, for example, with one or more fingers
thereof, as well as attachments and other files linked to web 35 202 (not drawn to scale in the figure). In some embodiments,
pages.
selectionofoneormoregraphicsoccurswhentheuserbreaks
In conjunction with RF circuitry 108, display system 112,
contact with the one ormore graphics. In some embodiments,
display system controller 156, contact module 130, graphics
the contact may include a gesture, such as one or more taps,
module 132, text input module 134, e-mail module 140, and
one or more swipes (from left to right, right to left, upward
browsermodule 147, the calendar module 148 may be usedto 40 and/or downward) and/or a rolling of a fmger (from right to
create, display, modify, and store calendars and data associleft, left to right, upward and/or downward) that has made
ated with calendars (e.g., calendar entries, to do lists, etc.).
contact with the device 100. In some embodiments, inadvertIn conjunction with RF circuitry 108, display system 112,
ent contact with a graphic may not select the graphic. For
display system controller 156, contact module 130, graphics
example, a swipe gesture that sweeps over an application icon
module 132, text input module 134, and browsermodule 147, 45 may not select the corresponding application when the gesthe widget modules 149 are mini-applications that may be
ture corresponding to selection is a tap.
downloaded and used by a user (e.g., weather widget 149-1,
The device 100 may also include one or more physical
stocks widget 149-2, calculator widget 149-3, alarm clock
buttons, such as "home" or menu button 204. As described
widget 149-4, and dictionary widget 149-5) or created by the
previously, the menu button 204 may be used to navigate to
user (e.g., u .
ted widget 149-6). In some embodiments, so any application 136 in a set of applications that may be
a widget includes an HTML (Hypertext Markup Language)
executed on the device 100. Alternatively, in some embodifile, a CSS (Cascading Style Sheets) file, and a JavaScript file.
ments, the menu button is implemented as a soft key in a GUI
In some embodiments, a widget includes anXML (Extensible
in touch screen 112.
Markup Language) file and a JavaScript file (e.g.,Yahoo!
Inoneembodiment,thedevice100includesatouchscreen
Widgets).
ss 112, a menu button 204, a push button 206 for powering the
In conjunction with RF circuitry 108, display system 112,
device on/offand locking the device, and volume adjustment
display system controller 156, contact module 130, graphics
button(s) 208. The push button 206 may be used to tum the
module 132, text inputmodule 134, and browsermodule 147,
power on/ofI on the device by depressing the button and
the widget creatormodule 150 may be usedby a user to create
holdingthe button inthe depressed state for a predefined time
widgets (e.g., turning a user-specified portion of a web page 60 interval; to lock the device by depressing the button and
into a widget).
releasing the button before the predefined time interval has
In conjunction with display system 112, display system
elapsed; and/or to unlock the device or initiate an unlock
controller 156, contact module 130, graphics module 132,
process. In an alternative embodiment, the device 100 also
and text input module 134, the search module 151 may be
may accept verbal input for activation ordeactivationof some
used to search for text, music, sound, image, video, and/or 65 functions through the microphone 113.
other files in memory 102 that match one or more search
Attention is now directed towards embodiments of user
criteria (e.g., one or more user-specified search terms).
interfaces ("UI") and associatedprocesses that may be imple-

n--.. ...-..:J-A km, i EODTT% Sweaan the DIDO lanetta

- AÊÑEÑÛ

APLNDC00022516

US 7,469,381 B2
19

20
menu button enables a user to access any desired application

mented on a portable multifunction device 100 and/or on a
with at most two inputs, such as activating the menu button
device 1700 with a touch-screen display (FIG. 17).
204 and then activating the desired application (e.g., by a tap
FIG. 3 illustrates an exemplary user interface forunlocking
orother finger gesture on the icon corresponding to the applia portableelectronic device in accordancewith some embodiments. In some embodiments, userinterface 300 includes the 5 cation).
following elements, or a subset or superset thereof
In some embodiments, UI 400 provides integrated access
Unlock image 302 that is moved with a finger gesture to
to both widget-based applications and non-widget-based
unlock the device;
applications. In some embodiments, all of the widgets,
Arrow 304 that provides a visual cue to the unlock gesture;
whetheruser-created or not, are displayed in UI 400. In other
Channel 306 that provides additional cues to the unlock 10 embodiments, activating the icon for user-created widget
149-6 may lead to another UI (not shown) that contains the
gesture;
Time 308;
user-created widgets or icons corresponding to the user-creDay 310;
ated widgets.
Date 312; and
In some embodiments, a usermay rearrangethe icons inUI
Wallpaper image 314.
15 400, e.g., usingprocesses described in U.S. patent application
In some embodiments, the device detects contact with the
Ser. No. 11/459,602, "Portable Electronic Device With Intertouch-sensitive display (e.g., a user's finger making contact
face Reconfiguration Mode;' filed Jul. 24, 2006, which is
on or near the unlock image 302) while the device is in a
hereby incorporated by reference herein in its entirety. For
user-interface lock state. The device moves the unlock image
example, a usermay move application icons in and out oftray
302 in accordance with the contact. The device transitions to 20 408 using finger gestures.
a user-interface unlock state if the detected contact corteIn some embodiments, UI 400 includes a gauge (not
sponds to a predefined gesture, such as moving the unlock
shown) that displays an updated account usage metric for an
image across channel 306. Conversely, the device maintains
account associated with usage of the device (e.g., a cellular
the user-interface lock state if the detected contact does not
25 phone account), as described in U.S. patent application Ser.
correspond to the predefined gesture. As noted above, proNo. 11/322,552, "Account Information Display For Portable
cesses that use gestures on the touch screen to unlock the
Communication Device," filed Dec. 23, 2005, which is
device are described in U.S. patent application Ser. No.
hereby incorporated by reference herein in its entirety.
11/322,549, "Unlocking a Device by Performing Gestures on
As discussed above, UI 400 may display all ofthe available
an Unlock Image," filed Dec. 23, 2005, which is hereby
30 applications 136 on one screen so that there is no need to
incorporated by reference herein in its entirety.
scroll through a list of applications. However, in some
FIG. 4 illustrates an exemplary user interface for a menu of
embodiments a touch-sensitive display may include a GUI
applications on a portable multifunction device in accordance
with one or more windows that display only a portion of a list
with some embodiments. In some embodiments, user interof items (e.g., information items) or of an electronic docuface 400 includes the following elements, or a subset or
3s ment. In response to detecting a movement ofan object on or
superset thereof:
nearthetouch-sensitivedisplay, the list may be scrolled orthe
Signal strength indicator 402 for wireless communication;
electronic document may be translated. Detecting the moveTime 404;
ment of the object may include determining speed (magniBattery status indicator 406;
tude), velocity (magnitude and direction), and/oran acceleraTray 408 with icons for frequently used applications, such
4e tion (including magnitude and/or direction) of the object.
as one or more of the following:
Scrolling through the list or translating the document may be
Phone 138;
accelerated in response to an accelerated movement of the
E-mail client 140, which may include an indicator410 of
object. In some embodiments, the scrolling and acceleration
the number of unread e-mails;
of the scrolling, or translation and acceleration of the transBrowser 147; and
45 lation, may be in accordance with a simulation of a physical
Music player 146; and
device having friction, i.e., damped motion. For example, the
Icons for other applications, such as one or more of the
scrolling or translation may correspond to a simulation of a
following:
force law orequationofmotion having amass or inertial term,
IM 141;
as well as a dissipative term. In some embodiments, the
Image management 144;
so simulation may correspond to a cylinder rotating about its
Camera 143;
axis.
Video player 145;
In some embodiments, accelerated movement of the
Weather 149-1;
detected object may include an accelerated movement of a
Stocks 149-2;
pointofcontactfollowedbyabrealdagofthepointofcontact.
Blog 142;
Calendar 148;
55 For example, the user may make contact with the touchCalculator 149-3;
sensitive display, swipe or sweep one or more of his or her
fingers along the display (i.e., move and/or accelerate the
Alarm clock 149-4;
Dictionary 149-5; and
point of contact), and optionally, break the point of contact
withthe display, i.e., move the one or more fingers away from
User-created widget 149-6.
In some embodiments, UI 400 displays all ofthe available so the display. The swipe or sweep may be along a predefined
axis ofthe touch-sensitive display or may be within a predeapplications 136 on one screen so that there is no need to
termined angle of a predefmed direction on the touch-sensiscroll through a list ofapplications (e.g., via a scroll bar orvia
tive display. In other embodiments, the accelerated movea swipe gesture). In some embodiments, as the number of
ment ofthe point of contact may include a first user gesture
applications i.....- , the icons corresponding to the applications may decrease in size so that all applications may be 65 orientedalongapredefmedaxisofthetouch-sensitivedisplay
or oriented within a predetermined angle of a predefined
displayed on a single screen without scrolling. In some
direction on the touch-sensitive display.
embodiments, having all applications on one screen and a

r

rwrisimme har I IRDTri fream the DIRA imana Datshnma nn BAI95/9011

APLNDC00022517

US 7,469,381 B2
21

22

Scrolling through the list of items or translating the electronic document may be further accelerated in response to
detection of a second movement of an object on or near the

letter in the alphabet that has one or more entries. In some
embodiments, the scrolling may be in accordance with a
simulation of an equation of motion having friction.
The scrolling may include scrolling through a respective
information item subset if the point of contact moves over a
corresponding respective index item in the index items. The
scrolling may have an associated scroll speed based on a
speed ofmovement ofthe point ofcontact over the respective
index item and the number of items in the information item
subset corresponding to the respective index item. For
example, the scroll speed may be faster for subsets that have
more entries than subsets with fewer entries. The scrolling
may include scrolling through all items in a plurality of the
information item subsets in response to the point of contact
moving over the corresponding index items in the displayed
index.
If it is determined that the point of contact with the index
corresponds to a respective index item in the index, the list of
information items may be scrolled to a corresponding subset
of the list of information items. For example, if the user
selects an index item, such as the letter 'R', in the set of index
symbols, the list of items may be smoothly scrolled to the
corresponding subset for the letter 'R' in the list of items.
Alternatively, the displayed list of information items jump
directly from a current scroll position to a scroll position in
which information items corresponding to the index item 'R'
are displayed.

touch-sensitive display, such as a second sweepingmotionof

the point of contact along the predefined axis or within the s
predetermined angle of a predefined direction on the touchsensitive display and/or a second user gesture oriented along
the predefined axis or within the predetermined angle of a
predefined direction on the touch-sensitive display. For
example, the user may swipe one or more ofhis orher fingers 10
along the touch-sensitive display two or more times.
The scrolling through the list of items or the translation of
the electronic document may be stopped in accordance with
the user breaking the point of contact and then establishing a

substantially stationary point of contact with the touch-sen- 15
sitive display for at least a pre-determinedperiod oftime. For
example, after swiping one or more ofhis orher fingers along
the touch-sensitive display and breaking the point ofcontact,
the user may touch the display and hold the one or more

fmgers that are touching the display stationary (or approxi- 20
mately stationary) for one or more seconds, or fractions of a

second.
The direction ofscrolling or translation may be reversed in
response to intersecting a virtual boundary corresponding to
a terminus of the list or an edge of the electronic document. 25

The scrolling reversal or translation reversal may correspond
to a damped motion. For example, during scrolling, a displayedportion ofthe list ofitems may appearto bounce offof
In the present document, the term"if" may be construed to
a boundary ofthe window in the touch-sensitive displaywhen
mean "when," or "upon," or "in response to determining," or
a beginning or an end ofthe list ofitems is reached. Similarly, 30 "in response to detecting," depending on the context Simiduring translation, a displayed portion ofthe electronic docularly, the phrase "ifit is determined" or "if [a stated condition
ment may appear to bounce off of a boundary ofthe window
or event] is detected" may be construed to mean "upon deterin the touch-sensitive display when an edge ofthe document
mining" or "in response to determining" or "upon detecting"
is reached. The apparent bounce may correspond to a simuthe statedcondition orevent, or"in response to detecting"the
lation of a viscous or elastic ball having momentum in a first 35 stated condition or event, depending on the context.
direction striking an immovable and/or inelastic object, such
If the point of contact with the touch-sensitive display
as a wall. The subsequentmotionofthe document (themotion
corresponds to a user selection of a respective information
of which corresponds to the ball in the aforementioned analitem in the list ofinformation items, information correspondogy) may be damped, for example, by including a friction or
ing to the respective information item may be displayed on
dissipative term in the simulation. A parametercorresponding 40 the touch-sensitive display. For example, if the user selects a
to the friction term in the simulation may be adjustable,
respective name, the corresponding contact information may
be displayed.
allowing the document to reach equilibrium in contact with
the virtual boundary, or displaced from the virtual boundary.
While scrolling throughrespective information subsets, an
In some embodiments movement ofthe point ofcontact by
index symbol may displayedin conjunctionwitheach respecthe user over an index on the touch-sensitive display may be 45 tive information item subset. In some embodiments, respecdetermined. In some embodiments, the index may be distive index symbols may be displayed adjacent to correspondplayed in a first region or a first window ofthe touch-sensitive
ing subsets (such as displayed text) of the list of information
display while the list ofitems or information items during the
items. In some embodiments, a respective index symbol may
be displayed at an upper edge of a window containing the
scrolling may be displayed in a second region or a second
window of the touch-sensitive display. The displayed inder 50 displayed text of the respective information item subset.
may have a sequence ofindex items. In an exemplary embodiThe index symbol corresponding to a respective informament, the sequence of index items may include letters in the
tion subset maybe translucently displayedoverthe respective
alphabet, i.e., the index may include an alphabetical index.
information item subset. The translucently displayed index
The list of information items may include an alphabetically
symbol may have a different font color than that used to
ordered list of information items. The alphabetically ordered ss display text in the information item subset, and/or it may be
list ofinformation items may include contact information, for
displayedusing a larger font than the font used to display text
example, in a user's contact list or address book.
in the information item subset.
In response to movement ofthe user's point ofcontact over
If the list of information items contains no items for a
a displayed index, the list of information items on the touchparticularindex symbol, i.e., no entries for a particularsubset,
sensitive display may be scrolled. The list of information so a first index symbol preceding a particular index symbol and
items may include a sequence of information item subsets
a second index symbol following the index symbol may be
corresponding to the sequence of index items. The subsets
displayed in conjunction with scrolling through the list of
may include one or more categories. For example, a respecinformation items from the information subset corresponding
tive category may include contact information for one or
to the first index symbol to the information subset corremore individuals whose first and/or last names beginwith one 65 sponding to the second index symbol. The particular index
or more respective letters, such as the letter 's'. In an exem-

symbol may not be displayed in conjunction with the dis-

plary embodiment, there is a subset corresponding to each

played text of the list of information items during the scroll

APLNDC00022518

US 7,469,381 B2
23

24

In response to detecting the movement, a list of items
displayed on the touch screen display is scrolled in a first
may be skipped whenthe list ofinformation items contains no
direction (504). In some embodiments, the list is a list of
items for the particular index symbol.
email messages, as illustrated in FIGS. 6A-6D. In some
In some embodiments, the list scrolling described here
operates without displaying a scroll bar. Similarly, in some 5 embodiments, the list of items is a list of instant message
conversations, a list of favorite phone numbers, a list of conembodiments, the translation of electronic documents
tact information (sometimes called a contact list or address
described here operates without displaying scroll bars. The
book list), a list oflabels, a list ofemail folders, a list ofemail
user's sweeping motion on the touch-sensitive display operaaddresses, a list ofphysical addresses, a list ofringtones, a list
tion may be performed directly on top ofthe displayed list or
displayed electronic document, and may include a sweeping 10 of album names, or a list of bookmarks. In some embodithrough. For example, display of a respective index symbol

or gliding motion, near or in contact with the display's sur-

ments, the first direction is a vertical direction; in some other

embodiments, the first direction is a horizontal direction. In
face, along apath anywherewithina display window inwhich
some embodiments, scrolling the list in the first direction
the list or electronic document is displayed. While a scroll bar
prior to reaching a terminus of the list has an associated
could potentially be displayed in conjunction with the displayed list, the scrolling or translation described here can be 15 scrolling speed corresponding to a speed ofmovement ofthe
object (506). In some embodiments, the list is scrolled in
independent of any such scroll bar. In some embodiments, if
accordancewith a simulation ofan equation ofmotion having
a scroll bar is used, then an upward movement of a point of
friction (508).
contact on the scroll bar may cause earlier entries in the list to
Ifa terminus ofthe list is reached (e.g., upon reaching the
be displayed, whereas a downward movement ofthe point of
contact onthe scroll barmay cause later entries in the list to be 20 terminus of the list) while scrolling the list in the first direction while the object is still detected on or near the touch
displayed.
In some embodiments, scrolling or translation may be in
accordance with a speed of movement of a detected object,
such as a speed ofmovement ofa point ofcontact. The speed
may be a time average of values determined during several 25

time intervals. In an exemplary embodiment, the speed,
velocity and/oraccelerationmay be determinedoverfive time

screen display, an area beyond the terminus of the list is

displayed (510-Yes, 514). In some embodiments, the list has
a first itern and a last item and the terminus is either the first
item or the last item. For example, in FIG. 6B the email 3534

fromAaronJones is the first item and thus the terminus ofthe
corresponding list of ernails. In some embodiments, the area

intervals, where a respective time interval corresponds to an

beyond the terminus of the list is white (516). In some

when one or more frames are skipped or not displayed. In

3536 and the background of the listed emails are white.

embodiments, the list ofitems has a background and the area
- ofa frame rate, such as 0.0167 s, ofa display. In some
embodiments, the speed, velocity and/or accelerationmay be 30 beyond the terminus of the list is visually indistinct from the
background (518). For example, in FIG. 6C both the area
determined even when a variable frame rate is used, such as

these embodiments, the speed, velocity, and/or acceleration

After the object is no longer detected on or near the touch

screen display, the list of items is scrolled in a second direcinterval and/or may be projected based on values determined 35 tion opposite the first direction until the area beyond the
terminus of the list is no longer displayed (520). In some
in a preceding and/or a subsequent time interval.
embodiments, the list is scrolled inthe second direction using
In some embodiments, the scrolling or translation after a

may be determined two or more times for the respective time

a damped motion (522). In some embodiments, the change
from scrolling the list in the first direction to scrolling the list
one ormore time intervals priorto the breakingofthe contact. 40 in the second direction until the area beyond the terminus of
the list is no longer displayed makes the t--- of the list
For example, the velocity v,of scrolling or translation one or
appearto be elastically attachedto an edge ofthe touch screen
more time intervals after breaking contact maybe determined
display or to an edge displayed on the touch screen display
usmg
(524).
/¯3e
•
45
In some embodiments, scrolling in the first direction prior
to reaching the terminus of the list has a first associated
where v, is a currentvalue ofthe velocity whenthe contact is
scrolling distance that corresponds to a distance ofmovement
broken, a is a current value of the acceleration when the
of the object prior to reaching the terminus of the list. For
contact is broken and At is an elapsed time, such as one time
example, a scrolling distance priorto reachingthe terminus of
interval. The velocities and/or acceleration in such a calcula- so the list shown in FIGS. 6A-6D may correspond to a distance
tion may be projected along an axis or directionofthe scrolltraversed on the touch screen display by the swipe gesture
ing or translation. In some embodiments, in subsequent time
3514 before the terminus is reached. Displaying an area
intervals following the determination ofthe velocitybased on
beyondthe terminus ofthe list includes scrolling the list inthe
the acceleration and/or the velocity in one or more time interfirst directionfor a secondassociated scrolling distancethat is
vals prior to the breaking of the contact, the velocity of the ,, less than a distance of movement of the object after the
scrolling or translation may be tapered. For example, in each
terminus is reached. For example, in FIG. 6C, after the tersuccessivetime interval the velocitymaybe decreasedby 5%.
minus is reachedthe list is scrolled for a distance 3538, which
When the velocity crosses a lower threshold, it may be set to
may be less than a distance traversed on the touch screen
zero.
display by the swipe gesture 3514 after the terminus is
FIG. 5 is a flow diagram illustrating a method 500 of so reached.
scrolling through a list in accordance with some embodiIn some embodiments, scrolling in the first direction prior
ments. The method 500 provides a simple visual indicator to
to reaching a terminus ofthe list has a first associated scrolla user that a terminus of a list has been reached.
ing speed that corresponds to a speed of movement of the
Movementofan object is detectedon orneara touch screen
object. For example, a scrolling speed prior to reaching the
display ofa device (502). In some embodiments, the object is 65 terminus ofthe list shown in FIGS. 6A-6D may correspondto
a speed on the touch screen display ofthe swipe gesture 3514
a finger. In some embodiments, the device is a portable multifunction device.
before the terminus is reached. Displaying an area beyondthe
user optionally breaks the contact may be in accordance with

the change in the acceleration and the speed or the velocity in
1

!

APLNDC00022519

US 7,469,381 B2
25

26

upward and/or vertically downward swipe gestures on the
touch . In the example ofFIG. 6A, a portion ofa list of
associated scrolling speed is slower than the first associated
emails is displayed in the screen area, including a top displayedemail 3530 from BruceWalker and a bottom displayed
scrolling speed. For example, in FIG. 6C, displayingthe area
3536 beyondthe terminus ofthe list may include scrollingthe 5 email 3532 from Kim Brook. A user performs a vertically
downward swipe gesture 3514 to scroll toward the top of the
list at a speed that is slowerthan the scrolling speed beforethe
list. The vertically downward gesture 3514, which may be a
terminus is reached. In some embodiments, the second assofmger gesture, corresponds to the movement of an object on
ciated speed is a fraction (e.g., one-half or one-third) of the
or near the touch screen that is detected in operation 502 of
first associated speed. In some embodiments, the second
associated speed is the square root of the first associated 10 process 500 (FIG. 5). The vertically downward gesture 3514
need not be exactly vertical; a substantially vertical gesture is
speed.
sufficient. In some embodiments, a gesture within a predeterIfa terminus ofthe list is not reachedwhile scrolling the list
mined angle of being perfectly vertical results in vertical
in the first direction while the object is still detected on ornear
scrolling. In one embodiment, a gesture within 27 degrees of
the touch screen display, the process 500 is complete (510No, 512). The process 500 may be re-initiated upon subse- 15 being perfectly vertical results in vertical scrolling.
As a result of detecting the vertically downward gesture
quent detection of another -.-ent of an object on or near
3514, in FIG. 6B the displayed emails have shifted down,
the touch screen display (502).
suchthat theprevious bottom displayedemail 3532 from Kim
FIGS. 6A-6D illustrate the scrolling of a list of items to a
Brook is no longer displayed, the previous top displayed
terminus of the list, at which point an area beyond the terminus is displayed and the list is then scrolled in an opposite 20 email 3530 from Bruce Walker is now second from the top,
and the email 3534 from Aaron Jones, which was not disdirection until the area beyond the terminus is no longer
played in FIG. 6A, is now displayed at the top ofthe list. This
displayed, in accordance with some embodiments. While
FIGS. 6A-6D illustrate this scrolling in the context of a porshifting ofemails is an example of the scrolling described in
operation 504 of process 500 (FIG. 5).
table multifunction device 100, this scrolling is not limited to
portable multifunction devices. In the example of FIGS. 25 In this example, the email 3534 from Aamn Jones is the
6A-6D, the listed items are email messages; FIGS. 6A-6D
first email in the list and thus is the terminus ofthe list. Upon
reaching this email 3534, in response to continued detection
illustrate an exemplary user interface 3500A formanaging an
inbox in accordance with some embodiments. An analogous
of the vertically downward gesture 3514, an area 3536 (FIG.
6C) above the first email 3534 (i.e., beyond the terminus of
user interface may be used to display and manage other mailboxes (e.g., drafts, sent, trash, personal, etc.). In addition, 30 the list) is displayed, as described in operation 514 ofprocess
500 (FIG. 5). In some embodiments, the area displayed
other types of lists are possible, including but not limited to
lists of instant message conversations, favorite phone numbeyond the terminus of the list is visually indistinct from the
background of the list, as described in operation 518 of pmbers, contact information, labels, email folders, email
cess 500 (FIG. 5). In FIG. 6C, both the area 3536 and the
addresses, physical addresses, ringtones, album names or
35 background of the emails (e.g., emails 3534 and 3530) are
bookmarks.
white and thus are visually indistinct.
In some embodiments, user interface 3500A include the
Once vertically downward gesture 3514 is complete, such
following elements, or a subset or superset thereof:
402, 404, and 406, as described above;
that a corresponding object is no longer detected on or near
the touch screen display, the list is scrolled in an opposite
a create email icon 3310 that when activated (e.g., by a
finger tap on the icon) initiates display of a UI to create 40 direction until the area 3536 is no longer displayed. FIG. 6D
illustrates the result ofthis scrolling in the opposite direction,
a new email message;
which corresponds to operation 520 ofprocess 500 (FIG. 5):
mailboxes icon 3502 that when activated (e.g., by a finger
the email 3534 fromAamn Jones is now displayed at the top
tap on the icon) initiates the display ofa UI listing email
ofthe screen area allotted to the list and the area 3536 is not
mailboxes (i.e., folders);
unread messages icon 3504 that displays the number of 45 displayed.
In the example of FIGS. 6A-6D, a vertically downward
unread messages in the inbox;
gesture resulted in display of an area beyond the first item in
names 3506 of the senders of the email messages;
the list. Similarly, a vertically upward gesture may result in
subject lines 3508 for the email messages;
display of an area beyond the last item of the list, if the
dates 3510 ofthe email messages;
unread message icons 3512 that indicate messages that so vertically upward gesture continues once the list has been
scrolled to the last item. The last item may be considered a
have not been opened;
terminus of the list, similar to the first item. As discussed
preview pane separator 3518 that separates the list ofmesabove, the gesture need not be exactly vertical to result in
sages from a preview of a selected message in the list;
vertical scrolling; a gesture within a predefmed range of
settings icon 3520 that when activated (e.g., by a fmgertap
on the icon) initiates the display of a UI to modify ss angles from perfectly vertical is sufficient.
In some embodiments, instead of scrolling a list ofitems in
settings;
one dimension, a user may desire to translate an electronic
move message icon 3522 that when activated (e.g., by a
document in two dimensions. Ifthe electronic document fills
finger tap on the icon) initiates the display of a UI to
more than the screen area allotted to display the document,
move messages;
Delete symbol icon 3524 that when activated (e.g., by a 60 the screen will only display a portion of the document. The
user may translate the electronic document to view portions
finger tap on the icon) initiates display ofa UI to confirm
ofthe document that are not initially displayed.
that the user wants to delete the selected email;
FIG. 7 is a flow diagram illustrating a method 700 of
Reply/Forward icon 3526 that when activated (e.g., by a
translating an electronic document in accordance with some
fmger tap on the icon) initiates display of a UI to select
how to reply or forward the selected email;
es embodiments. The method 700 provides a simple visual indi-

terminus of the list includes scrolling the list in the first
direction at a second associated scrolling speed. The second

If the list of emails fills more than the allotted screen area,

the user may scroll through the emails using vertically

cator to a user that one or more edges of an electronic docu-

ment are being displayed.

APLNDC00022520

US 7,469,381 B2

27
Movementofanobjectisdetectedonornearatouchscreen

28
associatedtranslatingdistance is less than a distance ofmove-

ment of the object after reaching the edge of the electronic
document. For example, in FIG. SC, after the edge is reached
the web page 3912 is translated by a distance indicated by
tifunction device.
In responseto detecting the movement, an electronic docu- s opposing arrows 3928-1 and 3928-2, which may be less than
a distance traversed on the touch screen display by the swipe
ment displayed on the touch screen display is translated in a
gesture 3925 after the terminus is reached.
first direction (704). In some embodiments, the electronic
In some embodiments, translating in the first direction
document is a web page, as illustrated in FIGS. 8A-8D. In
priorto reachingan edgeofthe electronic document has a first
some embodiments, the electronic document is a digital
display of a device (702). In some embodiments, the object is
a finger. In some embodiments, the device is a portable mul-

image. In some embodiments, the electronic document is a 10 associated translating speed that corresponds to a speed of

movement of the object. For example, a speed of translation
prior to reaching the edge of the web page 3912 shown in
FIGS. SA-8D may correspond to a speed ofmovement ofthe
some embodiments, the first direction corresponds to the
swipe gesture 3925. Displaying an area beyond the edge of
direction of movement of the object detected on or near the 15 the electronic document includes translating the electronic
document in the first direction at a second associated transdisplay but is not necessarily identical to the direction of
lating speed. The second associated translating speed is
movement of the object.
slower than the first associated translating speed. For
In some embodiments, translating the electronic document
example, in FIG. SC, displaying the area 3930 beyond the
in the first direction prior to reaching an edge ofthe electronic
document has an associated speed of translation correspond- 20 edge ofthe web page 3912 may include translating the web
page 3912 at a speed that is slower than the speed of translaing to a speed of movement of the object (706). In some
tion before the edge is reached. In some embodiments, the
embodiments, the electronic document is translated in accorsecond associated speed is a fraction (e.g., one-half or onedance with a simulation of an equation of motion having
third) ofthe first ...- «ted speed. In some embodiments, the
friction (708).
Ifan edge ofthe electronic document is reached (e.g., upon 25 second associated speed is the square root of the first associated speed.
reaching the edge of the document) while translating the
If an edge ofthe electonic document is not reached while
electronic document in the first direction while the object is
translatingthe electronic document inthe first direction while
still detected on or near the touch screen display, an area
the object is still detected on or near the touch screen display,
beyond the edge of the electronic document is displayed
(710-Yes, 714). In some embodiments, the area beyond the 30 the process 700 is complete (710-No, 712). The process 700
may be re-initiated upon subsequent detection of another
edge ofthe electronic document is black, gray, a solid color,
movement of an object on or near the touch screen display
or white (716). In some embodiments, the area beyond the
(702).
edge of the electronic document is visually distinct from the
FIGS. SA-8D illustrate the translation of an electronic
document (718). For example, the area 3930 beyondthe edge
of the web page 3912 in FIG. SC is black, in contrast to the 35 document to an edge ofthe document, at which point an area
beyond the edge is displayed and the document is then transwhite background of the web page 3912. In some other
lated in a second direction until the area beyond the edge of
embodiments, a wallpaper image such as a picture or pattern
thedocumentisnolongerdisplayed,inaccordancewithsome
may be displayed in the area beyond the edge ofthe electronic
embodiments. While FIGS. 8A-8D illustrate this translation
document.
After the object is no longer detected on or near the touch 40 in the context of a portable multifunction device 100, this
translation is not limited to portablemultifunction devices. In
screen display, the electronic document is translated in a
the example of FIGS. 8A-8D, the document is a web page
second direction until the area beyond the edge of the elec3912; FIGS. 8A-8D illustrate an exemplary user interface for
tronic document is no longer displayed (720). Forexample, in
a browser in accordance with some embodiments. An analoFIG. 8D the web page 3912 has been translated such that the
area 3930 beyond its edge is no longer displayed. In some 45 gous user interface may be used to display other types of
electronic documents, such as word processing, spreadsheet,
embodiments, the second direction is opposite the first direcemail, presentation documents, or digital images.
tion. In some embodiments, the electronic document is transIn some embodiments, user interface 3900A of FIGS.
lated in the second direction using a damped motion (722). In
BA-8D includes the following elements, or a subset or supersome embodiments, the change from translating the electronic document in the first direction to translating the elec- so set thereof
402, 404, and 406, as described above;
tronic document in the second direction until the area beyond
Previous page icon 3902 that when activated (e.g., by a
the edge of the electronic document is no longer displayed
finger tap on the icon) initiates display of the previous
makes the edge of the electronic document appear to be
elastically attached to an edge of the touch screen display or
web page;
to an edge displayed on the touch screen display (724).
ss Web page name 3904;
Next page icon 3906 that when activated (e.g., by a finger
In some embodiments, translating in the first direction

word processing, spreadsheet, email, or presentation document. In some embodiments, the first direction is a vertical
direction, a horizontal direction, or a diagonal direction. In

prior to reachingan edgeofthe electronic documenthas a first

associated translating distance that corresponds to a distance
of movement of the object prior to reaching the edge of the
electronic document. For example, a distance oftranslationof so
the web page 3912 shown in FIGS. 8A-8D prior to reaching
the edge of the document may correspond to a distance traversed on the touch screen display by the swipe gesture 3925
before the edge is reached. In some embodiments, displaying
an area beyond the edge of the electronic document includes 65

tap on the icon) initiates display of the next web page;

URL (Uniform Resource Locator) entry box 3908 for
inputting URLs ofweb pages;
Refresh icon 3910 that when activated (e.g., by a finger tap
on the icon) initiates a refresh of the web page;
Web page 3912 or other structured document, which is
made ofblocks 3914 oftext content and other graphics
(e.g., images);
Settings icon 3916 that whenactivated (e.g., by a finger tap

translating the electronic document in the first direction for a

on the icon) initiates display of a settings menu for the

second associated translating distance, wherein the second

browser;

Copy provided by USPTO from the PIHS Image Database on 04 « ww 1

APLNDC00022521

US 7,469,381 B2
29

30

until the area 3930 is no longer displayed. FIG. 8D illustrates
Bookmarks icon 3918 that when activated (e.g., by a fmger
the result of this translation, which corresponds to operation
tap on the icon) initiates display of a bookmarks list or
720 of process 700 (FIG. 7): block 9 (3914-9) is now dismenu for the browser;
played in the lower right-hand corner of the portion of the
Add bookmark icon 3920 that when activated (e.g., by a
fmger tap on the icon) initiates display ofa UI for adding 5 screenallottedto displaythewebpage 3912 and the area 3930
not displayed. In some embodiments, the direction of transbookmarks; and
New window icon 3922 that when activated (e.g., by a
lation is not necessarily opposite to the original direction but
fmger tap on the icon) initiates display ofa UI for adding
may be in any direction such that, upon completion of the
translation, the area beyond the edge(s) of the electronic
new windows to the browser.
In some embodiments, the device analyzes the render tree 10 document is no longer displayed.
ofthe web page 3912 to determine the blocks 3914 in the web
FIG. 9 is a flow diagram illustrating a process 900 of
page. In some embodiments, a block 3914 corresponds to a
displaying an electronic document having a document length
render node that is: replaced; a block; an inline block; or an
and a document width, in accordance with some embodiinline table.
ments. The process 900 provides a simple visual indicator to
In FIG. 8A, the web page fills more than the allotted screen 15 a user that an electronic document is being displayed at a
area: only the left sides of block 7 (3914-7) and block 8
minimum magnification (e.g., the electronic document can(3914-8) are displayed and only the top left comer ofblock 9
not be 2comed out and/or demagnified further).
(3914-9) is displayed. To view the partially displayed blocks,
The process 900 is performed at a device with a touch
a user may translate the displayed document by gesturing on
screendisplay. In some embodiments, the device is a portable
the touch screen in accordance with some embodiments.
20 multifunction device. In some embodiments, the electronic
In some embodiments, in response to a substantially vertidocument is a web page (e.g., web page 3912, FIGS. 10Acal upward (or downward) swipe gesture by the user, the web
10C). In some embodiments, the electronic document is a
page (or, more generally, other electronic documents) may
digital image. In someembodiments, the electronic document
translate one-dimensionally upward (or downward) in the
is a word processing, spreadsheet, email or presentation
vertical direction. In some embodiments, a gesture is consid- 25 document.
ered substantially vertical ifit is withina predeterminedangle
The electronic document is displayed (902) at a first magof being perfectly vertical. For example, in response to an
nification on the touch screen display. A gesture is detected
upward swipe gesture by the user that is within a predeter(904) on or near the touch screen display corresponding to a
mined angle (e.g., 27°) of being perfectly vertical, the web
page may scroll one-dimensionally upward in the vertical 30 command to zoom out by a user-specified amount. In some
embodiments, the gesture is a pinching gesture (e.g., gesture
direction.
3951/3953, FIG. 10A).
Conversely, in some embodiments, in responseto a gesture
In response to detecting the gesture, the electronic docuthat is not within a predetermined angle (e.g., 27°) of being
ment is displayed (906) at a magnification less than the first
perfectly vertical, the web page may translate two-dimensionally (i.e., with simultaneous movement in both the verti- 35 magnification. For exarnple, the web page 3912 is shown at a
lesser magnification in FIG. 10B than in FIG. 10A.
cal and horizontal directions). For example, in responseto an
upward swipe gesture by the user that is not within a predeIf the document length or document width is not entirely
displayed (908-No) while the gesture is still detected on or
termined angle (e.g., 27°) ofbeing perfectly vertical, the web
page may translate two-dimensionally along the direction of
near the touch screen display, the process 900 is complete
the swipe.
40 (910).
In the example of FIG. 8A, an upward swipe gesture 3925
If, however, the document length (e.g., 3957, FIG. 10B) or
is not within a predetermined angle of being perfectly vertidocument width (e.g., 3959, FIG. 10B) is entirely displayed
cal. Therefore, as a result of detecting the upward swipe
(908-Yes) while the gesture (e.g., 3951/3953) is still detected
gesture 3925, the web page is translated intwo dimensions. In
onornearthe touch screendisplay,the electronic document is
this example, the translation is approximately diagonal. FIG. 45 displayed (912) at a magnification wherein areas beyond
8B illustrates the result of this translation: blocks 8 (3914-8)
opposite edges of the electronic document (e.g., areas 3955,
and9 (3914-9) arenow fully displayed;blocks1(3914-1)and
FIG. 10B) are displayed.
2 (3914-2) are now only partially displayed, and block 3
In some embodiments, the areas beyond opposite edges of
(3914-3) is no longer displayed at all. This translation is an
the electronic document include an area beyond a top edge of
example of the translation described in operation 704 ofpro- 50 the document and an area beyond a bottom edge ofthe docucess 700 (FIG. 7).
ment. Insome embodiments, the areas beyond opposite edges
In FIG. 8B, block 9 (3914-9) is in the lower right-hand
of the electronic document include an area beyond a right
corner ofthe web page 3912; both the bottom and right edges
edge of the document and an area beyond a left edge of the
of the web page have been reached while translating the web
document. In some embodiments, the areas beyond opposite
page. Upon reaching these edges of the document, in 55 edges ofthe electronic document include an area beyond a top
response to continued detection ofthe upward gesture 3925,
edge of the document, an area beyond a bottom edge of the
an area 3930 (FIG. SC) beyond the bottom and right edges of
document, an area beyond a right edge of the document, and
the web page is displayed. In some embodiments, the area
an area beyond a left edge of the document (e.g., FIG. 10B).
displayed beyond the edge(s) of an electronic document is
visually distinct from the document, as describedin operation 60 In some embodiments, the areas beyond opposite edges of
the electronic document are black, gray, a solid color, or
718ofprocess700(FIG.7).InFIG.8C,thearea3930isblack
white. In some embodiments, the areas beyond opposite
and thus is visually distinct from the white background ofthe
edges of the electronic document are visually distinct from
web page 3912.
the document. For example, the areas 3955 (FIG. 10B) are
Once the upward gesture 3925 is complete, such that a
corresponding object is no longer detected on or near the 65 black and thus are visually distinct from the web page 3912.
touch screen display, the web page 3912 is translated (e.g., in

a direction opposite to the original direction of translation)

Upon detecting termination of the gesture, the electronic

document is displayed (914) at a magnification wherein the

Copy provided by USPTO trom the PIRS Image Database on 04125/2011

APLNDC00022522

US 7,469,381 B2
31

32
gesture 3931/3933, a portion of the web page 3912 is displayed at the predefined magnification, as illustrated in FIG.
12C,andaportionofthedigitalimage1302isdisplayedatthe
predefined magnification, as illustrated in FIG. 13C.
In some embodiments, immediately prior to detecting termination of the gesture, a last decreased portion of the electronic document is displayed at a first resolution. Upon
detecting termination ofthe gesture, the respective portion of
the electronic document is displayed at a second resolution
that is greater than the first resolution.
FIGS. 12A-12C illustrate the display ofan electronic document at multiple magnifications in accordance with some
embodiments. While FIGS. 12A-12C illustrate displaying
these multiple magnifications in the context of a portable
multifunction device 100, displaying these multiple magnifications is not limited to portable multifunction devices. In the
example of FIGS. 12A-12C, the document is a web page
3912; FIGS. 12A-12C (like FIGS. SA-8D) illustrate an exemplary user interface for a browser in accordance with some
embodiments. An analogous user interface may be used to
display other types of electronic documents, such as digital
images or word processing, spreadsheet, email, or presentation documents.
In FIG. 12A, a first portion of the web page 3912 is displayed at a first magnification. The web page 3912 fills more
than the allotted screen area: only the left sides of block 7
(3914-7) and block 8 (3914-8) are displayed and only the top
left corner of block 9 (3914-9) is displayed.
In response to detecting a de-pinching gesture 3931/3933
(FIG. 12A), decreasing portions of the web-page 3912 are
displayed at increasing magnifications compared to the magnification shown in FIG. 12A. For example, the portion ofthe
web page 3912 shown in FIG.12B is smaller than and has a
higher magnification than the portion of the web page 3912
shown in FIG. 12A.
In the example of FIG. 12B, the magnification exceeds a
predefined magnification. Upon detecting termination of the
gesture 3931/3933, a portion of the web page 3912 is displayed at the predefined magnification, as illustrated in FIG.
12C.
FIGS.13A-13Cillustratethedisplayofanelectronicdocument at multiple magnifications in accordance with some
embodiments. While FIGS. 13A-13C illustrate displaying
these multiple magnifications in the context of a portable
multifunction device 100, displaying these multiple magnifications is not limited to portable multifunction devices. In the
example of FIGS. 13A-13C, the document is a digital image
1302 that includes an image of a person 1304.
In FIG. 13A, a digital image 1302 is displayed at a first
magnification. In response to detecting a de-pinching gesture
3931/3933, decreasing portions ofthe digital image 1302 are
displayed at increasing magnifications compared to the magnification shown in FIG. 13A. For example, the portion ofthe
digital image 1302 shown in FIG.13B is smaller than and has
a higher magnification than the portion of the digital image
1302 shown in FIG. 13A.
In the example of FIG. 13B, the magnification exceeds a
predefmed magnification. Upon detecting termination of the
so gesture 3931/3933, a portion of the digital image 1302 is
displayed at the predefined magnification, as illustrated in
FIG. 13C.
FIG. 14 is a flow diagram illustrating a process 1400 of
executing a screen rotation command in accordance with
65 some embodiments. The process 1400 provides a simple
visual indicator to a user that the user has not provided a
sufficient gesture to initiate a 90° screen rotation command.

areas beyond opposite edges of the electronic document are
no longer displayed. For example, the areas 3955 are not
displayed in FIG. 10C.
FIGS. 10A-10C illustrate the display ofan electronic document at multiple magnifications in accordance with some 5
embodiments. While FIGS. 10A-10C illustrate displaying
these multiple magnifications in the context of a portable
multifunction device 100, displaying these multiple magnifications is not limited to portable multifunction devices. In the
example of FIGS. 10A-10C, the document is a web page 10
3912; FIGS.10A-10C(likeFIGS.SA-8D)illustrateanexemplary user interface for a browser in accordance with some
embodiments. An analogous user interface may be used to
display other types of electronic documents, such as digital
images or word processing, spreadsheet, email, or presenta- 15
tion documents.
In FIG. 10A, the web page 3912 is displayed at a first
magnification. The web page 3912 fills more thanthe allotted
screen area: only the left sides of block 7 (3914-7) and block
8 (3914-8) are displayed and only the top left corner ofblock 20
9 (3914-9) is displayed.
In response to detecting a pinching gesture 3951/3953
(FIG.10A), the web-page is displayed at a magnification less
than the first magnification, as shown in FIG. 10B. Ifa document length 3957 or a document width 3959 is entirely dis- 25
played while the gesture 3951/3953 is still detected, areas
3955 beyond opposite edges of the web page 3912 are displayed. Upon detecting termination ofthe gesture3951 " ,
the web page 3912 is displayedat a magnification whereinthe
areas 3955 are no longer displayed, as shown in FIG. 10C. 30
FIG. 11 is a flow diagram illustrating a process 1100 of
displaying an electronic document at multiple magnifications
in accordance with some embodiments. The process 1100
provides a simple visual indicator to a user that an electronic
document is being displayed at a
-m magnification 35
(e.g., the electronic document cannot be zoomed in and/or
magnified further).
The process 1100 is performed at a device with a touch
screen display. In some embodiments, the device is a portable
multifunction device. In some embodiments, the electronic
document is a web page (e.g., web page 3912, FIGS. 12A12C). In some embodiments, the electonic document is a
digital image (e.g., digital image 1302, FIGS. 13A-13C). In
some embodiments, the electronic document is a word pro45
cessing, spreadsheet, email or presentation document.
At least a first portion of the electronic document is displayed (1102) at a first magnification. A gesture is detected
(1104) on ornear the touch screen display corresponding to a
command to zoom in by a user-specified amount. In some
embodiments, the gesture is a de-pinching gesture (e.g., So
3931/3933, FIGS. 12A and 13A).
In response to detecting the gesture, decreasing portions of
the electronic document are displayed (1106) at increasing
magnifications. For example, in FIG. 12B a decreasedportion
of the web page 3912 is displayed at a higher magnification 55
than the portion in FIG. 12A, and in FIG. 13B a decreased
portion of the digital image 1302 is displayed at a higher
magnification than the portion in FIG. 13A.

If, upon detecting termination of the gesture, the magnification does not exceed a predefined magnification (1108-No),
the process 1100 is complete (1110).

If, however, upon detecting termination ofthe gesture, the
magnification exceeds a predefined magnification
(1108-Yes), a respectiveportion ofthe electronic document is
displayed (1112) at the predefined magnification. In the
examples of FIGS. 12B and 13B, the magnification exceeds a
predefined magnification. Upon detecting termination of the

APLNDC00022523

US 7,469,381 B2
33

34

The process 1400 is performed at a device with a touch
screen display. In some embodiments, the device is a portable

multifunction device.
A multifmger twisting gesture (e.g., 1506, FIG. 15A, or
1508, FIG.15C)is detected (1402) on ornearthe touch screen 5
display. The multifingertwisting gesture has a corresponding
degree of rotation. In some embodiments, the multifinger
twisting gesture includes gestures by two thumbs1604-L and
1604-R (FIGS. 16A and 16D)
If the corresponding degree of rotation exceeds a pre- 10
defined degree of rotation (1404-Yes), a 90° screen rotation
command is executed (1406). For example, the digital image
1502 of FIGS. 15A and 16A is otated from a portrait orientation to a landscape orientation, as shown respectively in
FIGS. 15B and 16B.
15
If the corresponding degree of rotation does not exceed a
predefined degree of rotation (1404-No), a screen rotation

command with an acute angle of rotation (i.e., less than90°)
is executed (1408). For example, the digital image 1502 of
FIGS. 15C and 16D is rotated by an acute angle, as shown 20
respectively in FIGS. 15D and 16E. Upon ceasing to detect
the multifingertwistinggesture, a screen rotation command is
executed (1410) with an angle of rotation opposite to the
acute angle (e.g., with the result shown in FIGS. 15E and
25
16F).
FIGS. 15A-15E illustrate rotating the display of an electronic document or other digital object in accordance with
some embodiments. While FIGS. 15A-15E illustrate display
rotation in the context ofa portable multifunction device 100,
display rotation is not limited to portable multifunction 30
devices. In the example of FIGS. 15A-15E, the electronic
document is a digital image 1502.
In FIGS. 15A and 15C, the digital image1502 is displayed
in a portrait orientation. A multifmger twisting gesture 1506
(FIG.15A)or1508(FIG.15C)isdetectedonthetouchscreen 35
display. The multifinger twisting gesture 1506 or 1508 has a
corresponding degree of rotation. In some embodiments, the
degree of rotation corresponds to a degree of rotation of an

ments, the first sense ofrotation is a clockwise rotation (e.g.,
FIG. 16C). The simultaneous rotation of the two thumbs has
a corresponding degree ofrotation.
In some embodiments, the sense ofrotation for each thumb
is detected by monitoring the change in orientation of the
contact area of the thumb with the touch screen display. For
example, if the contact area of the thumb is elliptical, the
change in the orientation of an axis of the ellipse may be
detected (e.g., from contact ellipse 1606-L in FIG. 16A to
contact ellipse 1608-L in FIG. 16B, as shown on an enlarged
portion of touch screen 112 in FIG. 16C). In some embodiments, the change in the orientation of the axis of the ellipse
determines the corresponding degree of rotation. In some
embodiments, at least some of a user's other fingers (i.e.,
fmgers other than thumbs 1604-L and 1604-R) support the
device 100 by contacting the backside of the device.
In some embodiments, the first sense of rotation is a counterclockwise rotation. For example, if thumb 1604-L is initiallyonthelowerleftsideoftouchscreen112(ratherthanthe
upper left side in FIG. 16A), thumb 1604-R is initially on the
upper right side of touch screen 112 (rather than the lower

right side in FIG.16A), and the thumbs are moved apart from
each other, then the sense of rotation detected by the touch
screen 112 will be counterclockwise for both thumbs.
If the corresponding degree of rotation exceeds a predefined degree of rotation, a 90° screen rotation command is
executed. For example, display of the digital image 1502 is
rotated from the portrait orientation of FIG. 16A to a landscape orientation in FIG. 16B.

If, however, the corresponding degree of rotation does not
exceedapredefineddegreeofrotation, a screenrotationcommand with an acute angle of rotation is executed. For
example, the digital image 1502 in FIG. 16D is rotated by an
acute angle, with the result shown in FIG. 16E. Once detec-

tion of the two thumbs 1604-L and 1604-R ceases, a screen
rotation command with an angle of rotation opposite to the

acute angle is executed, thereby restoring the digital image
1502 to a portrait orientation, as shown in FIG. 16F.

While FIGS. 6A-6D, BA-8D, 10A-10C, 12A-12C, 13Aaxis betweenthe contact points on the touch screen display of
the two fmgers in the multifinger gesture (e.g., an axis 40 13C, 15A-15E, and 16A-16F illustrate scrolling, translation,
between the center points or centroids of the contact regions
scaling, and rotation operations in the context of a portable
multifunction device 100, similar operations may be perof the two fmgers).
formed on any device with a touch-screen display, in accorIn the example of FIG. 15A, the multifinger twisting gesture 1506 has a corresponding degree ofrotationthat exceeds
dance with some embodiments. The device, such as device
a predefined degree of rotation. Thus, a 90° screen rotation 45 1700 below, may or may not be portable and the function or
functions performed by the device may vary.
command is executed, with the result that the digital image is
FIG. 17 is a block diagram illustrating a device 1700 with
displayedin a landscape orientation, as shown in FIG.15B. In
a touch -- display in accordance with some embodithe example of FIG. 15C, however, the multifmger twisting
ments. Device 1700 need not be portable. The device 1700
gesture 1508 has a corresponding degree ofrotation that does
not exceed a predefined degree of rotation. A screen rotation 50 typically includes one or more processing units (CPU's)
1710, one or more network or other ..
.. .ations intercommand with an acute angle ofrotation is executed, withthe
faces 1760, memory 1770, and one or more e
ication
result shown in FIG. 15D. Upon ceasing to detect the multibuses 1720 for interconnecting these components. The comfinger twisting gesture 1508, a screen rotation command with
an angle opposite to the acute angle is executed, with the
munication buses 1720 may include circuitry (sometimes
result that the portrait orientation ofthe digital image 1502 is ss called a chipset) that interconnects and controls communications between system components. The device 1700 includes
restored, as shown in FIG. 15E.
FIGS. 16A-16F illustrate an exemplary screen rotation
auserinterface 1730 comprisinga touch-screendisplay 1740.
gesture in accordance with some embodiments. While FIGS.
The user interface 1730 also may include a keyboard and/or
mouse (or other pointing device) 1750. Memory 1770
16A-16F illustrate this screen rotation gesture in the context
of a portable multifunction device 100, this screen rotation 60 includes high-speedrandom access memory, such as DRAM,
SRAM, DDR RAM or other random access solid state
gesture is not limited to portable multifunction devices. In the
memorydevices; andmay include non-volatile memory, such
example of FIGS. 16A-16F, this screen rotation gesture is
as one or more magnetic disk storage devices, optical disk
used to rotate the digital image 1502.
In FIG.16A, the device 100 displays the digital image 1502
storage devices, flash memory devices, or other non-volatile
in a portrait orientation. Simultaneous rotation oftwo thumbs 65 solid state storage devices. Memory 1770 may optionally
(e.g., 1604-L and 1604-R) in a first sense of rotation is
include one or more storage devices remotely located from
the CPU(s) 1710. In some embodiments, memory 1770 stores
detected on the touch screen display 112. In some embodi-

has iIC

ihm DIRE lemmassa

som HÆlü¾IGHM1

APLNDC00022524

US 7,469,381 B2
35

36

programs, modules, and data structures analogous to the pro5. The computer-implementedmethod of claim 1, wherein
grams, modules, and data structures stored inthe memory 102
the first directionis a vertical direction, a horizontal direction,
of portable multifunction device 100 (FIG. 1), or a subset
or a diagonal direction.
thereof. Furthermore, memory 1770 may store additional
6. The computer-implemented method ofclaim 1, wherein
programs, modules, and data structures (not shown) not 5 the electronic document is a web page.
present in the memory 102 of portable multifunction device
7. The computer-implemented method ofclaim 1, wherein
100.
the electronic document is a digital image.
Each of the above identified elements in FIG. 17 may be
8. The computer-implemented method of claim 1, wherein
stored in one or more of the previously mentioned memory
the electronic document is a word processing, spreadsheet,
devices. Each ofthe above identified modules corresponds to 10 email or presentation document.
a set of instructions for performing a function described
9. The computer-implemented method of claim 1, wherein
above. The above identified modules or programs (i.e., sets of
the electronic document includes a list of items.
instructions) need not be implemented as separate sofhvare
10. The computer-implemented method of claim 1,
programs, procedures or modules, and thus various subsets of
wherein the second direction is opposite the first direction.
these modules may be combined or otherwise re-arranged in 15
11. The computer-implemented method of claim 1,
various embodiments.1n some embodiments, memory 1770
wherein translating in the first direction prior to reaching an
may store a subset ofthe modules and data structures identiedge of the document has an associated speed of translation
fied above. Furthermore, memory 1770 may store additional
that corresponds to a speed of movement of the object.
modules and data structures not described above.
12. The computer-implemented method of claim 1,
The foregoing description, for purpose ofexplanation, has 20 wherein translating in the first direction is in accordance with
been describedwith reference to specific embodiments. Howa simulation of an equation of motion having friction.
ever, the illustrative discussions above are not intended to be
13. The computer-implemented method of claim 1,
exhaustive or to limit the invention to the precise forms diswherein the area beyond the edge of the document is black,
closed. Many modifications and variations are possible in
gray, a solid color, or white.
view of the above teachings. The embodiments were chosen 25
14. The computer-implemented method of claim 1,
and described in order to best explain the principles of the
whereinthe area beyond the edge ofthe document is visually
invention and its practical applications, to thereby enable
distinct from the document.
others skilled in the art to best utilize the invention and vari-

15. The computer-implemented method of claim 1,
ous embodiments with .-i,,- modifications as are suited to
the particular use contemplated.
3o wherein translating the document in the second direction is a
damped motion.
What is claimed is:
16. The computer-implemented method of claim 1,
1. A computer-implemented method, comprising:
wherein changing from translating in the first direction to
at a device with a touch screen display:
translating in the second direction until the area beyond the
displaying a first portion of an electronic document;
edge ofthe document is no longer displayed makes the edge
detecting a movement of an object on or near the touch
ofthe electronic document appearto be elastically attachedto
screen display;
an edge ofthe touch screen display or to an edge displayed on
the touch screen display.
in response to detecting the movement, translating the
electronic document displayed on the touch screen
17. The computer-implemented method of claim 1,
display in a first direction to display a second portion
wherein translating in the first direction prior to reaching the
of the electronic document, wherein the second poredge of the electronic document has a first associated transtion is different from the first portion;
lating distance that corresponds to a distance ofmovement of
in response to an edge of the electronic document being
the object prior to reaching the edge of the electronic docureached while translating the electronic document in
ment; and wherein displaying an area beyond the edge ofthe
the first direction while the object is still detected on 4, electronic document comprises translating the electronic
document in the first direction for a second associated transor near the touch screen display:
displaying an area beyond the edge of the document,
lating distance, wherein the second associated translating
and
distance is less than a distanceofmovement ofthe object after
reaching the edge of the electronic document.
displaying a third portion ofthe electronic document,
wherein the third portion is smaller than the first so 18. The computer-implemented method of claim 1,
wherein translating in the first direction prior to reaching the
portion; and
edge of the electonic document has a first associated transin response to detecting that the object is no longeron
lating speed that corresponds to a speed of movement of the
or near the touch screen display, translating the
object, andwherein displaying an area beyondthe edge ofthe
electronic document in a second direction until the
area beyond the edge of the electronic document is ss electronic document comprises translating the electronic
document in the first direction at a second associated transno longer displayed to display a fourth portion of
lating speed, whereinthe second ----tedtranslating speed
the electronic document, wherein the fourth poris slower than the first associated translating speed.
tion is different from the first portion.
19. A device, comprising:
2. The computer-implemented method ofclaim1, wherein
the first portion of the electronic document, the second por- so a touch screen display;
tion of the electronic document, the third portion ofthe elec-

one or more processors;

tronic document, and the fourth portion of the electronic
document are displayed at the same magnification.
3. The computer-implemented method ofclaim 1, wherein

memory; and
one or more programs, wherein the one or more programs

the movement of the object is on the touch screen display,

4. The computer-implemented method ofclaim 1, wherein
the object is a finger.

65

are stored in the memory and configured to be executed
by the one or more processors, the programs including:

instructions fordisplaying a first portionofan electronic
document;

APLNDC00022525

US 7,469,381 B2
37

38

instructions for detecting a movement ofan object on or

display a first portion of an electronic document;

near the touch screen display;
instructions for translating the electronic document displayed on the touch screen display in a first direction
to display a second portion of the electronic docu- 5
ment, whereinthe second portion is different from the
first portion, in response to detecting the movement;
instructions for displaying an area beyond an edge ofthe
electronic document and displaying a third portion of
the electronic document, wherein the third portion is to
smaller than the first portion, in response to the edge
ofthe electronic document being reachedwhile translating the electronic document in the first direction
while the object is still detected on or near the touch
screen display; and
15
instructions for translating the electronic document in a

detect a movement ofan object on or near the touch screen
display;
translate the electronic document displayed on the touch
screen display in a first direction to display a second
portion ofthe electronic document, wherein the second
portion is different from the first portion, in response to
detecting the movement
display an area beyond an edge ofthe electronic document
and display a third portion of the electronic document,
whereinthethird portionis smaller than the first portion,
if the edge of the electronic document is reached while
translating the electronic document in the first direction
while the object is still detected on or near the touch
screen display; and

second direction until the area beyondthe edge ofthe

electronic document is no longer displayed to display
a fourth portion of the electronic document, wherein

the fourth portion is difTerent from the first portion, in 20
response to detecting that the object is no longer on or

near the touch screen display.
20. A computer readable storage medium having stored
therein instructions, which when executed by a device with a
touch screen display, cause the device to:

translate the electronic document in a second direction

until the area beyond the edge of the electronic document is no longer displayedto display a fourthportionof
the electronic document, wherein the fourth portion is
different from the first portion, in response to detecting

that the object is no longer on or near the touch screen
display.

*****

Copy rovided by uur av from the PIRS Ima e Database on 04/25/2011

APLNDC00022526

UNITED STATES PA i

i

Nu •<~»EMARKOFFICE

CERTIFICATE OF COur siscCTION
PAim i NO.
APPLICATION NO.
DATED
INVENTOR(S)

: 7,469,381 B2
: 11/956969
: December 23, 2008
: Ording

Page 1 of 1

It is certified that error appears in the above-identified patent and that said Letters Patent is
hereby corrected as shown below:

Column 38, line 8, please insert --; --after movement.

Signed and Sealed this

Seventeenth Day of February, 2009

JOHN DOLL
Acting Director ofthe United States Patent and Trademark Ofice

copy provided by USPTO from the PlHS Image Database on 04/25/2011

APLNDC00022527