Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 946

EXHIBITS re #943 Declaration in Support,, Exhibits to Arnold Declaration (Ex. 84 (Part 1)) filed bySamsung Electronics America, Inc., Samsung Electronics Co. Ltd., Samsung Telecommunications America, LLC. (Attachments: #1 Ex. 84 (Part 2), #2 Ex. 84 (Part 3), #3 Ex. 84 (Part 4), #4 Ex. 84 (Part 5), #5 Ex. 85 (Part 1), #6 Ex. 85 (Part 2), #7 Ex. 85 (Part 3), #8 Ex. 86 (Part 1), #9 Ex. 86 (Part 2), #10 Ex. 86 (Part 3), #11 Ex. 86 (Part 4))(Related document(s) #943 ) (Maroulis, Victoria) (Filed on 5/18/2012)

U.S. Patent

Nov. 30, 2010

Sheet 33 of 37

US 7,844,915 B2

3200
US

CACHE
(OPTIONAL)

02

MICROPROCEmoun

3204
MEMORY
3206

3 10
DISPLAY
CONTROLLER
AND
DISPLAY
DEVICE
(OPTIONAL)

3212

CONTROLLER(S)

«

3216
3214 ,

llO DEVICE(S)
(E.G., KEYBOARD,
CURSOR CONTROL

SENSOR(S)

FOR USER
ACTIVITY

DEVICE, NETWORK
liv mNFACE)

FIG. 32

Copv provided by USPTO from the PIRS Image Database on 04/25/2011

APLNDC00025415

TEX 7
U.S. Patent

Nov. 30, 2010

Sheet 34 of 37

US 7,844,915 B2

3306

3308

3302
3304

FIG. 33A

eam DIDO I..maman rintakaan saan finl94198111

APLNDC00025416

U.S. Patent

Nov. 30, 2010

Sheet 35 of 37

US 7,844,915 B2

3358

3356

3352
3354

FIG. 338

APLNDC00025417

U.S. Patent

Nov. 30, 2010

Sheet 36 of 37

US 7,844,915 B2

3362
3366

3360

FIG. 33C

pagay an

Hinwi ho i ERMhA RIRR Imaria Omtshame an

APLNDC00025418

U.S. Patent

Nov. 30, 2010

Sheet 37 of 37

US 7,844,915 B2

3400

CONSTRUCT A HIERARCHY OF VIEWS
OPERATING ON TOP OF A HIERARCHY OF
LAYERS
3402

PROVIDE ACCESS TO THE HIERARCHY OF
VIEWS WITHOUT PROVIDING ACCESS TO THE
HIERARCHY OF LAYERS
3404

FIG. 34

APLNDC00025419

1

2

US 7,844,915 B2
At least certain embodiments of the present disclosure
include an environment with user interface software interacting with a software application. A method for operating
through an application programming interface (API) in this
s environment includes transferring a set bounce call. The
OF THE DJu omURE
method further includes setting at least one ofmaximum and
minimum bounce values. The setbounce call causes a bounce
This disclosure relates to application programming interof a scrolled region in an opposite direction of a scroll based
faces that provide scrolling operations.
on a regionpast an edge ofthe scrolled region being visible in
lo a display region at the end of the scroll.
CO•< PROGRAM LISTING
APPLICATION PROG A ---G
., . ..-ACES FOR SCROLLING
OF ATIONS

At least certain embodiments of the present disclosure

include an environment with user interface software interactA portion of the disclosure of this patent document coning with a software application. A method for operating
tains material which is subject to copyright protection. The
through an application programming interface (API) in this
copyright owner has no objection to facsimile reproduction
by anyone ofthe patent document or the patent disclosure, as 15 environment includes transferringa rubberband call. Rubberbanding a scrolled region within a display region occurs by a
it appears in the Patent & Trademark Office patent file or
predetermined
displacement when the scrolled
records, but otherwise reserves all copyright rights whatsoregion exceeds a display edge. The method further includes
ever
transferring an edge rubberband call to set displacement valApplicant has submitted herewith ComputerProgram List20 ues for at least one edge of the display (e.g., top and bottom
ings which are included as Appendix A, attached.
edges, left and right edges).
At least certain embodiments of the present disclosure
- -GROUNDOFTHED t•nu
include gesture operations for a display of a device. The
gesture operations include performing a scaling transform
AnAPI is a source code interface that a computer systemor
program library provides in order to support requests for 26 such as a zoom in or zoom out in response to a user input
having two or more input points. The gesture operations also
services from a soûware application. An API is specified in
include performing a rotation transform to rotate an image or
terms ofa programming language that can be interpretative or
view in response to a user input having two or more input
compiled when an application is built, rather than an explicit
points.
low level description of how data is laid out in memory. The
At least certain embodiments of the present disclosure
software that provides the functionality described by an API
include a method for performing animations for a display of
is said to be an implementation of the API.
a device. The method includes starting at least one animation.
Various devices such as electronic devices, computingsysThemethod furtherincludes determiningthe progress ofeach
tems, portable devices, and handheld devices have software
animation. The method further includes completing each aniapplications. The API interfaces between the software applimationbased on a single timer. The single timer can be based
cations and user interface software to provide a user of the
on a redraw interval of the display hardware.
device with certain features and operations.Ausermay desire
Various devices which perform one or moæ ofthe foregocertain operations such as scrolling, selecting, gesturing, and
ing methods and machine readable media which, when
animating operations for a display of the device.
executedbyaprocessingsystem,causetheprocessingsystem
Scrolling is the act of sliding a directional (e.g., horizontal 40 to perform these methods, are also described.
or vertical) presentation ofcontent, such as text, drawings, or
Other methods, devices and machine readable media are
images, across a screen or display window. In a typical
also described.
graphical user interface, scrolling is done with the help of a
scrollbar or using keyboard shortcuts often the arrow keys.
-- DES. - ON OF THE DRAWINGS
Gesturing is a type of user input with two or more input 45
points. Animating operations include changing content
The disclosure is described by way of example with referwithin a given time period.
ence to the accompanying drawings, wherein:
The various types of devices may have a limited display
FIG. 1 is flow chart of a method for responding to a user
size, user interface, software,API interface and/orprocessing
input of a data processing device;
capability which limits the ease of use of the devices. User so
FIG. 2 illustrates details of an application programming
interfaces of devices implement APIs in order to provide
interface in flow chart form according to certain teachings of
requested functionality and features. These user interfaces
the present disclosure;
can have difficulty interpreting the wi- types of user
FIG. 3 illustrates details of an application programming
inputs and providing the intended functionality associated
with the user inputs.
55 interface in flow chart form according to certain teachings of
the present disclosure;
ST 71***^ V OF THE DESCRIPTION
FIG. 4 is a schematic diagram illustrating an embodiment
ofuser interface of a portable electronic device 400 having a
touch-sensitive display 408;
At least certain embodiments of the present disclosure
includeoneormoreapplicationprogmmminginterfacesinan so FIG. 5A-5C illustrate at least some embodiments of user
interface of a portable electronic device 400 having a touchenvironment with user interface software interacting with a
sensitive display;
software application. Various function calls or messages are
FIG. 6A-6D illustrate the scrolling of a list of items to a
transferred via the application progomming interfaces
terminus ofthe list, at which point an area beyond the termibetween the user interface software and software applications. Example application programming interfaces transfer 65 nus is displayed and the list is then scrolled in an opposite
direction until the area beyond the terminus is no longer
function calls to implement scrolling, gesturing, and animatdisplayed, in accordance with some embodiments;
ing operations for a device.

en.- menuland in, IIRDTn fram tha ÞIRR Imann Ontahane on 04/25/2011

APLNDC00025420

US 7,844,915 B2

3

4

FIG. 32 shows another example of a device in accordance
FIG. 7 illustrates details of an application programming
with one embodiment of the present disclosure;
interface in flow chart form according to certain teachings of
FIG. 33A is a perspective view of a device in a first conthe present disclosure;
figuration (e.g. in a laptop configuration) in accordance with
FIG. 8 illustrates first and second scroll angles for locking
a scroll of a display of a device in a horizontal or vertical 5 one embodiment of the present disclosure;
FIG. 33B is a perspective view ofthe device ofFIG. 33A in
direction according to certain teachings ofthe present discloa second configuration (e.g. a transition configuration) in
sure;
accordance with one embodiment of the present disclosure;
FIG. 9 illustrates details of an application programming
FIG.33Cis aperspectiveview ofthedeviceofFIG. 33A in
interface in flow chart form according to certain teachings of
lo a third configuration (e.g., a tablet configuration) in accorthe present disclosure;
dance with one embodiment of the present disclosure; and
FIG. 10 illustrates details of an application programming
FIG. 34 illustrates details of an application programming
interface in flow chart form according to certain teachings of
interface in flow chart form according to certain teachings of
the present disclosure;
the present disclosure.
FIG. 11 illustrates details of an application programming
interface in flow chart form according to certain teachings of is
DETAILED D- <> = ION
the present disclosure;
FIG. 12 illustrates details of an application programming
Various embodiments and aspects ofthe disclosure will be
interface in flow chart form according to certain teachings of
described with reference to details discussed below, and the
the present disclosure;
FIG. 13 illustrates details of an application programming 20 accompanying drawings will illustrate the -I- embodiments. The following description and drawings are illustrainterface in flow chart form according to certain teachings of
tive of the disclosure and are not to be construed as limiting
the present disclosure;
the disclosure. N--- specific details are described to
FIG. 14 illustrates details of an application programming
provide a through understanding of various embodiments of
interface in flow chart form according to certain teachings of
the present disclosure;
25 the present disclosure. However, in certain instances, wellknown or conventional details are not described in order to
FIG. 15 illustrates a display of a device having a scaling
provide a --I- discussion of embodiments of the present
transform of a view:
disclosure.
FIGS. 16A and 16B illustrate a display of a device with a
Some portions of the detailed descriptions which follow
view having a first and a second scaling factor;
FIG. 16C illustrates changing a view from a scale factor of 30 are presentedin terms ofalgorithms which include operations
on data stored within a computer memory. An algorithm is
2x to a scale factor of 1x in at least some embodiments ofthe
generally a self-consistent sequence of operations leading to
present disclosure;
a desired result. The operations typically require or involve
FIG. 17 illustrates details of an application programming
physical manipulations of physical quantities. Usually,
interface in flow chart form according to certain teachings of
though not necessarily, these quantities take the form ofelecthe present disclosure;
36
trical ormagnetic signals capable ofbeing stored, transferred,
FIG. 18 illustrates details of an application programming
combined, compared, and otherwise manipulated. It has
interface in flow chart form according to certain teachings of
proven convenient at times, principally for reasons of comthe present disclosure;
mon usage, to refer to these signals as bits, values, elements,
FIG. 19 is flow chart of a method for animating views
symbols, characters, terms, numbers, or the like.
displayed on a display of a device;
It should be borne in mind, however, that all of these and
FIG. 20 is flow chart of a method for animating views
similar terms are to be associated with the appropriate physidisplayed on a display of a device;
cal quantities and are merely convenient labels applied to
FIG. 21 illustrates details of an application programming
these quantities. Unless specifically stated otherwise as
interface in flow chart form according to certain teachings of
apparent from the following discussion, it is appreciated that
the present disclosure;
throughout the description, discussions utilizing terms such
FIGS.22Aand 22B illustrate synchronizingthe resizing of
as "processing" or "computing" or "calculating" or "deterwindows of a display of a device;
mining" or "displaying"orthe like, can refer to the action and
FIG. 23 illustrates a method for switching ownership of a
processes of a data processing system, or similar electronic
view of an application displayed on a display of a data prodevice, that manipulates and transforms data represented as
cessing device;
physical (electronic) quantities within the system's registers
FIG. 24 illustrates a method for memory management of a
and
into other data similarly represented as physiview of an application displayed on a display of a device;
cal quantities within the system's -- es or registers or
FIGS. 25A and 258 illustrate a data structure having a
other such information storage, transmission or display
hierarchy of layers with a layer being associated with a view;
devices.
FIG. 26 illustrates a method for compositing media and
The present disclosure can relate to an apparatus for pernon-media content of user interface for display on a device;
forming one or more ofthe operations described herein. This
FIG. 27 illustrates a data structure or layer tree having a

apparatus may be specially constructed for the required pur-

poses, or it may comprise a general purpose computer selechierarchy of layers;
FIG. 28 is a perspective view ofa device in accordancewith so tively activatedorreconfiguredbya computerprogram stored
one embodiment of the present disclosure;
in the computer. Such a computer program may be stored in a
machine (e.g. computer) readable storage medium, such as,
FIG.29isaperspectiveviewofadeviceinaccordancewith
but is not limited to, any type of disk including floppy disks,
one embodiment ofthe present disclosure;
optical disks, CD-ROMs, and magnetic-optical disks, readFIGS. 30A and 30B illustrate a device 3070 according to
65 only memories (ROMs), random access memories (RAMs),
one embodiment of the disclosure;
erasable programmable ROMs (EPROMs), electrically erasFIG. 31 is a block diagram of a system in which embodiableprogrammableROMs (EEPROMs), flash memory, magments of the present disclosure can be implemented;

APLNDC00025421

US 7,844,915 B2

5

6

The portable media player may include a media selection
netic or optical cards, orany type ofmedia suitableforstoring
device, suchas a clickwheel input device on an iPod® or iPod
electronic instructions, and each coupled to a bus.
Nano® media player from Apple Computer, Inc. of CuperA machine-readable medium includes any mechanism for
tino, Calif., a touch screen input device, pushbutton device,
storing information in a form readable by a machine (e.g., a
computer). For example, a machine-readable medium 5 movable pointing input device or other input device. The
media selection device may be usedto select the media stored
includes read only memory ("ROM"); random access
on the storage device and/or the remote storage device. The
memory ("RAM"); magnetic disk storage media; optical storportable media player may, in at least certain embodiments,
age media; flash memory devices; etc.
include a display device which is coupled to the media proAt least certain embodiments of the present disclosure
includeoneorapplicationprogramminginterfacesinanenvi- 10 cessing system to display titles or other indicators of media
being selected through the input device and being presented,
ronment with user interface software interacting with a softeither through a speaker or earphone(s), or on the display
ware application. Various function calls or messages are
device, or on both display device and a speaker or ear
transferred via the application programming interfaces
phone(s). In some embodiments, the display device and input
between the user interface software and software applications. Transferringthe functioncalls ormessages may include 15 device are integrated while in other embodiments the display
device and input device are separate devices. Examples of a
issuing, initiating, invoking or receiving the function calls or
portable media player are described in published U.S. patent
messages. Example application programming interfaces
application Nos. 2003/0095096 and 2004/0224638, both of
transfer function calls to implement scrolling, gesturing, and
which are incorporated by reference.
animating operations for a device having a display region. An
API may also implement functions having parameters, vari- 20 Embodiments of the disclosure described herein may be
part of other types of data processing systems, such as, for
ables, or pointers. An API may --i,. parameters as disexample, entertainment systems or personal digital assistants
closed or other combinations ofparameters. In additionto the
(PDAs), or general purpose computer systems, or special
APIs disclosed, otherAPIs individually orin combinationcan
purpose computer systems, or an embedded device within
perform similar functionality as the disclosed APIs.
The display region is a form of a window. A window is a 25 another device, or cellular telephones which do not include
media players, or multi touch tablet devices, or other multi
display region which may not have a border and may be the
touch devices, or devices whichcombine aspects or functions
entire display region or area of a display. In some embodiof these devices (e.g., a media player, such as an iPod®,
ments, a display region may have at least one window and/or
combined with a PDA, an entertainment system, and a celluat least oneview (e.g., web, text, or image content).Awindow
may have at least one view. The methods, systems, and appa- 30 lar telephone in one device). In this disclosure, electronic
devices and consumer devices are types of devices.
ratuses disclosed can be implemented with display regions,
FIG. 1 is flow chart of a method for responding to a user
windows, and/or views.
input of a device. The method 100 includes receiving a user
At least certain embodiments of the present disclosure
input at block 102. The user input may be in the form of an
include scrolling operations for scrolling a display of a
device. The scrolling operations include bouncing a scrolled 35 input key, button, wheel, touch, or other means for interacting
with the device. The method 100 further includes creating an
region in an opposite direction of a scroll when a scroll
event object in response to the user input at block 104. The
completes, rubberbanding a scrolled region by a predetermethod 100 further includes determining whether the event
mined -----i-------- displacement when the scrolled region
object invokes a scroll or gesture operation at block 106. For
exceeds a display edge, and setting a scrolling anglethat locks
40 example, a single touch that drags a distance across a display
the scroll in a horizontal or vertical direction.
ofthe device may be interpreted as a scroll operation. In one
At least certain embodiments of the present disclosure
embodiment,atwoormorefmgertouchofthedisplaymaybe
include gesture operations for a display of a device. The
interpreted as a gesture operation. In certain embodiments,
gesture operations include performing a scaling transform
determining whether the event object invokes a scroll or gessuch as a zoom in or zoom out in response to a user input
having two or more input points. The gesture operations also 45 ture operation is based on receiving a drag user input for a
certain time period. The method 100 further includes issuing
include performing a rotation transform to rotate an image or
at least one scroll or gesture call based on invoking the scroll
view in response to a user input having two or more input
or gesture opemtion at block 108. The method 100 further
points.
includes responding to at least one scroll call, if issued, by
At least certain embodiments of the present disclosure
include a method for performing animations for a display of so scrolling a window having a view (e.g., web, text, or image
content) associated with the event object based on an amount
a device. The method includes starting at least one animation.
ofa scroll with the scroH stoppedat a predetermined position
The method further includes determiningthe progress ofeach
in relation to the user input at block 110. For example, an
animation. The method further includes completing each aniinput may end at a certain position on a display ofthe device.
mation based on a single timer. The single timer can be based
on a redraw interval of the display hardware.
55 The scrolling may continue until reaching a predetermined
position in relation to the last input received from the user.
At least certain embodiments ofthe disclosure may be part
The method 100 further includes responding to at least one
of a digital media player, such as a portable music and/or
gesture call, ifissued, by changing a view associated with the
video media player, which may include a media processing
event object based on receiving a plurality of input points in
system to present the media, a storage device to store the
media and may further include a radio frequency (RF) trans- 60 the form of the user input at block 112.
In certain embodiments of the present disclosure scroll
ceiver (e.g., an RF transceiver for a cellular telephone)
operations include attaching scron indicators to a content
coupled with an antenna system and the media processing
system. In certain embodiments, media stored on a remote

edge of a display. Alternatively, the scroll indicators can be

attached to the display edge. In some embodiments, user
storage device may be transmitted to the media player
through the RF transceiver. The media may be, for example, 65 input in the form of a mouselfmger down causes the scroll
indicators to be displayed on the display edge, content edge,
one or more of music or other audio, still pictures, or motion
orwindow edge ofthe scrolled region. If a mouselfmger up is
pictures.

annu nrnvidad hv usPTO from the PIRS Imaae Database on 04/25/2011

APLNDC00025422

US 7,844,915 B2
7
In certain embodiments ofthe present disclosure, transferthen detected, the scroll indicators are faded out from the
ring the rubberband call is either one of issuing, initiating,
display region, content edge, or window edge ofthe scrolled
invoking or receiving the rubberband call.
region.
FIG. 4 is a schematic diagram illustrating an embodiment
In certain embodiments of the present disclosure, gesture
operations include responding to at least one gesture call, if 5 ofuser interface ofa portable electronic device 400 having a
touch-sensitive display 408. The display 408 may include a
issued, by rotating a view associated with the event object
window 410. The window 410 may include one or more
basedonreceiving a plurality ofinput points inthe form ofthe
displayed objects, such as information objects 412-1 to 412user input. Gesture operations may also include scaling a
4. In an exemplary embodiment, the information objects 412
view associated with the event object by zooming inor zoom10 may correspond to contact information for one or more indiing out based on receiving the user input.
viduals in a list ofitems. The displayed objects may be moved
In some embodiments, a device includes a display region
in response to detecting or determining movement 414 of a
having multiple views or windows. Each window may have a
point of contact with the display, such as that associated with
multiple views including superviews and subviews. It is necone or more digits 416 ofa user (whichare not drawnto Scale
essary to determine which window, view, superview, or subview is contacted by a user input in the form of a mouse up, 15 in FIG. 4). In some embodiments, movement ofthe displayed
objects may be accelerated in response to detecting or determouse down, or drag, etc. An API can set various modes for
mining accelerated movement ofthe point of contact. While
making this determination. In one embodiment, a pass mode
embodiment 400 includes one window 410, in other embodisends mouse down, mouse up, and drag inputs to the nearest
ments there may be two or more display windows. In addisubview. In another embodiment, an intercept on drag mode
sends a drag input to the superview while mouseup and down 2o tion, while embodiment 400 illustrates movement 414 in a
particular direction, in other embodiments -..__t of the
inputs are sent to the subview. In another embodiment, an
displayedobjects may be inresponse to movement 414 in one
intercept mode sends all drag, mouse up and down inputs to
or more other directions, or in response to a scalar (i.e., a
the superview. The superview may be scroller software operdetermined or detected movement independent of the direcating as a subclass of a view software. The subview may be
view software operating as a subclass of the user interface 25 tion).
FIGS. 5A-5C illustrate the scrolling of a list of items on a
software.
FIG. 2 illustrates details of an application programming
device to a terminus of the list, at which point one or more
interface in flow chart form according to certain teachings of
displayed items at the end ofthe list smoothly bounce offthe
the present disclosure. The application programming interendofthedisplay,reversedirection,andthenoptionallycome
face operates in an environment with user interface software 30 to a stop. FIG. 5A is a schematic diagram illustrating an
interacting with a software application in order to provide a
embodiment ofuser interface of a portable electronic device
bounce operation. The method 200 for providing a bounce
400 having a touch-sensitive display. One or more displayed
operation includes transferring a set bounce call at block 202.
objects, such as infornation object 412-1 may be a distance
The method 200 further includes setting at least one ofmaxi512-1fromaterminus 514 ofthe list ofitems which is an edge
mum and
bounce values at block 204. The mini- 35 ofa scrolled region and may be moving with a velocity 510-1
mum and - um bounce values may be associated with at
while the list is being scrolled. Note that the terminus 514 is
least one edge ofa window that has received a user input. The
a virtual boundary associated with the displayed objects, as
method 200 further includes causing a bounce of a scrolled
opposed to a physical boundary associated with the window
region in an opposite direction of a scroll based on a region
410 and/or the display 408. As illustrated in FIG. 5B, when
past the scrolled region being visible in a display region at the 4o the one or more displayed objects, such as the information
end of the scroll at block 206. The scrolled region may be a
object 412-1, reach or intersect with the terminus 514, the
content region.
movement corresponding to the scrolling may stop, i.e., the
In certain embodiments ofthe present disclosure, transferscrolling velocity may be zero at an instant in time. As illusring the set bounce call is either one of issuing, initiating,
trated in FIG. SC, the one or more displayed objects, such as
invoking or receiving the set bounce call.
45 the information 412-1, may subsequently ...._ direction.
FIG. 3 illustrates details of an application programming
At a time after the intersection with the terminus 514, the
interface in flow chart form according to certain teachings of
information object 412-1 may have velocity 510-2 and may
the present disclosure. The application programming interhe a distance 512-2 from the terminus 514. In some embodiface operates in an environment with user interface software
ments, the magnitude of velocity 510-2 may be less than the
interacting with a software application in order to provide a so magnitude of velocity 510-1 when the distance 512-2 equals
rubberband operation. The method 300 for providing a rubthe distance 512-1, i.e., the motion of the one or more disberband operation includes transferring a rubberband call to
played objects is damped after the scrolling list reaches and
cause rubberbanding a scrolled region displayed within a
"bounces" at its terminus.
display at block 302. The method 300 further includes transIn at least some embodiments ofthe present disclosure, the
ferring an edge rubberband call to set displacement values for ss method 200 performs the bounce operations described in
at least one edge ofthe display at block 304. In some embodiFIGS. 5A-5C. The bounce call transferred at block 202 determents, the displacement values are set for top and bottom
mines whether a bounce operation is enabled. The maximum
edges, left and right edges, or all edges.
and minimum bounces values determine the amount of
Rubberbandingascrolledregionaccordingtothemethod
bouncingofthescrolledregioninanoppositedirectionoftbe
300 occurs by a predeterminedmaximum displacement value so scroll.
when the scrolled region exceeds a display edge of a display
PIGS. 6A-6D illustrate the scrolling of a list of items to a
ofa device based on the scroll. If a user scrolls content ofthe
terminus of the list, at which point an area beyond the termidisplay making a region past the edge ofthe content visible in
nus is displayed and the list is then scrolled in an opposite
the display, then the displacement value limits the
direction until the area beyond the terminus is no longer
amount for the region outside the content. At the end of the 65 displayed, in accordance with some embodiments. The rubscroll, the content slides back making the region outside of
berbandoperationofmethod300is illustrated inthe example
the content no longer visible on the display.
ofFIGS. 6A-6D with the listed items being email messages.

re--.. ..-..iAaA har linDTF3 fewam the DIRA imann fininkaan on OA

ENOli

APLNDC00025423

US 7,844,915 B2
9

10

transferring a directional scroll angle call to determine if
FIGS. 6A-6D illustrate an exemplary user interface 3500A
directional scrolling is enabled at block 702. The method 700
for managing an inbox in accordance with some embodifurther includes transferring a directional scroll angle call to
ments. An analogous user interface may be used to display
set a scroll angle for locking the scrolling in at least one of a
and manage other mailboxes (e.g., drafts, sent, trash, personal, etc.). In addition, other types of lists are possible, 5 vertical or a horizontal direction at block 704. The method
700 further includes locking the scrolling in the horizontal
including but not limited to lists of instant message
.direction if a user input forms an angle with a horizontal
sations, favorite phone numbers, contact information, labels,
directionthatislessthanorequaltoafirstscrollangleatblock
email folders, email addresses, physical addresses, ringtones,
706. The method 700 further includes locking the scrolling in
or album names.
If the list of emails fills more than the allotted screen area, 10 the vertical direction if a user input forms an angle with the
vertical direction that is less than or equal to a second scroll
the user may scroll through the emails using vertically
angle at block 708.
upward and/or vertically downward swipe gestures on the
In certain embodiments, a user input in the form of a drag
touch screen. In the example of FIG. 6A, a portion ofa list of
forms an angle with the horizontal direction that is less than
emails is displayed in the screen area, including a top displayedemail 3530 from BruceWalker and abottom displayed 15 the first scroll angle. In this case, the userp.-.....bly intends
to scroll in the horizontal direction. The scrolling will be
email 3532 from Kim Brook. A user performs a vertically
locked in the horizontal direction until the user input exceeds
downward swipe gesture 3514 to scroll toward the top ofthe
the first scroll angle. A second scroll angle may be used for
list. The vertically downward gesture 3514 need not be
locking the user input in the vertical direction. The second
exactly vertical; a substantially vertical gesture is sufficient.
In some embodiments, a gesture within a predetermined 20 scroll angle may be set equal to the first scroll angle.
FIG.8 illustrates first and second scroll angles for locking
angle of being perfectly vertical results in vertical scrolling.
a scroll of a display of a device in a horizontal or vertical
As a result of detecting the vertically downward gesture
3514, in FIG. 6B the displayed emails have shifted down,
direction. The horizontal direction 802 and vertical direction
804 are in reference to a window or a display of a device. As
such that the previous bottom displayedemail 3532 from Kim
Brook is no longer displayed, the previous top displayed 25 discussed in the method 700, a user input such as a drag
movement forming anangle withthe horizontal direction 802
email 3530 from Bruce Walker is now second from the top,
less than or equal to the first scrolling angle 806 or 808 will
and the email 3534 from Aaron Jones, which was not displayed in FIG. 6A, is now displayed at the top of the list.
lock the user input in the horizontal direction. In a similar
manner, a user input forming an angle with the vertical direcIn this example, the email 3534 from Aaron Jones is the
first email in the list and thus is the terminus ofthe list. Upon 30 tion 810 less than orequal to the second scrolling angle 810 or
812 will lock the user input in the vertical direction. The first
reaching this email 3534, in response to continued detection
of the vertically downward gesture 3514, an area 3536 (FIG.
and second scrolling angles may be set at the same angle or at
different angles as well. For example, the first and second
6C) above the first email 3534 (i.e., beyond the terminus of
scrolling angles may be set at 25 degrees. A user input less
the list) is displayed. In some embodiments, the area displayed beyond the terminus of the list is visually indistinct 35 than or equal to 25 degrees with respect to the horizontal or
fromthebackgroundofthelist.InFIG.6C,boththearea3536
vertical direction will lock the scrolling in the appropriate
direction.
and thebackground ofthe emails (e.g., emails 3534 and3530)
In some embodiments, the horizontal and vertical locking
are white and thus are visually indistinct.
angles can be determined in part by the aspect ofthe content.
Once vertically downward gesture 3514 is complete, such
that a corresponding object is no longer detected on or near 40 For example, content in the form of a tall page may ....L - a
largervertical locking angle compared to the horizontal lockthe touch screen display, the list is scrolled in an opposite
direction until the area 3536 is no longer displayed. FIG. 6D
ing angle.
FIG. 9 illustrates details of an application programming
illustrates the result ofthis scrolling inthe opposite direction,
interface in flow chart form according to certain teachings of
the email 3534 from Aaron Jones is now displayed at the top
of the screen area allotted to the list and the area 3536 is not 45 the present disclosure. The application programming interdisplayed.
face operates in an environment with user interface software
interacting with a software application in order to provide a
In the example of FIGS. 6A-6D, a vertically downward
deceleration scroll operation. The method 900 for providing
gesture resulted in display of an area beyond the first item in
the deceleration scroll operation includes transferring a
the list. As described in FIG. 3, the values for the predetermined -2-- displacement (e.g., display of an area so deceleration scroll call to set a deceleration factor for a drag
user input at block 902. The method 900 further includes
beyondthe first item in the list) are set at block 304 fortop and
slowingthe scroll to a stop based onthe speed ofthe drag user
bottom edges or at block 306 for all edges of the window.
input and the deceleration factor at block 904.
Similarly, a vertically upward gesture may result in display
In certain embodiments, a user input in the form of a drag
of an area beyond the last item of the list, if the vertically
upward gesture continues once the list has been scrolledto the 55 invokes a scroll operation for a certain time period. The user
input has a certain speed. The scroll ofthe scrolled region of
last item. The last item may be considered a terminus of the
list, similar to the first item. As discussed above, the gesture
a window or a display region ofa display of a device will be
stopped after the user input stops by applying a deceleration
need not be exactly vertical to result in vertical scrolling; a
factor to the speed of the user input during the drag movegesture within a predefined range of angles from perfectly
vertical is sullicient.
60 ment.
FIG. 10 illustrates details of an application programming
FIG. 7 illustrates details of an application programming
interface in flow chart form according to certain teachings of
interface in flow chart form according to certain teachings of
the present disclosure. The application programming interthe present disclosure. The application programming interface operates in an environment with user interface sonware
face operates in an environment with user interface software
interacting with a software application in order to provide a es interacting with a software application in order to provide a

directional scrolling operation. The method700 foroperating
through an application programming interface (API) includes

scroll hysteresis operation. The method 1000 for providing
the scroll hysteresis operation includes transferring a scroll

I

APLNDC00025424

US 7,844,915 B2
11

12

hysteresis call to determine whether a user input invokes a
scroll at block 1002. The method 1000 further includes set-

or display. A mask may merely permit certain changes while
limiting or not permitting other changes. Events of all kinds
ting a hysteresis value for determining whether a user input
come into the application via a graphics framework. They are
invokes a scroll at block 1004.
enqueued, collaleced if necessary and dispatched. If the
In certain embodiments, a user input in the form of a drag s events are system level events (e.g., application should susover a certain distance across a display or window within a
pend, device orientation has chanted, etc) they are routed to
the application having an instance of a class of the user
display of a device invokes a scroll operation. The hysteresis
value determines the certain distance which the user input
interface software. If the events are hand events based on a
must drag across the display or window prior to invoking a
user input, the events are routedto the window they occurred
scroll operation. A user input that does not drag the certain 10 over. The window then routes these events to the appropriate
control by calling the instance's mouse and gesture methods.
predetermined distance will not invoke a scroll operation and
The control that
a mouse down or mouse entered
may be considered a mouse up or down input or other type of
function will continue to get all future calls until the hand is
input.
FIG. 11 illustrates details of all application programming

lifted. If a second finger is detected, the gesture methods or

interface in flow chart form according to certain teachings of 15 functions are invoked. These functions may include start,
change, and end gesture calls. The control that receives start
the present disclosure. The application programming intergesture call will be sent all future change gesture calls until
face operates in an environment with user interface software
interacting with a software application in order to attach a
the gesture ends.
scroll indicator to a scroll region edge or a window edge ofa
FIG. 13 illustrates details of an application programming
device. In some embodiments, the scroll region edge is asso- 20 interface in flow chart form according to certain teachings of
ciated with a content edge. The window or display edge may
the present disclosure. The application programming interface operates in an environment with user interface software
be associated with the edge of a display region. The method
1100 for providing the scroll indicator includes transferring a
interacting with a software application in order to provide a
scroll indicator call to determine whether at least one scroll
gesture operation. The method 1300 for providing the gesture
indicator attaches to an edge of a scroll region or a window 25 operation includes transferring a handle gesture event call at
edge at block 1102. A scroll indicator may be displayed on
block 1302. The method 1300 further includes transferring a
gesture change call in response to the handle gesture event
any display edge, window edge or scroll region edge. The
call at block 1304.
method1100 furtherincludes optionally attachingatleastone
scroll indicator to the edge of the scroll region based on the
In certain embodiments, a user input in the form of two or
scroll indicator call at block 1104. Alternatively, the method 30 more points is received by a display ofa device.A multi-touch
1100 further includes optionally attaching at least one scroll
driver ofthe device....i... the user input and packages the
event into an event object. Awindow server--1... the event
indicator to the window edge of the view based on the scroll
object and determines whether the event object is a gesture
indicator call at block 1106.
event object. If the window server determines that a gesture
In some embodiments, the operations ofmethod 1100 can
be altered, modified, combined, or deleted. For example, 35 event object has been received, then user interface software
issues or transfers the handle gesture call at block 1302 to a
block 1104 can be deleted. Likewise, block 1106 can be
software application associated with the view. The software
deleted from the method 1100. Alternatively, the order of
application confirms that a gesture event has been received
block 1104 and block 1106 can be switched. Other methods
and passes the handle gesture call to a library of the user
having various operations that have been disclosed within the
interface software. The window server also --ates the
present disclosure can also be altered, modified, rearranged,
gesture event objectwiththe view that received the user input.
collapsed, combined, or deleted.
The library responds by transferring a gesture change call in
In certain embodiments ofthe present disclosure, transferresponse to the handle gesture event call at block 1304.
ring the scroll indicator call is either one ofissuing, initiating,
In one embodiment, a window or view associated with the
invoking or receiving the scroll indicator call. For example,
user input ....i... the change call in order to perform the
the user interface software (e.g., software kit or library) may 45 gesture event. The user software that provides the view
receive the scroll indicator call from the software application.
receives a gesture start event call, a gesture changed event
FIG. 12 illustrates details of an application programming
call, a zoom to scale setting for the view, and a gesture end
interface in flow chart form according to certain teachings of
call. The gesture calls ....l.. an input of a gesture event
the present disclosure. The application programming interwhich may be base event having a type such as a hand event,
face operates in an environment with user interface software so keyboardevent, system event, etc.A delegate associated with
interacting with a sofhvare application in order to determine
theapplication....:-a start gesture call, gesture didchange
if an inadvertent user input contacts a view of a display of a
call, and gesture did finish call. The user software is dynamidevice. The method 1200 includes transferringan inadvertent
cally linking into the application during the run time of the
user input call to determine whether the user input was inadgesture process
vertent at block 1202. The method 1200 further includes ss In some embodiments, the gesture changed function call
ignoring the inadvertent user input based on the determinacontains the following information about the gesture:
tion of the inadvertent user input call at block 1204. In one
the number of fmgers currently down;
embodiment, the inadvertent user input call comprises a
thumb detection call to determine whetherthe user input was
the number of fingers initially down;
an inadvertent thumb.
the rotation of the hand;
In certain embodiments ofthe present disclosure, transferring the inadvertent user input call is either one of issuing,
the scale of the hand;
initiating, invoking or receiving the inadvertent user input
the translation of the hand;
call.
AgestureAPI provides an interface betweenan application

and user software in orderto handle gesturing. Gesturingmay
include scaling, rotating, or other changes to a view, window,

65 the position ofthe inner and outermost fingers; and
the pressure of the first finger.

APLNDC00025425

US 7,844,915 B2

13

14

to represent that a larger portion of content is not being
displayed on display 1652 in FIG. 16B as a result ofthe zoom
in operation.
the stage of the finger Oust touch down, fully pressed, lifting
In at least some embodiments of the present disclosure, a
off, etc);
y
user desires to change a view 1670 from a scale factor of 2x
the position of the finger;
to a scale factor of 1x as illustrated in FIG. 16C. A first set of
the proximity of the finger (how hard you're touching);
user inputs 1672 and1674 that moveto the second set ofuser
inputs 1676 and1678 will decreasethe scale factor from 2x to
the orientation of the finger (what angle the ovoid is at);
io lx. It may be desirable for the user to scale from 2x to 1x

In other embodiments, more information about each finger
down may be included as follows.

the length of the major and minor axis,

withouthavingtomovetheuserinputs a large distance across

the view 1670. In anenvironmentwith user interface software
the velocity of the finger; and
interacting with a software application, a gesture scaling
transform flag may be set in order to determine a scaling
the eccentricity of the finger's ovoid.
A gesture event object may be a chord event object having is transform for a view associated with a user input having a
plurality of input points. The scaling transform flag scales
a chord count (e.g., number of fingers contacted the view or
eitherfrom a current scale factor to a -i---scale factor or
display), a chord start event, a chord change event, and a
from the current scale factor to a maximum scale factor. For
chord end event. A chord change event may include a scaling
example, a flag may be set at the position associated with a
or rotation transform.
20 1.5x scale factor and a third set ofuser inputs 1680 and 1682.
FIG. 14 illustrates details of an application programming
Auserdesiringtochangethescalefactorfrom2×to1xwould
interface in flow chart form according to certain teachings of
onlyhavetomovehisfingers,theuserinputs,fromthefirstset
the present disclosure. The application programming inter1672 and 1674 to the third set 1680 and 1682 if the gesture
face operates in an environment with user interface software
scaling transform flag has been set at a scale factor of 1.5×.
interacting with a software application in order to provide a
25 FIG. 17 illustrates details of an application programming
scaling transform of a display region, window, or view of a
interface in flow chart form according to certain teachings of
display of a device. The method 1400 for providing the scalthe present disclosure. The application programming intering transform includes transferring a scaling transform call to
face operates in an environment with user interface sonware
determine a scaling transform for a view associated with a 30 interacting with a software application in order to provide a
user input having a plurality ofinput points at block 1402. The
rotation transform of a view, window, or display region of a
method 1400 further includes transferring a scaling gesture
display of a device. The method 1700 for providing the rotastart call at block 1404. The method 1400 further includes
tion transform includes transferring a rotation transform call
transferring a scaling gesture progress call at block1406. The
to determine a rotationtransform for a view associated with a
method 1200 further includes transferring a scaling gesture
userinputhaving a plurality ofinputpoints at block 1702. The
end call at block 1408.
method 1700 further includes transferring a rotation gesture
start call at block 1704. The method 1700 further includes
In certain embodiments, a user input in the form of two or
transferring a scaling gestureprogress call at block 1706. The
more input points (e.g., fingers) moves together or apart to
method 1700 further includes transferring a scaling gesture
invoke a gesture event that performs a scaling transform on
end call at block 1708.
the view associated with the user input. A scale transform
In certain embodiments, a user input in the form of two or
includes a minimum and
scale factor. FIG. 15
more input points rotates to invoke a gesture event that perillustrates a display 1502 of a device having a scaling transforms a rotation transform on the view associated with the
form of a view. The view 1504 (e.g., web, text, or image
user input. The totation transform includes a ...:..:...- and
content) has a first scale factor. A user input (e.g., two fingers
degree of rotation for associated ---'--I-------- and
moving apart) associated with the view 1504 is interpreted as 45
maximum rotation views. The user input may temporarily
a gesture event to zoom out from view 1504 to view 1508
rotate a view past a maximum degree ofrotation prior to the
having a second scale factor that exceeds the maximum scale
view snapping back to the maximum degree of rotation.
factor ofthe view 1516. A snapback flag determines whether
the zoom out can proceed past the -----:---um scale factor to
FIG. 18 illustrates details of an application programming
view 1508 prior to snapping back to the -i-- scale so interface in flow chart form acconiing to certain teachings of
factor associated with view 1516.
the present disclosure. The application programming interface operates in an environment with user interface software
FIG. 16A illustrates a display 1604 of a device having a
interacting with a software application in order to notify a
first scaling factor of a view 1616. A user input (e.g., two
delegate of at least one animation associated with a display
fingers 1608 and 1610 moving together) associated with the
view 1614 is interpreted as a gesture event to zoom in from ss region, window, orview ofa display ofa device. A delay inthe
animation may be specified by the API. Also, multiple aniview 1614 to view 1664 having a second scale factor as
illustrated in FIG. 16B. The dashed regions 1602 and 1650
mations may be assigned priority by the API. The method
1800 for notifying the delegate includes determining whether
represent the total area of the content with the only content
being displayed in the display area 1604 and 1652. In perany animationoccurs at block 1802. The method 1800 further
formingthe scaling transform from FIG.16Ato FIG.16B, the so includes checking the progress ofan animation at block 1804.
If progress has occurred, then the next state (e.g., position,
center of the gesture event, center 1612 for FIG. 16A and
opacity, or transform) of the animation can be calculated at
center 1660 for FIG. 16B, remains in the same position with
block 1806. Ifprogress has completed at block 1806, then at
respect to the display 1604. The scroll indicator1606 shrinks
block 1808 it is determined whether the view associated with
to become scroll indicator 1654 during the transform to indicate that a smaller portion of the total content 1650 is being 65 the completedanimationis associatedwith a delegate. Ifso, a

displayedon display 1604 as a result ofthe mom in operation.
The dashed region 1650 is larger than the dashed region 1602

delegate call is transferred to notify the delegate of the animation for the view at block 1810. The delegate operating

APLNDC00025426

US 7,844,915 B2
15

16

underthecontrolofthesoftwareapplicationcanchangeother
enabled at block 2106. The synchronization flag can be
views in response to the view being modified by the animaenabled when the processes to be synchronized have each
tion.
sent messages to a window server operating the user interface
In certain embodiments, software invokes an animation
software. The method 2100 further includes updating the
thatperformsascalingtransformontheviewassociatedwith s attributes of the views from a first state to a second state
the user input. A display may include numerous views. The
independently at block 2108. In one embodiment, the winview being increased in size by the scaling transform may
dow server
the updated attributes from each process
obstruct other views in which case the other views may need
at different times. The method 2100 further includes transferto be reduced in size. Alternatively, the view being decreased
ring a start animation call to draw the requested animations
in size by the scaling transform may create additional area for 10 when both processes have updated attributes associated with
other views to increase in size.
the second state at block 2110.
FIG. 19 is flow chart of a method for animating a display
In some embodiments, a first data structure or layer tree
region, windows, or views displayed on a display ofa device.
represents a hierarchy of layers that correspond to the views
The method 1900 includes starting at least two animations at
or windows of the processes. A second data structure or
block 1902. The method 1900 further includes determining 15 render tree represents a similar copy of the layer tree. Howthe progress of each animation at block 1904. The method
ever, the render tree is not updated until the independent
1900 further includes completing each animation based on a
processes have completed their separate animations. At this
single timer at block 1906.
time, the render tree updates and redraws the screen with the
Incertain embodiments ofthepresentdisclosure,the single
new animations.
timer includes a timer based on a redraw interval which is a 2o FIGS.22Aand22B illustrate synchronizingthe resizing of
time period between the display ofa current frame and a next
views or windows of a display of a device. For example, a
frame of the display of the device. In this case, changes in
window 2210 associated with a first process with a size
animation are updated to the display during the redraw interattribute may :--- in size by changing from a first state,
val in orderto display the chances duringthe next frame ofthe
window 2210 in FIG. 22A, to a second state window 2210 m
display. The progress of each animation may be calculated 25 FIG. 22B. At approximately the same time, a second window
periodically or based upon a progress call.
2220 may decrease in size inproportion to the i
. in size
The method 1900 may further include determining
ofthefirstwindow2210.Themethod2100providessynchrowhether each animation is associated with a delegate. The
nization of the resizing ofthe windows 2210 and 2220 illusdelegate is then notified of the animation. Other views not
trated in FIGS. 22A and 22B. The animations in changing
associated with an animation may be changed depending on 30 from the first state to the second state may occur incrementhe software application controlling the delegate.

tally and occur with the synchronization of method 2100.

FIG. 20 is flow chart of a method for animating a display
FIG. 23 illustrates a method for switching ownership of a
region, windows, or views displayed on a display ofa device.
view of an application displayed on a display of a data proThe method 2000 includes providing a single animationtimer
cessing device. The method 2300 includes constructing a data
at block 2002. The method 2000 further includes animating a as structure having a hierarchy of layers with a layer being
plurality of animations with the single animation timer at
associated with a view and owning the view at block 2302.
block 2004. For example, a single timer may control all
The layers may be content, windows, video, images, text,
animations whichoccur simultaneously. The animations may
media, or any other type of object for user interface of the
include a transform, a frame, and an opacity animation. A
application. The method 2300 further includes removing the
animation transform may include a scaling or rotation trans- 40 layer from the data structure at block 2304. The method 2300
form. A frame animation may include resizing ofa frame. An
further includes switching ownership of the view from the
opacity animation changes the opacity from opaque to translayer to the view at block 2306.
parent or vice versa.
In some embodiments, each layerfrom the data structure is
FIG. 21 illustrates details of an application programming
associated with a view. The layer associated with the view
interface in flow chart form according to certain teachings of 45 sends a delegate function call to the view in order to generate
the present disclosure. The application programming intercontent provided by the view. A first pointer reference points
face operates in an environment with user interface software
fromthe layer to the view. A second pointer reference points
interacting with multiple software applications or processes
from the view to the layer. The number ofreferences pointing
in order to synchronize animations associated with multiple
to an object such as the view is defmed as the retained count
views or windows of a display ofa device. The method 2100 so ofthe object. The view may .., . notification that the layer
for synchronizing the animations includes setting attributes
will be removed from the data stmeture. Removing the layer
ofviews independently with each view being associated with
from the data structure may occur based on the view associa process at block2102. For example, an attribute or property
ated with the layer being removed from the display of the
of a view may include a position, size, opacity, etc. An anidevice. When the layer is removed from the data structure or
mation alters one or more attributes from a first state to a ss layer tree the pointer from the layer to the view will be
second state. The method 2100 further includes transferring a
removed. The view will have a retained count of zero and be
synchronization call to synchronize animations for the muldeallocated or removed from memory ifthe ownership ofthe
tiple views ofthe display at block 2104. The synchronization
view is not reversed.The view will have a retained count ofat
call may include input parameters or arguments such as an
least one if ownership is reversed.
identification of the synchronization of the processes and a 6o FIG.24 illustrates a method for memory management of a
list of the processes that are requesting animation of the

view ofan applicationdisplayedon a display ofa device. The

includes the identification and the number of processes that

hierarchy of layers with at least one layer being associated

multiple views. In one embodiment, the synchronization call

method 2400 includes constructing a data structure having a

are requesting animation. In one embodiment, each applicawith the view at block 2402. The method 2400 further
tion or process sends a synchronizationcall at different times. 65 includes storing the data structure in memory at block 2404.
The method 2100 further includes transferring a synchroniThe method 2400 further includes maintaining a retained
zation confirmation message when a synchronization flag is
count of the number of references to the view from other

r -, ..-wiana he I mpTO Imm thn PIRS Imaae Database on 04/25/2011

APLNDC00025427

US 7,844,915 B2
17

18

objects at block 2406. The method 2400 further includes
deallocating the view from memory if the retained count is
zero at block 2408. As discussed above, the retained count of
the view will be decremented ifthe layer is removed from the
data structure. Removing the layer from the data structure
may occur based on the view associated with the layer being
removed from the display of the device,
FIGS. 25A and 25B illustrate a data structure having a
hierarchy of layers with a layer being associated with a view.
The data structure includes layers 2502, 2504, and 2506.
Layer 2506 is associated with the view 2510. The layer 2506
associated with the 2510 view sends a delegate call to the
view in orderto generate content providedby the view. A first
pointerreference 2508 points from the layer2506 to the view
2510. A second pointer reference 2512 points from the view
2510 to the layer 2506. A third pointer reference 2532 may
point from user interface (UI) controller 2530 to the view
2510. The UI controller 2530 may control operations associated with the view 2510 such as scrolling the view 2510 in
response to a user input. The view 2510 in FIG. 25A has a
retained count of two based on the pointer references 2508
and 2532

duce other colors. The non-media content may be scanned to
the display at a slower rate compared to the media content.
FIG. 27 illustrates a data structure or layer tree having a
hierarchy of layers. The layers can be associated with media
s andnon-mediacontent.Forexample,1ayer2704isassociated
with media content 2706 such as a video. Layer 2710 is
associated with non-media content 2712 which may be user
interfaceviewforthevideo.Layers2720,2730,and2740are
associated with non-media content 2722, 2732, and 2742,
10 respectively, that forms the components of the non-media
content 2712. The method 2600 will determine whether each
layer of the data structure is associated with media or noncontent. Any layers associated with media content such as
layer 2704 will be removed from the data structure and pro15 cessed in a separate memory location.
In some embodiments, the methods, systems, and apparatuses ofthe present disclosure can be implemented in various
devices including electronic devices, consumer devices, data
processing devices, desktop computers, portable computers,
20 wireless devices, cellular devices, tablet devices, handheld
devices, multi touch devices, multi touch data processing
devices, any combination of these devices, or other like
devices. FIGS. 4-6 and 28-33 illustrate examples of a few of
If the layer 2506 is removed from the data stmeture as
these devices.
illustrated in FIG. 25B, then the pointer 2508 is removed. 25 FIG. 28 illustrates a device 2800 according to one embodiView 2510 will have a lower retained count as illustrated m
ment of the disclosure. FIG. 28 shows a wireless device in a
FIG. 25B. Ifview 2510 has a retained count of zero, then the
telephone configuration having a "candy-bar" style. In FIG.
memory storing the view 2510 will be deallocated.
28, the wireless device 2800 may include a housing 2832, a
FIG. 26 illustrates a method for compositing media and
display device 2834, an input device 2836 which may be an
non-media content of user interface for display on a device 30 alphanumeric keypad, a speaker 2838, a microphone 2840
The method 2600 includes constructing a data structure havand an antenna 2842. The wireless device 2800 also may
ing a hierarchy of layers associated with the user interface of
include a proximity sensor 2844 and an accelerometer 2846.
the device at block 2602. The method 2600 further includes
ItwillbeappreciatedthattheembodimentofFIG.28 may use
determining whether each layer ofthe data structure is asso¯
more or fewer sensors and may have a different form factor
ciated with media or non-media content at block 2604. The as from the form factor shown in FIG. 28.
data structure or layer tree is traversed in order to determine
The display device 2834 is shown positioned at an upper
whether each of the layers of the data structure is associated
portion of the housing 2832, and the input device 2836 is
with media or non-media content. The method 2600 further
shownpositioned at a lowerportion ofthe housing 2832. The
includes detaching a layer associated with media content
antenna2842 is shownextendingfromthe housing 2832 at an
from the data structure at block 2606. The method 2600 40 upper portion of the housing 2832. The speaker 2838 is also
further includes storing media content in a first memory loca¯
shown at an upper portion of the housing 2832 above the
tion at block 2606. The method 2600 further includes storing
display device 2834. The microphone 2840 is shown at a
non-media content in a second memory location at block
lower portion of the housing 2832, below the input device
2608. The method 2600 further includes compositing the
3286. It will be appreciatedthat the speaker 2838 and micromedia and non-media content for display on the device at 45 phone 2840 can be positioned at any location on the housing,
block 2610·
but are typically positioned in accordance with a user's ear
In some embodiments, compositing the media and nonand mouth, respectively.
The display device 2834 may be, for example, a liquid
media content includes retrieving the media content from the
crystal display (LCD) which does not include the ability to
first memory location, retrieving the non-media content from
the second memory location, and scanning the media and so accept inputs or a touch input screen which also includes an
LCD. The input device 2836 may include, for example, butnon-media content directly to the display. The memory location can be any type of memory located in cache, main
tons, switches, dials, sliders, keys or keypad, navigation pad,
touch pad, touch screen, and the like. Any well-known
memory, a graphics processing unit, or other location withm
spe
microphone and antenna can be used for speaker
a device. The media content may include video, video plugin, audio, image, or other time varying media. The media ss 2838, microphone 2840 and antenna 2842, respectively.
The data acquired from the proximity sensor 2844 and the
content may be in the form of a YUV model with the Y
representing a 1

°

component (the brightness) and U

accelerometer2846 can be combinedtogether, or used alone,

and V representing chrominance (color) components. The
to gather information about the user's activities. The data
from the proximity sensor 2844, the accelerometer 2846 or
media content may be scanned to the display at a rate of
substantially twenty to forty frames per second. The media so bothcan be used, for example, to activate/deactivate a display
backlight, initiatecommands, make selections, control scrollcontent may be scaled prior to being scanned to the display of
ing, gesturing, animating or other movement in a display,
the device.
control input device settings, or to make other changes to one
The non-media content may include content, views, and
or more settings ofthe device. In certain embodiments ofthe
images that do not require frequent updating. The non-media
content may be in the form of a RGB model which is an 65 presentdisclosure, the device 2800 can be used to implement
at least some ofthe methods discussed in the present discloadditive model in which red, green, and blue (often used in
additive light models) are combined in various waysto reprosure.

m-., -.,sa-a as, neom imm the uma Imann natabase on 04125/2011

APLNDC00025428

US 7,844,915 B2

19

20

FIG. 29 shows a device 2950 in accordance with one
embodiment of the disclosure. The device 2950 may include
a housing 2952, a displaylinput device 2954, a speaker 2956,
a microphone 2958 and an optional antenna 2960 (whichmay
be visible on the exterior of the housing or may be concealed
within the housing). The device 2950 also may include a
proximity sensor 2962 and an accelerometer 2964. The
device 2950 may be a cellular telephone or a device which is
an integrated PDA and a cellular telephone or a device which
is an integrated media player and a cellular telephone or a
device which is both an entertainment system (e.g. forplaying
games) and a cellular telephone, or the device 2950 may be
other types of devices described herein. In one particular
embodiment, the device 2950 may include a cellular telephone and a media player and a PDA, all containedwithin the
housing 2952. The device 2950 may have a form factor which
is small enough that it fits within the hand of a normal adult
and is light enough that it can be carried in one hand by an
adult. It will be appreciatedthat the term "portable"means the
device can be easily held in an adult user's hands (one or
both); for example, a laptop computer and an iPod are portable devices.

telephone integrated with a media player which plays MP3
files, such as MP3 music files.
Each ofthe devices shown in FIGS. 4, SA, 5B, GC, 6A, 6B,
6C, 6D, 28, 29, 30A and 30B may be a wireless communicas tion device, such as a cellular telephone, and may include a
plurality of components which provide a capability for wireless communication. FIG. 31 shows an embodiment of a
wireless device 3070 which includes the capability for wireless w-Scation. The wireless device 3070 may be
lo included in any one ofthe devices shown in FIGS. 4, SA, 5B,
5C, 6A, 6B, 6C, 6D, 28, 29, 30A and 308, although alternative embodiments of those devices of FIGS. 4, 5A, 5B, 5C,
6A, 6B, 6C, 6D, 28, 29, 30A and 308 may include more or
fewer components than the Wireless device 3070.
is
Wireless device 3070 may include anantenna system 3101.
Wireless device3070 may also include a digital and/oranalog
radio frequency (RF)transceiver3102, coupled to the antenna
system 3101, to transmit and/or ....: . voice, digital data
and/or media signals through antenna system 3101.
20 Wirelessdevice3070mayalsomcludeadigitalprocessing
system 3103 to control the digital RF transceiver and to
manage the voice, digital data and/or media signals. Digital
processing system 3103 may be a general purpose processmg
In one embodiment, the displaylinput device 2954 may
device, such as a microprocessor or controller for example.
dludeay,
a
D.a
n
25 Digital LrocessLne, sœs3103Smayalsonbe a eci p
se

point touch screen is a capacitive sensing medium configured
grated circuit), FPGA (field-programmable gate array) or
to detect multiple touches (e.g., blobs on the display from a
DSP (digital signal processor). Digital processing system
user's face or multiple fingers concurrently touching or
3103 may also include other devices, as are known inthe art,
nearlytouchingthe display) orneartouches (e.g., blobs onthe ao to interface with other components of wireless device 3070.
display) that occurat the same time and at distinct locations in
For example, digital processing system 3103 may include
the plane of the touch panel and to produce distinct signals
analog-to-digitalanddigital-to-analogconverters to interface
representative of the location of the touches on the plane of
with other components of wireless device 3070. Digital prothe touch panel for each of the multiple touches
cessing system 3103 may include a media processing system
In certain embodiments of the present disclosure, the 35 3109, which may also include a general purpose or special
device 2800 can be used to implement at least some of the
purpose processing device to manage media, such as files of
methods discussed in the present disclosure·
audio data.
FIGS. 30A and 30B illustrate a device 3070 according to
Wireless device 3070 may also include a storage device
3104, coupled to the digital processing system, to store data
one embodiment ofthe disclosure. The device 3070 may be a
cellular telephone which includes a hinge 3087 that couples a 40 and/oroperatingprograms fortheWireless device 3070. Stordisplay housing 3089 to a keypad housing 3091. The hinge
age device 3104 may be, for example, any type of solid-state
3087 allows a userto open and close the cellulartelephone so
or magnetic memory device.
Wireless device 3070 may also include one or more input
that it can be placed in at least one of two different configurations shown in FIGS. 30A and 30B. In one particular
devices 3105, coupledto the digital processing system 3103,
embodiment, the hinge 3087 may rotatably couple the display 49 to accept user inputs (e.g., telephone numbers, names,
housing to the keypad housing. In particular, a user can open
addresses, media selections, etc.) Input device 3105 may be,
the cellular telephone to place it in the open configuration
for example, one or more of a keypad, a touchpad, a touch
shown in FIG. 30A and can close the cellular telephone to
screen, apointingdevice incombinationwith a display device
place it in the closed configuration shown in FIG. 30B. The
or similar input device.
keypad housing 3091 may include a keypad 3095 which so Wireless device 3070 may also include at least one display
device 33106, coupled to the digital processing system 3103,
receives inputs (e.g. telephone number inputs or other alpha--A inputs) from a user and a microphone 3097 which
to display information suchas messages, telephonecall inforreceives voice input from the user. The display housing 3089
mation, contact information, pictures, movies and/or titles or
other indicators of media being selected via the input device
may include, on its interior surface, a display 3093 (e.g. an
LCD) and a speaker 3098 and a proximity sensor 3084; on its ss 3105. Display device 3106 may be, for example, an LCD
display device. In one embodiment, display device 3106 and
exterior surface, the display housing 3089 may include a
input device 3105 may be integrated together in the same
speaker 3096, a temperature sensor 3094, a display 3088 (e.g.
another LCD), an ambient light sensor 3092, and a proximity
device (e.g., a touch screen LCD such as a multi-touch input
sensor 3084A. Hence, in this embodiment, the display houspanel which is integrated with a display device, such as an
ing 3089 may include a first proximity sensor on its interior so LCD display device). The display device 3106 may include a
backlight 3106A to illuminate the display device 3106 under
surface and a second proximity sensor on its exterior surface.
certain circumstances. It will be appreciatedthat the Wireless
In at least certain embodiments, the device 3070 may condevice 3070 may include multiple displays.
tain components which provide one or more ofthe functions
Wireless device 3070 may also include a battery 3107 to
of a wireless communication device such as a cellular telephone, a media player, an entertainment system, a PDA, or 65 supply operating power to components of the system including digital RF transceiver 3102, digital processing system
other types of devices described herein. In one implementa3103, storage device 3104, input device 3105, microphone
tion of an embodiment, the device 3070 may be a cellular

m.- --..:2-2 a.. see

ek. nine 1......

-- ftAIORIOnid

APLNDC00025429

US 7,844,915 B2

21

22

310A, audiotransducer3108,mediaprocessingsystem3109,
the hierarchy of layers at block 3404. An application may
sensor(s) 3110, and display device 3106. Battery 3107 may
interact with the hierarchy of views via the API without
be, for example, a rechargeable or non-rechargeable lithium
accessing the hierarchy of layers operatmg below the hieraror nickel metal hydride battery. Wireless device 3070 may
chy of views.
also include audio transducers 3108, which may include one s
In some embodiments, a platform provides various scrollor more speakers, and at least one microphone 3105A. In
ing, gesturing, and animating operations. The platform
certain embodiments of the present disclosure, the wireless
includes hardware components and an operating system. The
device 3070 can be used to implement at least some of the
hardware components may include a processing unit coupled
methods discussed in the present disclosure,
to an input panel and a memory coupled to the processor. The
FIG. 32 shows another example ofa device accordingto an 10 operating system includes one or more programs that are
embodiment ofthe disclosure. This device 3200 may include
stored in the memory and configured to be executed by the
a processor, such as microprocessor 3202, and a memory
processing unit. One or more programs include various
3204, which are coupled to each other through a bus 3206.
instructions for transferring function calls or messages
The device 3200 may optionally include a cache 3208 which
through an application programming interface in order to
is coupled to the microprocessor 3202. This device may also 15 perform various scrolling, gesturing, and animating operaoptionally include a display controller and display device
tions.
3210 which is coupled to the other components through the
In an embodiment, the one or more programs include
bus 3206. One or more input/output controllers 3212 are also
instmetions for transferring a bounce call through an API to
coupled to the bus 3206 to provide an interface for input/
cause a bounce ofa scrolledregion in an opposite direction of
output devices 3214 and to provide an interface for one or 20 a scroll based on a region past an edge ofthe scrolled region
more sensors 3216 which are for sensing user activity. The
being visible in a display region at the end of the scroll. In an
bus 3206 may include one or more buses connected to each
embodiment, the one or more programs include instructions
other through various bridges, controllers, and/or adapters as
for transferring a rubberband call through an API to cause a
is well known in the art. The inputloutput devices 3214 may
rubberband effect on a scrolled region by a predetermined
include a keypad or keyboard or a cursor control device such 25
displacement when the scrolled region exceeds a
as a touch input panel. Furthermore, the inputloutput devices
display edge based on a scroll. In an embodiment, the one or
3214 may include a network interface which is either for a
more programs include instructions for transferring a direcwirednetworkorawirelessnetwork(e.g.anRFtransceiver).
tional scroll call through an API to set a scroll angle for
The
.. 3216 may be any one of the sensors described
locking the scrolling in at least one ofa vertical or ahorizontal
herein including, for example, a proximity sensor or an ambi- 30 direction,
ent light _ .. In at least certain implementations of the
In an embodiment, the one or more programs include
device 3200, the microprocessor 3202 may
.. data from
instructions fortransferring a scroll hysteresis call throughan
one or more sensors 3216 and may perform the analysis of
API to determine whether a user input invokes a scroll. In an
that data in the manner described herein. For example, the
embodiment, the one or more programs include instructions
data may be analyzed through an artificial intelligence pro- 35 fortransferring adecelerationscroll callthroughanAPIto set
cess or in the other ways described herein. As a result ofthat
a deceleration factor for a user input based on the user input
analysis, the microprocessor 3202 may then automatically
invoking a scroll. In an embodiment, the one or more procause an adjustment in one or more settings of the device,
grams include instructions for transferring a scroll indicator
In certain embodiments of the present disclosure, the
call through an API to determine whether at least one scroll
device 3200 can be used to implement at least some of the 40 indicator attaches to a content edge or a display edge of a
methods discussed in the present disclosure.
display region.
FIGS. 33A-C show another example ofa device according
In some embodiments, the platform includes a framework
to at least certain embodiments of the disclosure. FIG. 33A
containing a library of software code. The framework interillustrates a laptop device 3300 with a keyboard 3302, a body
acts with the programs ofthe platform to provide application
3304, a display frame 3306, and a display 3308. The laptop 45 programming interfaces for performmg various scrolling,
device 3300 canbe converted into a tablet device as illustrated
gesturing, and animating operations. The framework also
in FIG. 33B and FIG. 33C. FIG. 33B illustrates the conversion
includes associated- _ (e.g., images, text, etc.) that are
ofthe laptop device into a tablet device. An edge of a display
stored in a single directory.
frame 3356 containing a display 3358 is slide withinthebody
In an embodiment, the library of the framework provides
3354 across the top ofa keyboard 3352 until forming a tablet so anAPI for specifyinga bounce operationto cause a bounce of
device as illustrated in FIG. 33C. The tablet device with a
a scrolled region in an opposite direction of a scroll based on
display 2362 and a display frame 3366 rests on top of a body
a region past an edge of the scrolled region being visible in a
3360.
display region at the end ofthe scroll. In an embodiment, the
In certain embodiments of the present disclosure, the laplibrary of the framework provides an API for specifying a
top device 3300 can be used to implement at least some ofthe 55 rubberband operation that has a rubberband effect on a
methods discussed in the present disclosure,
scrolled region by a predetermined --- displacement
FIG. 34 illustrates details of an application programming
when the scrolled region exceeds a display edge based on a
interface in flow chart form according to certain teachings of
scroll. In an embodiment, the library of the framework prothe present disclosure. The application programming intervides an API for specifying a directional scroll operation to
face operates in an environment with user interface software 60 set a scroll angle for locking the scrolling m at least one of a
interacting with a software application. In some embodiverticalorahorizontaldirection.
ments, a hierarchy ofviews operates on top ofa hierarchy of
In an embodiment, the library of the framework provides
layers within the user interface software. TheAPI operates as
an API for specifying a scroll hysteresis operation to deterillustrated in method 3400 that includes constructing a hierminewhetherauser input invokes a scroll. In an embodiment,
archy of views operating on top of a hierarchy of layers at 65 the library ofthe framework provides anAPI for specifying a
block 3402. The method 3400 further includes providing
deceleration scroll operation to set a deceleration factor for a
access to the hierarchy of views without providing access to
user input based on the user input invoking a scroll. In an

.- --,dana hw impTr3 Imm the PIRS Imsae Database on 04/25/2011

APLNDC00025430

US 7,844,915 B2
23
embodiment, the library ofthe frameworkprovides anAPI for
specifying a scroll indicator operation to determine whether

at least one scroll indicator attaches to a content edge or a
display edge of a display region.
In the foregoing specification, the disclosure has been
described with reference to specific exemplary embodiments
thereof. It will be evident that various modifications may be
made thereto without departing from the broader spirit and
scope of the disclosure as set forth in the following claims· 10

The specification and drawings are, accordingly, to be
regardedin an illustrative sense ratherthan a restrictive sense.

24
receiving a user input, the user input is one or more input
points applied to a touch-sensitive display that is integrated with the data processing system;

creating an event object in response to the user input;
determining whether the event object invokes a scroll or
gesture operation by distinguishing between a single
input point applied to the touch-sensitive display that is
interpreted as the scroll operation and two or more input
points applied to the touch-sensitive display that are
interpreted as the gesture operation;
issuing at least one scroll or gesture call based on invoking
the scroll or gesture operation;

What is claimed is:
15
1. A machine imp1--*ed method for scrolling on a
touch-sensitive display of a device comprising:
receiving a user input, the user input is one or more input

responding to at least one scroll call, if issued, by scrolling
a window having a view associated with the event
object; and

respondingto at least one gesture call, if issued, by scaling
the view associated with the event object based on

receiving the two or more input points in the form ofthe
user input.
9. The medium as in claim 8, further comprising:
rubberbandinga scrolling region displayedwithin the window by a predetermined maximum displacement when
the scrolled region exceeds a window edge based onthe
scroll.
10. The medium as in claim 8, further comprising:

points applied to the touch-sensitive display that is inte- go

grated with the device;
creating an event object in response to the user input;
determining whether the event object invokes a scroll or
gesture operation by distinguishing between a single
input point applied to the touch-sensitive display that is
interpreted as the scroll operation and two ormore input
points applied to the touch-sensitive display that are
interpreted as the gesture operation;
issuing at least one scroll or gesture call based on invoking 3o
the scroll or gesture operation;
responding to at least one scroll call, if issued, by scrolling
a window having a view associated with the event object

based on an amount of a scroll with the scroll stoppedat
a predetermined position in relation to the user input;
and

responding to at least one gesture call, ifissued, by scaling
the view associated with the event object based on

attaching scroll indicators to a content edge of the view.
11. The medium as in claim 8, further comprising:
attaching scroll indicators to a window edge of the view.

12. The medium as in claim 8, wherein determining
whetherthe event object invokes a scroll or gesture operation
is based on receiving a drag user input for a certain time
period.
13. The medium as in claim 8, further comprising:
respondingto at least one gesture call, ifissued, by rotating
a view associated with the event object based on receiving a plurality of input points in the form of the user

receiving the two or more input points in the form ofthe
user input.
input.
2. The method as in claim 1, further comprising:
14. The medium as in claim 8, wherein the data processing
rubberbandinga scrolling region displayedwithinthe winsystem is one of: a data processing device, a portable device,
dow by a predetermined maximum displacement when
a portable data processing device, a multi touch device, a
the scrolling region exceeds a window edge based on the 45 multi touch portable device, a wireless device, and a cell
scroll.
phone.
3. The method as in claim 1, further comprising:
15. An apparatus, comprising:
attaching scroll indicators to a content edge ofthe window.
means for receiving, through a hardware device, a user
input on a touch-sensitive display of the apparatus, the
4. The method as in claim 1, further comprising:
so
user input is one or more input points applied to the
attaching scroll indicators to the window edge.
touch-sensitive display that is integrated with the appa5. The method as in claim 1, wherein determining whether
ratus;
the event object invokes a scroll or gesture operation is based
means for creating an event object in response to the user
on receiving a drag user input for a certain time period.
input;
6. The method as in claim 1, further comprising:
means for determining whether the event object invokes a
respondingto at least one gesture call, ifissued, by rotating
scroll or gesture operation by distinguishing between a
a view associated with the event object based on receivsingle input point applied to the touch-sensitive display
ing a plurality of input points in the form of the user
that is interpretedas the scroll operation and two or more
input.
so
input points applied to the touch-sensitive display that
7. The method as in claim 1, wherein the device is one of:
are interpreted as the gesture operation;
a data processing device, a portable device, a portable data
means for issuing at least one scroll or gesture call based on
processing device, a multi touch device, a multi touch porinvoking the scroll or gesture operation;
table device, a wireless device, and a cell phone.
means for respondingto atleastone scroll call, ifissued, by
8. A machine readable storage medium storing executable 65
scrolling a window having a view associated with the
program instructions which when executed cause a data proevent object; and
cessing system to perform a method comprising:

conv nrovided bv USPTO from the PIRS Im

APLNDC00025431

US 7,844,915 B2
25
means for responding to at least one gesture call, ifissued,
by scaling the view associated with the event object

26
19. The apparatus as in claim 15, wherein determining
whetherthe event object invokes a scroll or gesture operation
is based on receiving a drag user input for a certain time

based on receiving the two or more input points in the
form of the user input.
period.
16. The apparatus as in claim 15, further comprising:
5
20. The apparatus as in claim 15, further comprising:
means for responding to at least one gesture call, if issued,
means for rubberbanding a scrolling region displayed
by rotating a view associated with the event object based
within the window by a predetermined
dison receiving a plurality of input points in the form ofthe
placement when the scrolling region exceeds a window
user input.
edge based on the scroll.
17. The apparatus as in claim 15, further comprising:
10 21. The apparatus as in claim 15, wherein the apparatus is
one of: a data processing device, a portable device, a portable
means for attaching scroll indicators to a content edge of
data processing device, a multi touch device, a multi touch
the window.
portable device, a wireless device, and a cell phone.
18. The apparatus as in claim 15, further comprising:
means for attaching scroll indicators to the window edge.
* * * * *

Conv nrovided bv i mnm imm the 9mm imann

on 04/25/2011

APLNDC00025432