BlackBerry Limited v. Facebook, Inc. et al

Filing 1

COMPLAINT Receipt No: 0973-21360760 - Fee: $400, filed by Plaintiff BlackBerry Limited. (Attachments: # 1 Exhibit A, # 2 Exhibit B, # 3 Exhibit C, # 4 Exhibit D, # 5 Exhibit E, # 6 Exhibit F, # 7 Exhibit G, # 8 Exhibit H, # 9 Exhibit I, # 10 Exhibit J) (Attorney James R Asperger added to party BlackBerry Limited(pty:pla))(Asperger, James)

EXHIBIT C

EXHIBIT C

US008279173B2

(12) United States Patent

(10) Patent No.:

BrOWn et al.
(54) USER INTERFACE FOR SELECTINGA
PHOTO TAG

(73) Assignee: Research In Motion Limited, Waterloo,
Ontario

7,693,906 B1
7,735,018 B2

2004/0039988 A1

2004/02521 19 A1

2005/OO30588 A1
2005, OO39108 A1
2005/0057576 A1

CN

FOREIGN PATENT DOCUMENTS
1193632 A
1, 2002
(Continued)

Second Office Action mailed Jun. 22, 2011. In Chinese patent appli
tion No. 200810144678.1.
CaO NO

Prior Publication Data

Continued
(Continued)

Jan. 26, 2012

Related U.S. Application Data
(63) Continuation of application No. 1 1/746,285, filed on
May 9, 2007, now Pat. No. 8,031,170.

(51) Ea's/00

2/2005 Reese et al.
2/2005 Hudson
3/2005 Shen et al.

OTHER PUBLICATIONS

Oct. 4, 2011

US 2012/0023436A1

2/2004 Lee et al.

12/2004 Hunleth et al.

(Continued)

(21) Appl. No.: 13/252,807
(65)

4/2010 Amidon et al.
6, 2010 Bakhash

7,840,907 B2 11/2010 Kikuchi et al.
8,031,170 B2 * 10/2011 Brown et al. ................. 345,156
2003/0088582 A1 5/2003 Pflug

Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.C. 154(b) by 0 days.
This patent is Subject to a terminal disclaimer.

(22) Filed:

*Oct. 2, 2012

6,650,889 B1 * 1 1/2003 Evans et al. ................ 455,412.1
7,636,450 B1 12/2009 Bourdev

(75) Inventors: Michael S. Brown, Kilchanar (CA);
Gerhard D. Klassen, Waterloo (CA);
Terrill Dent, Waterloo (CA)

(*) Notice:

US 8.279,173 B2

(45) Date of Patent:

(2006.01)

Primary Examiner — Nitin Patel
(74) Attorney, Agent, or Firm — Novak Druce + Quigg LLP
(57)

ABSTRACT

There is disclosed a user interface for selecting a photo tag. In

.
E. c - - - -ificati- - - - - -s
so 34554 E.
(58) Field of Classification Search ..................
s
S
lication file f
i.838,
5/412.1

an embodiment, the user interface embodies a method of
selecting a photo tag for a tagged photo, comprising: provid
ing a tag entry field for entering a photo tag; in dependence
upon a string entered by a user, displaying in a matching tag
list any tags from one or more selected tag sources matching

(56)

the entered String. The method may further comprise display
ing a tag type for each tag appearing in the matching tag list.
The method may further comprise allowing user selection of

ee appl1cauon Ille Ior complete searcn n1Story.

References Cited

5,479,602 s EN points
aecker et al.
6,002,401 A

6,317,142 B1

a tag in the matching tag list to complete thetag entry field.

12, 1999 Baker

1 1/2001 Decoste et al.

20 Claims, 10 Drawing Sheets

410
404

408

404

ext i typed before

406
412

EXHIBIT C
Page 149

Thee Boukeluth
Tim Jackson

US 8,279,173 B2
Page 2
U.S. PATENT DOCUMENTS
2005. O19301.0
2006/0173918
2006/02621 16
2008/002 1876
2008/0106594
2008, 0215583
2008/0282177
2008/0306921

A1
A1
A1
A1
A1
A1
A1
A1

2008/0309617 A1
2009/0225.178 A1

9, 2005
8, 2006
11/2006
1/2008
5, 2008
9/2008
11, 2008
12/2008

DeShan et al.
Nakase et al.
Moshiri et al.
Ahern et al. ...................... 707/3
Thrun
Gunawardena et al.
Brown et al.
Rothmuller et al.

12/2008 Kong et al.

9, 2009 Nakase et al.

1.

FOREIGN PATENT DOCUMENTS
CN
WO

1933643 A
2006102656 A

the internet (on Aug. 17, 2007): URL:http://ieeexplore.ieee.org/ie15/
7756/33539/01593574.pdf?tp=&isnumber=arnumber=1593574.
Ahern et al., Zone tag: Designing context-aware mobile media cap
ture to increase participation, yahoo! Research Berkeley, Sep. 6.
2006. http://groups.ischool.berkeley.edu/pics/papers Ahern et
al Zonetag pics06.pdf.
Communication Pursuant toArticle 94(3) EPC mailed Jan. 30, 2008.
In European patent application No. 07107866.1.
Summons to Attend Oral Proceedings Pursuant to Rule 115(1) EPC
mailed Aug. 27, 2008. In European patent application No. 07107866.

3, 2007
9, 2006

OTHER PUBLICATIONS

English translation on Second Office Action mailed Jun. 11, 2011. In
Chinese patent application No. 200810144678.1.
Third Office Action mailed Feb. 29, 2012. In Chinese patent appli
cation No. 200810144678.1.

English translation on Third Office Action mailed Feb. 29, 2012. In
Chinese patent application No. 2008101446781.
Office Action mailed Jan. 10, 2012. In Canadian patent application
No. 2,630,947.

Office Action mailed Nov. 18, 2010. In Canadian patent application
No. 2,630,947.

First Office Action mailed May 5, 2010. In corresponding Chinese
patent application No. 200810144678.1.
English translation on First Office Action mailed May 5, 2010. In
corresponding Chinese patent application No. 200810144678.1.
Ballagas, R. et al.; The Smart phone: A ubiquitous input device,
pervasive computing, IEEE (online) vol. 5, No. 1, Jan. 2006,pp.
70-77, XP002447 195, ISSN: 1536-1268: Retrieved (by EPO) from

Decision of the Examining Division mailed Dec. 12, 2008. In Euro
pean patent application No. 07107866.1.
Extended European Search Reportmailed Febuary 23, 2009. In Euro
pean patent application No. 08170532.9.
Communication Pursuant toArticle 94(3) EPC mailed Jan. 11, 2010.
In European patent application No. 08170532.9.
Wiseman, Josh; "iPhoto, meet Facebook'; Mar. 15, 2007; http://blog.
facebook.com/blog.php?post–22536571.
Extended European Search Report mailed Aug. 28, 2007. In Euro
pean patent application No. 07107866.1.
First Office Action mailed May 2010. In corresponding Chinese
patent application No. 2008.10144678.1.
Ballagas Retal. The Smartphone: A ubiquitous input device, perva
sive computing, IEEE (online) vol. 5, No. 1, Jan. 2006,pp. 70-77.
XPO02447195, ISSN: 1536-1268: Retrieved (by EPO) from the
internet (on Aug. 17, 2007): URL:http://ieeexplore.ieee.org/ie15/
7756/33539/01593574.pdf?tp=&isnumber=arnumber=1593574.
Ahern etal. Zone tag: Designing context-aware mobile media capture
to increase participation, yahoo! Research Berkeley, Sep. 6, 2006,
http://groups. iSchool.berkeley.edu/pics/papers Ahern et al
Zonetag pics06.pdf.

* cited by examiner

EXHIBIT C
Page 150

U.S. Patent

Oct. 2, 2012

126

Sheet 1 of 10

128

132

SIMIRUIM

US 8,279,173 B2

W+

130

Battery

110

Display
134

Auxiliary I/O

108

136

112
Data Port

Flash Memory
106

Application

114

Keyboard

Image

116
104

Applications

Trackball

Communication

117

Subsystem

Main
PrOCeSSOr

148

146

102

GPS
Subsystem

126

Short-Range
Communications
122

FIG. 1
EXHIBIT C
Page 151

N 100

U.S. Patent

Oct. 2, 2012

Sheet 2 of 10

127

N.
FIG.2
EXHIBIT C
Page 152

US 8,279,173 B2

U.S. Patent

Oct. 2, 2012

Sheet 3 of 10

302

301

US 8,279,173 B2

304

&
SSS

Š
Š&isi

302

308

arrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrra

sinsists

s

ss:
SSS

SSS

FIG. 3B N-3s
EXHIBIT C
Page 153

:

306

U.S. Patent

Oct. 2, 2012

Sheet 4 of 10

US 8,279,173 B2

302

S

:S

SS
Sas
SSSSSSSSS
s

ŠxŠišš:

Š& S.
310

312
302

s

Y S & Ess: S
S&SSSSS 3.

s
,

s&SS:

As Š

Šiš SSSSSSSSSSS&

310

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

EXHIBIT C
Page 154

U.S. Patent

Oct. 2, 2012

Sheet 5 of 10

US 8,279,173 B2

316
&
SSSSSSSSSSSSSSS

sT. T. T. T.....................Nix

ises:

is is is is:

s &s

Š

SSS
314

EXHIBIT C
Page 155

U.S. Patent

Oct. 2, 2012

SSSSS

SS & Siss

Sheet 6 of 10

:S

US 8,279,173 B2

SSSSSSSSSSSSS

404

acebook
:S

SSSSSSS 8.

SSSSSSSSSSSSSSSSSS

:::::: ::::::::: i: ::::::

Š

&
s
s

Ssssssssssssssssssssssssssssssssssssssssssss

&
:

&

406

S.&

N 400A

FIG. 4A

410

S$
y

SSSSSSSSSSSSSS

Š
:S

&
s:

O

aramie
Terrill Deft

text i typed before

412

404

406

S. Tee EOluket

in.

FIG. 4B
EXHIBIT C
Page 156

ge

U.S. Patent

Oct. 2, 2012

Sheet 7 of 10

US 8,279,173 B2

OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOY YO

&

& S. S

SSSSSSSSSS

410

O

SSSSSSSSSSSS

an Inters.
S
Šss

SSS

404

SSSSSSSSSSSSS

k's text i typed before
412a

&

406

'''''''''''' &s

412b

'

Sac

FIG. 4C

X.

410

SSS
SSSSSSSSS

S.

irrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

s

412a

is a

‘S’

-ss Š

SSSSSSSSSSSSSSSSSSSS

406

&S

412b

412C

ser

404

N
$:
...:

S.
Š

FIG. 4D N-op
EXHIBIT C
Page 157

EXHIBIT C
Page 158

U.S. Patent

Oct. 2, 2012

Sheet 9 of 10

US 8,279,173 B2

rt

Display a photo in a display
504

Provide a photo tagging mode for
editing tags for the displayed photo
For each tag used to tag a Subjector
object in the photo, identify the tag
type and aSSOciate a unique pointer
for each tag type to highlight the
Corresponding tagged Subjector
object

506

508

Display a tag list with the photo
510

Identify the tag type of each tag in the
tag list using a visual identifier
Upon user Selection of a tag in the tag
list, highlight theaSSOciated tagged
Subjector object in the photo using the

512

unique pointer

Upon user Selection of a tagged
Subjector object in the photo, highlight
the aSSOciated tag in the tag list

500
514

516

Displaying Context dataaSSOciated

with the tag

In the photo tagging mode, Constrain
the navigation device pointer to be
navigable only within the boundaries
of the photo
End
EXHIBIT C
Page 159

F G. 5
518

U.S. Patent

Oct. 2, 2012

Sheet 10 of 10

US 8,279,173 B2

Start
602

Providing a tag entry field for entering
a photo tag
Independence upon a string entered
by the user, displaying in a matching
tag list any tags from One or more
selected tag sources matching the
entered string

604

606

Displaying a tag type for each tag
appearing in the matching tag list
608

Orderingtags in the in the tag list by
tag type

Allowing user Selection of a tag in the
matching tag list to complete the tag
entry field
If there are no tags remaining in the
matching tag list, adding a new free
fom text string to a free-form text

610

612

Cache
End

600

FIG. 6
EXHIBIT C
Page 160

US 8,279,173 B2
1.
USER INTERFACE FOR SELECTINGA
PHOTO TAG
CROSS-REFERENCE TO RELATED
APPLICATIONS

This application is a continuation of U.S. application Ser.
No. 1 1/746,285 entitled “USER INTERFACE FOR

SELECTING A PHOTO TAG” and filed on May 9, 2007 now
U.S. Pat. No. 8,031,170. U.S. application Ser. No. 1 1/746.285
is fully incorporated by reference herein.

10

FIELD OF THE INVENTION

network 200, and device-resident functions such as a calcu

The present invention relates generally to a user interface
for selecting a photo tag.

15

BACKGROUND

Identifying people or objects in photographs is popular in
many online contexts, such as photo sharing, Social network
ing, etc. Selecting a “tag” to associate with an identified point
in a photograph can be a complicated task if there are many
potential tags to choose from. In addition, wireless mobile

25

communication device where there are constraints on the size

of the display and the flexibility of the input method, some of
these common techniques used on desktops and laptops with
full sized screens do not work as well.

What is needed is an improved user interface for selecting
tags in a smaller display, such as may be found ona wireless

30

mobile communication device.
BRIEF DESCRIPTION OF THE DRAWINGS
35

FIG. 1 is an illustration of a device in accordance with an

embodiment;

FIG. 2 is a schematic block diagram ofa communication
subsystem component in the device of FIG. 1;

40

FIGS. 3A to 3E are illustrative user interface screens for

editing photo tags in accordance with an embodiment;
FIGS. 4A to 4F are illustrative user interface screens for

selecting a photo tag in accordance with an embodiment;
FIG. 5 is an illustrative flowchart of a method for editing
photo tags in accordance with an embodiment; and
FIG. 6 is an illustrative flowchart of a method for selecting
a photo tag in accordance with an embodiment.
DETAILED DESCRIPTION

45

50

As noted above, the present invention relates to a user
interface for selecting a photo tag, particularly within a
Smaller display, Such as may be found on a wireless mobile
communication device.

55

In an illustrative embodiment, the invention may be prac

Once captured by the CCD or CMOS sensor array, the
image may then be processed by camera logic 128 into a
Suitable digital image file format such as Joint Photographic
Experts Group (JPEG), Tagged-Image File Format (TIFF),
Bit Mapping (BMP), different variations on these standard
image file formats, ora vendor proprietary RAW image for
mat. The image file format may allow for the addition of
image metadata to an image file in an industry standards
exchangeable image file format (EXIF), or in some vendor
proprietary meta-data format. The image file may then be
stored in available device storage such as RAM 106 or flash
memory 108, and displayed on display 110.
Still referring to FIG. 1, operating system software used by
the main processor 102 is typically stored in a persistent store
such as flash memory 108. Those skilled in the art will appre
ciate that the operating system, specific device applications,
or parts thereof, may be temporarily loaded into a volatile
store, such as the RAM 106, for processing by main processor
102.

The communication device 100 may send and receive com
munication signals over the wireless network 200 after
required network registration or activation procedures have
been completed. Network access may be associated with a

ticed with a handheld mobile communication device in a

wireless operating environment. Shown in FIG. 1 is a sche
matic block diagram of an illustrative handheld mobile com
munication device 100. The communication device 100 may
comprise a number of components, including a main proces
sor 102 which controls the overall operation of communica
tion device 100. Communication functions, including data
and Voice communications, may be performed through a
communication Subsystem 104. The communication Sub
system 104 may receive messagesfrom and send messages to

lator or task list. The trackball 117 may be used for various
navigation functions, such as navigating through a graphical
user interface (GUI) menu displayed on display 110. The
trackball 117 may also be configured with a secondary actua
tion feature, Such as allowing a user to depress the trackball,
to allow selection of a highlighted item.
The camera module 126 may be adapted to capture an
image through a lens onto a light sensitive image sensor Such
as a charge coupled device (CCD) sensor array or a comple
mentary metal oxide semiconductor (CMOS) sensor array.
The camera lens may bea fixed focus lens, or a variable focus
lens with or without Zoom features and controlled by camera
logic 128 to focus an image onto the CCD or CMOS sensor
array. The size and pixel density of the CCD or CMOS sensor
array may be Suitably selected for the image resolution
required for a particular application. Camera logic 128 may
also control the camera lens aperture and/or shutter speed by
incorporating a suitable light exposure meter. Image capture
using camera module 126 may initiated by a user controlling
a dedicated camera shutter, or a context dependent program
mable button or key (on keyboard 116, for example) that may
act as a camera shutter button.

In the figures which illustrate exemplary embodiments:

a wireless network 200.

2
The main processor 102 may also interact with additional
Subsystems such as a random access memory (RAM) 106, a
flash memory 108, a display 110, an auxiliary input/output
(I/O) subsystem 112, a data port 114, a keyboard 116, a
trackball 117, a speaker 118, a microphone 120, short-range
communications 122, a GPS Subsystem 124, a camera mod
ule 126, and associated camera logic 128.
Some of the subsystems of the communication device 100
may perform communication-related functions, whereas
other subsystems may provide “resident’ or on-device func
tions. By way of example, the display 110 and the keyboard
116 may be used for both communication-related functions,
Such as entering a text message for transmission over the

60

subscriber or user of the communication device 100.

The communication device 100 may be a battery-powered
device and may include a battery interface 132 for receiving
one or more rechargeable batteries 130. In some embodi
ments, the battery 130 may be a smart battery with an embed
65 ded microprocessor. The battery interface 132 is coupled to a
regulator (not shown), which assists the battery 130 in pro
viding power V+ to the communication device 100. The bat
EXHIBIT C
Page 161

US 8,279,173 B2
3
tery 130 may beused to power all components and modules in
the communication device 100, including the camera module
126 and associated camera logic 128.
The main processor 102, in addition to its operating system
functions, enablesexecution of various Software applications
134 on the communication device 100. A subset of software

applications 134 that control basic device operations, includ
ing data and Voice communication applications, will nor
mally be installed on the communication device 100 during
its manufacture.

10

The Software applications 134 may include a messaging
application 136. The messaging application 136 can be any
suitable software program that allows a subscriber or user of
the communication device 100 to send and receive wireless
text communications. Various alternatives exist for the mes

15

saging application 136 as is well known to those skilled in the
art. Messages that have been sent or received by the user are
typically stored in local storage such as flash memory 108 of
the communication device 100, or in some other suitable

storage element in the communication device 100. In an alter
native embodiment, some of the sent and received messages
may be stored remotely from the communication device 100
Such as in a data store ofan associated host system that the
communication device 100 communicates with. In an

embodiment, the messaging application 136 may include a
Message List user interface that is configured to allow a user
to see a list of message objects (i.e. email messages) in a

25

convenient list form. This will be described in detail further
below.

Still referring to FIG. 1, communication device 100 may
execute an Image Applications Module 148 that may be
operatively integrated with camera module 126, camera logic
128, main processor 102, RAM 106, display 110 and various
other modules and components to provide various image
application functions for the images captured by the camera
module 126. Image Applications Module 148 may include
various Sub moduleswhich may interact with each other, and
with other application modules such as the messaging appli

30

35

cation 136, Internet browser module 138,address book mod

ule 142, etc. in order to perform various functions.
In an embodiment, one of the Sub modules of the Image
Applications Module 148 may be a photo tagging module
148A configured to allow a user to identify various subjects
and objects within a photo. Photo tagging module 148A will

40

be discussed in more detail further below.

45

In an embodiment, another of the submodules of the Image
Applications Module 148 may be a photo tagging module
148A configured to allow a user to select a photo tag from a
list of tags associated with various Subjects and objects within
a photo. Photo tag selection module 148B will be discussed in

50

more detail further below.

The communication device 100 may further include a
device state module 140, an address book 142, a Personal

4
image applications 148. Display 110 may also be configured
to provide a touch-sensitive screen input in response to a
prompt or query displayed on display 110. The communica
tion device 100 may further include a camera lens that may be
used to capture an image as described above with reference to
FIG. 1. In an embodiment, the integrated camera 126 may
provide a camera lens 127 on the back of the communication
device 100, such that a user may use the display 110 as a
camera viewfinder for framing an image.
Now referring to FIG. 3A, shown is an illustrative user
interface screen 300A in which photo tagging module 148A
may be configured for tagging a photograph in accordance
with an embodiment. As shown, a photo 301 of a subject 302
is displayed within the boundaries of the user interface. With
this user interface, a tag list 304 may include various tags
associated subject 302 or other subjects or objects within the
photo 301. The user may click an “Add” button 306 in order
to enter a photo tagging mode as described below.
Now referring to FIG. 3B, as shown in screen 300B, once
in the photo tagging mode, the photo tagging module 148A
may be configured to display to the user a pointer, such as
cross-hair pointer 308 on the photo 301. The user may move
the cross-hair pointer 308 around the photo 301, but unlike
common web browser interfaces, the crosshair pointer 308
may be constrained by the photo tagging module 148A to be
navigable only within the boundaries of the photo 301. The
user does not have the ability to move the cross-hair pointer
around the rest of the screen and perform other tasks, and the
navigation device (e.g. trackball 117) is thus dedicated tothis
photo tagging function until the user exits tagging mode. As
will be appreciated, this may allow users to avoid inadvert
ently clicking on some button or menu option just outside of
the borders photo 301 when attempting to tag a subject or
object near an edge of the photo. This may be particularly
useful on a smaller display, where greater dexterity may be
required in order to navigate within a small area using a small
navigation device.
When in tagging mode, the user may cancel tagging mode
using the escape key and return to simply viewing the existing
tag information. Alternatively, the user may choose to add a
tag at the current location of the crosshairpointer 308 using an
enter key or clicking on the navigation device (e.g. trackball
117).
If the user chooses to add a tag, the user may be presented
with another user interface to select the tag type and the
context information associated with the tag. As an illustrative
example, the tag types could include a free-form alphanu
meric string, FacebookTM friends, address book entries (in
address book 142), browser bookmarks (in Internet browser
module 138), etc.
Now referring to FIG. 3C, as shown by way of illustration
in screen 300C, when a tag has been added by a user using
photo tagging module 148A, the added tag 310 associated
with subject 302 may appear in the tag list 304 near the photo

Information Manager (PIM) 144, and various other modules
150. Additional software applications may also be loaded 55 301.
onto the communication device 100 through at least one of the
In an embodiment, thetag 310 could also include an icon,
wireless network 200, the auxiliary I/O subsystem 112, the or some other type of visual indicator adjacent the tag 310,
data port 114, the short-range communications Subsystem indicating what type oftag it is. Thus, many different types of
tags may be used to tag subjects or objects in the photo 301.
122, or otherdevice subsystem 124.
Now referring to FIG. 2, shown is an illustrative front view 60 Now referring to FIG. 3D, as shown by illustration in
of a handheld mobile communication device 100 that may screen 300D, when the user scrolls over the tag 310 in the tag
provide a Suitable operating environment. As shown, the com list 304, the corresponding subject 302 in the photo may be
munication device 100 may include a display 110, a keyboard highlighted by the photo tagging module 148A in Some way.
116, and other input or navigation means such as a trackball This highlighting could be different, depending on what type
117. The display 110 may be configured to display various 65 of tag it is. For example, for a FacebookTM friend the pointer
screens allowing the user of device 100 to view screen outputs could be a grey square pointer 312 (e.g. like Facebook uses
from the various software applications 134, including the online). As another example shown in screen 300E of FIG.
EXHIBIT C
Page 162

US 8,279,173 B2
5
6
entry field 406. The matchingtag list 412 then disappears, and
3E, for a free-form text tag 314, the tagged point in the photo
the selected tag may appear beside the photo (e.g. tag 310 as
301 could be indicated with a circle pointer 316.
When the user scrolls over a tag 310 in the tag list 304, a shown in FIG. 3C associated with subject 302 in the photo
menu may be activated with options associated with the tag. 301).
For example, there may be menu items to edit or delete the 5 Significantly, as the matching tag list 412 includespossible
tag. There may also be context sensitive menu options asso tags that may be used from various selected tag sources (such
ciated with the specific tag type. For example, for a Facebook as the user's Facebook friends, the user's address book 142, a
friend there may be an item to view the friend's Facebook list of the user's browser bookmarks from Internet browser
profile. For an address book entry, there may be an item to
138, a cache of the recent
view the user's address card. For a browser bookmark, there 10 is provided with a simple free-form text entries, etc.), the user
way to associate Subjects or objects
may be an item to visit that website.
in a photo with a predefined “tag” from one of a number of
In another embodiment, once Subjects or objects have been
tagged in the photo 301, photo tagging module 148A may be selected tag sources, as may be defined by the user. Thus, the
free-form text cache would just be one of many possible tag
configured such that upon user selection of a tag310 in thetag
list 304 the corresponding tagged subject or object in the 15 Sources, all of which contribute matching tag entries to the
photo 301 may be highlighted using the corresponding matching tag list 412.
Now referring to FIG. 4F, once a tag has been entered into
pointer identifying the tag type.
In another embodiment, once subjects (e.g. subject 302) or the tag entry field 406, photo tag selection module 148B may
objects have been tagged in the photo 301, photo tagging be configured to allow the user to choose to view some con
module 148A may be configured such that upon user selec text data associated with the tag (e.g. an address card if the tag
tion ofor scrolling over a tagged Subject or object in the photo identifies a contact in the user's address book 142). Finally,
301, the corresponding tag 310 may be highlighted in the tag photo tag selection module 148B may be configured to allow
list 304. As well, context data associated with the tag may be the user to accept the new tag and return to the photo tagging
displayed (e.g. in a pop-up window) upon user selection, or user interface (described above with respect to FIGS. 3A to
25 3E), or cancel the tag selection and return to the photo tagging
automatically if so configured.
Thus, using the above described user interface, a plurality user interface.
oftag types may be used to tag subjects and objects in a photo,
In an embodiment, in addition to the tag sources mentioned
and a type-specific data may be associated with each tag— above, another type of tag source may be landmark tags with
Such as the visual indicator or symbol used to highlight the associated geographic location information. For example, ifa
tagged subject or object in the photo, the custom actions 30 photo contains a number of distinctive landmarks, it may be
possible for each landmark to be tagged with a unique geo
available for the tag, etc.
Now referring to FIG. 4A, shown in screen 400A is an graphic location tag (e.g. specific latitude and longitude coor
illustrative tag selection user interface 404 for displaying a dinates for each landmark). Such a list of geographic location
tagsearch facility as may be presented by photo tag selection tags may be obtained, for example, as a user visits each of the
module 148B. As shown in FIG. 4A, the user is initially 35 landmarks identified in the photo.
presented with a tag entry field 406 indicating that he should
For example, an aerial photo of the National Mall in Wash
start typing a tag. Upon completion of typing, the user may ington D.C. may show a number of famous landmarks such as
the Lincoln Memorial, Vietnam Veterans Memorial, and The
click “OK” 408 to select the tag.
In an embodiment, as the user begins to type, photo tag Washington Monument in the same photo. A user who has
selection module 148B may be configured to search one or 40 visited each of these landmarks, provided appropriate names,
more selected “tag sources' for tags that match the currently and recorded geographic location information at each land
entered text. As shown by way of illustration in screen 400B mark location may then Subsequently select a landmark tag
of FIG. 4B, thesetag sources could include, for example,a list by name from the prepared landmark tag list in order to tagthe
of friends from an online service like FacebookTM, a list of appropriate points in the photo at which each of the landmarks
contacts from the user's address book 142, a list of the user's 45 appear. Once a user tags each landmark appearing in the
browser bookmarks (in Internet browser 138), a cache of photo using the appropriate landmark tag, the corresponding
recent free-form text entries, etc.
geographic coordinates also become available as context
based information accessible through the tag.
As shown in screen 400C of FIG. 4C,and 400D of FIG.4D,
photo tag selection module 148B may be configured to dis
In an alternative embodiment, a list of famous landmarks
play any matching tags (e.g. 412a, 412b, 412c) from one of 50 for various cities may be prepared by a third party Such that a
the tag sources to the tag being typed by the user in the tag user need not be at each location to record the geographic
entry field 406 in a matching tag list 412. Each tag may have coordinates. In this case a landmark tag may be selected by a
an icon or some other visual identifier associated with it that
user froma prepared landmark list that may be selected as one
clearly indicates its type, and allows the user to quickly dis of the tag sources.
55
tinguish between different types of tags.
In the tag selection interface described above, each tag may
If the usertypes text that does not match any tag from the have some context associated with it, depending on the tag
tag sources in the matching tag list 412, the photo tag selec type. As will be appreciated, this associated context may
tion module 148B may create a new free-form tag entry and allow the user to visually identify and differentiate between
add it to a free-form text cache as a new tag entry. The the tag types in the tag selection user interface 404, and
free-form text cache may then become one of thetag sources 60 present other context sensitive menu options. Such as viewing
for any Subsequent tag selection by the user.
associated address cards, Facebook profiles, websites, geo
As the user continues to type, if a tag that the user wishes to graphic location information for landmarks, etc.
select appears in the matching tag list 412, the user can scroll
Now referring to FIG. 5, shown is a flowchart of an illus
to the tag in the matching tag list 412and select it by pressing trative method 500 in accordance with an embodiment. As
enter or clicking on the navigation device (e.g. trackball 117). 65 shown, method 500 begins and at block 502 where method
For example, as shown in screen 400E of FIG. 4E, the user 500 displays a photo in a display (e.g. display 110) to be
tagged.
may select a tag 412c which may then be placed into the tag
EXHIBIT C
Page 163

US 8,279,173 B2
7
Method 500 then proceeds to block504, where method 500
may configure photo tagging module 148A to provide a photo
tagging mode for editing tags for the displayed photo.
Method 500 may then proceed to block 506, where for each
tag used to tag a subject or object in the photo, method 500
may configure photo tagging module 148A to identify thetag
type and associate a unique pointer for each tag type to high
light the corresponding tagged Subjector object. For example,
as described earlier, a grey Square could be used for Facebook
friends, and another shape and/or color may be used for
address book tag types.
Method 500 may then proceed to block 508, where method
500 may configure photo tagging module 148A to display a
tag list with the displayed photo.
Method 500 may then proceed to block510, where method
500 may configure photo tagging module 148A toidentify the
tag typeof each tag in thetag list using a visual identifier. For
example, the visual identifier may be an icon, or some Suitable
graphic appearing beside a tag name.
Method 500 may then proceed to block 512, where upon
user selection of a tag in thetag list, method 500 may config
ure photo tagging module 148A to highlight the associated
tagged Subject or object in the photo using the unique pointer.
Method 500 may then proceed to block 514, where upon
user selection of a tag in thetag list, method 500 may config
ure photo tagging module 148A to highlight the associated
tag in the tag list.
Method 500 may also proceed to block 516, where method
500 may configure photo tagging module 148A to display
context data associated with a tag. For example, context data
associated with an address book type may be the email

10

15

25

form text.
30

address, name, and other information stored in the address

book module 138 for the corresponding address book entry.
In an embodiment, the context data associated with the tag
could be displayed within a menu that may be accessed by a
suitable menu selection associated with each tag. Method 500

35

then ends.

Method 500 may also proceed to block 518, where method
500 may configure photo tagging module 148A to constrain
the navigation device pointer to be navigable only within the
boundaries of the photo.
Now referring to FIG. 6, shown is a flowchart of an illus

40

trative method 600 in accordance with an embodiment. As

45

shown, method 600 begins and at block 602 where method
600 provides a tag entry field for entering a photo tag.
Method 600 may then proceed to block 604, where in
dependence upon a string entered by the user in thetag entry
field, method 600 may configure photo tag selection module
148B to display in a matching tag list 412 any tags (412.a.
412b, 412c) matching the entered string.
Method 600 may then proceed to block 606, where method
600 may configure photo tag selection module 148B to dis
play a tag type for each tag appearing in the matching tag list
412. For example, the visual identification may be an icon, or
other Suitable graphic identification appearing with the tag
aC.

8
Method 600 may then proceed to block 612, where if there
are not matching tags appearing in the matching tag list 412,
then method 600 may add the new free-from text tag to a
free-form text tag cache.
Method 600 may then proceed to block 614, where method
600 may configure photo tag selection module 148B to select
the free-form text tag cache as one of thetag sources for any
subsequent tag selection by the user. Method 600 then ends.
Thus, in accordance with an embodiment, there is provided
a method of selecting a photo tag for a tagged photo, com
prising: providing a tag entry field for entering a photo tag; in
dependence upon a string entered by a user, displaying in a
matching tag list any tags from one or more selected tag
Sources matching the entered string.
In an embodiment, the method further comprises display
ing a tag type for each tag appearing in the matching tag list.
In another embodiment, the method further comprises
ordering tags in the matching tag list by tag type.
In another embodiment, the method further comprises
allowing user selection of a tag in the matching tag list to
complete the tag entry field.
In another embodiment, the method further comprises add
ing a new free-form text string to a free-form text cache if
there are no tags remaining in the matching tag list.
In an embodiment, the method further comprises selecting
as a tag source one or more ofan online network profile, an
address book, browser bookmarks, landmark tags, and free
In another aspect of the invention, there is provided a
system for selecting aphoto tag for a tagged photo, compris
ing: means for providing a tag entry field for entering a photo
tag; means for displaying in a matching tag list, in depen
dence upon a string entered by a user, any tags from one or
more selected tag sources matching the entered string.
In an embodiment, the system further comprises means for
displaying a tag type for each tag appearing in the matching
tag list.
In another embodiment, the system further comprises
means for ordering tags in the matching tag list by tag type.
In another embodiment, the system further comprises
means for allowing user selection ofa tag in the matching tag
list to complete the tag entry field.
In another embodiment, the system further comprises
means for adding a new free-form text string to a free-form
text cache if there are no tags remaining in the matching tag
list.

50

55

In another embodiment, the system further comprises
means for selecting as a tag source one or more ofan online
network profile, an address book, browser bookmarks, land
mark tags, and free-form text.
In another aspect of the invention, there is provided a data
processor readable medium storing data processor code that
when loaded into a device adapts the device to select aphoto
tag for a tagged photo, the data processor readable medium
comprising: code for providing a tag entry field for entering a
photo tag; code for displaying in amatching tag list, indepen
dence upon a string entered by a user, any tags from one or
more selected tag sources matching the entered string.
In an embodiment, the data processor medium further com
prises code for displaying a tag type for each tag appearing in
the matching tag list.
In another embodiment, the data processor medium further
comprises code for ordering tags in the matching tag list by

Method 600 may then proceed to block 608, where method 60
600 may configure photo tag selection module 148B to order
the matching tags in the tag list by tag type in order to make
is easier for the user to select a Suitable tag by tag type.
Method 600 may also proceed to block 610, where method tag type.
600 may configure photo tag selection module 148B to allow 65 In another embodiment, the data processor medium further
user selection of a tag in the matching tag list to complete the comprises code for allowing user selection of a tag in the
matching tag list to complete thetag entry field.
tag entry field 406.
EXHIBIT C
Page 164

US 8,279,173 B2
In another embodiment, the data processor medium further
comprises code for adding a new free-form text string to a
free-form text cache if there are no tags remaining in the
matching tag list.
In another embodiment, the data processor medium further
comprises code for selecting as a tag source one or more of an
online network profile, an address book, browser bookmarks,
landmark tags, and free-form text.

5

While illustrative embodiments have been described

above, it will be appreciated that various changes and modi
fications may be made. More generally, the scope of the
invention is defined by the following claims.

10

We claim:

1. A method of selecting a photo tag for a tagged photo,
comprising:
displaying a tag list including tags from one or more tag
Sources matching a search String;
displaying a tag type indicator for each tag appearing in the
tag list, said tag type being indicative of a tag source
associated with the tag.
2. The method of claim 1, further comprising providing a
tag entry field for entering the search String.
3. The method of claim 1, further comprising ordering tags
in thetag list by tag type.
4. The method of claim 1, further comprising providing a
graphical user interface to select a tag in the tag list.
5. The method of claim 1, further comprising:
adding a new free-form text string to a free-form text cache
based upon the search String.
6. The method of claim 1, further comprising selecting as a
tag source one or more of an online network profile, an
address book, browser bookmarks, landmark tags, and free

15

25

30

form text.

7. A system for selecting a photo tag for a tagged photo,
comprising:
an apparatus operable to display a tag list including tags
from one or more tag sources matching a search string;
an apparatus operable to display a tag type indicator for
each tag appearing in the tag list, said tag type being
indicative of a tag source associated with the tag.
8. The system of claim 7, further comprising an apparatus
operable to provide a tag entry field for entering a photo tag.

35

10
9. The system of claim 7, further comprising an apparatus
operable to order tags in thetag list by tag type.
10. The system of claim 7, further comprising an apparatus
operable to allow selection of a tag in thetag list to complete
the tag entry field.
11. The system of claim 1, further comprising:
an apparatus operable to adda new free-form text string to
a free-form text cache based upon the search String.
12. The system of claim 1, further comprising an apparatus
operable to select as a tag source one or more ofan online
network profile, an address book, browser bookmarks, land
mark tags, and free-form text.
13. A computer readable medium storing computer code
that when loaded into a device adapts the device to select a
photo tag for a tagged photo, the computer readable medium
comprising:
code for displaying a tag list including tags from one or
more tag sources matching a search String;
code for displaying a tag type indicator for each tag appear
ing in the tag list, said tag type being indicative of a tag
Source associated with the tag.
14. The computer readable medium of claim 13, further
comprising code for providing a tag entry field for entering
the search string.
15. The computer readable medium of claim 13, further
comprising code for ordering tags in the tag list by tag type.
16. The computer medium of claim 13, further comprising
code for providing a graphical user interface to select a tag in
the tag list to complete the tag entry field.
17. The computer medium of claim 13, further comprising:
code for adding a new free-form text string to a free-form
text cachebased upon the search string.
18. The computer readable medium of claim 13, further
comprising code for selecting as a tag source one or more of
an online network profile, an address book, browser book
marks, landmark tags, and free-form text.
19. The computer readable medium of claim 13, further
comprising code for displaying context data with the photo
tag.

40

20. The computer readable medium of claim 19, wherein
the context data is based on information in an address book.

EXHIBIT C
Page 165

k

k

k

k

k