Motorola Mobility, Inc. v. Microsoft Corporation

Filing 125

NOTICE by Microsoft Corporation re #124 Defendant's MOTION Claim Construction of Filing Exhibits (Attachments: #1 Appendix (Index to Exhibits), #2 Exhibit 101, #3 Exhibit 102, #4 Exhibit 201, #5 Exhibit 301, #6 Exhibit 302, #7 Exhibit 304, #8 Exhibit 305, #9 Exhibit 501, #10 Exhibit 502, #11 Exhibit 504, #12 Exhibit 601, #13 Exhibit 801 (Part 1), #14 Exhibit 801 (Part 2), #15 Exhibit 901, #16 Exhibit 902, #17 Exhibit 903, #18 Exhibit 904, #19 Exhibit 905, #20 Exhibit 1101, #21 Exhibit 1102, #22 Exhibit 1105, #23 Exhibit 1110, #24 Exhibit 1114, #25 Exhibit 1115, #26 Exhibit 1116, #27 Exhibit 1201, #28 Exhibit 1301, #29 Exhibit 1302, #30 Exhibit 1303, #31 Exhibit 1304, #32 Exhibit 1305, #33 Exhibit 1401, #34 Exhibit 1402, #35 Exhibit 1403, #36 Exhibit 1404, #37 Exhibit 1405, #38 Exhibit 1406, #39 Exhibit 1407, #40 Exhibit 1408, #41 Exhibit 1409)(Miner, Curtis)

Exhibit 101

111111

1111111111111111111111111111111111111111111111111111111111111
US006791536B2

(54)

United States Patent

(10)

Keely et a1.

(12)

(45)

SIMUIATING GESTURES OF A POINTING
DEVICE USING A STYLUS AND PROVIDING
FEEDBACK THERETO

(75)

Notice:

(21)

Filed:

4,899,138
5,404,458
5,592,566
5,602,570
5,000,113
5,864,635
5,880,411
0,049,329
6,262,719
6,266,050

A *
A *
A *
A *
A *
A *
A *
A *
B1 *
B1 *

2/1990
4/1995
1/1997
2/1997
9/1997
1/1999
3/1999
4/2000
7/2001
7/2001

Araki et al. ................ 345/175
Zetts
710/73
Pagallo et al. .............. 382/187
Capps ct al. ................ 345/173
Logan ......................... 341/34
Zetts et al. ................. 382/187
Gillespie et al. ......... 178/18.01
Zetts et al. ................. 345/179
Bi et al. ..................... 345/179
Oh et al. .................... 345/173

App!. No.: 09/815,272

(22)

References Cited
U.S. PATENT DOCUMENTS

Assignee: Microsoft Corporation, Redmond, WA
(US)

(*)

(56)

Inventors: Leroy B. Keely, Portola Valley, CA
(US); David F. Jones, Redmond, WA
(US); David Switzer, Redmond, WA
(US); Michael Hin-cheung Tsang,
Bellevue, WA (US); William Hong
Vong, Seattle, WA (US)

(73)

Patent No.:
US 6,791,536 B2
Date of Patent:
Sep.14,2004

* cited by examiner
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.c. 154(b) by 69 days.

Primary Examiner-Henry N. Tran
(74) Attorney, Agent, or Firm-Banner & Witcoff, Ltd.
(57)

(65)

Mar. 23, 2001
Prior Publication Data

US 2002/0057263 A1 May 16, 2002

Related U.S. Application Data
(60)

Provisional application No. 60/247,844, filed on Nov. 10,
2000.

(51)
(52)

Int. CI.7 .................................................. G09G 5/00
U.S. CI........................ 345/173; 345/161; 345/163;
345/168; 345/173; 345/179; 345/863
Field of Search ................................. 345/173, 156,
345/161, 163, 167, 168, 179, 863; 178/19.01,
19.04,20.01; 703/17, 19, 21-24

(58)

ABSTRACT

A method and apparatus for simulating at least one gesture
of a pointing device such as a mouse. A left click, right click,
left drag, right drag, and/or mouse movement may be
simulated using a stylus in conjunction with a touchsensitive display surface. For example, a computer having
the display surface may detect whether a stylus is being held
down on a touch-sensitive display surface for at least a
threshold amount of time. The computer may further detect
whether the stylus is then removed from the touch-sensitive
display surface after at least the threshold amount of time.
Responsive to the stylus being removed, the computer may
generate at least one event representing a right mouse button
being pressed.
47 Claims, 4 Drawing Sheets

MS-MOTO_ SDFLA_00000770888

e
.
rJJ.
.

166

_S~~T§~ ~§.~O!~ _I
(ROM) 140
I

1~~

BIOS

~

~

100

1120

~
"""'"

I DIGITIZER

,--

I

-------(RAM) 150
IOPERATING 195
SYSTEM

PROCESSING
UNIT

110

VIDEO
ADAPTER

I

108

APPLICATION 196' ,
PROGRAMS
I I

I

165

16
107

=
"""'"
rJj
~

'!=i

....

~~

N

o
o

~

OTHER
PROGRAM
MODULES

197
MAGNETIC
DISK DRIVE
INTERFACE

HARD
DISK
INTERFACE
PROGRAM
DATA

I

~170

/...-/

s::
en
s::

~-

///

.,.../',

~80

"..,./......

t

""" ; - 0
e 190
,

11

0

",

b

~9111

t

192

OTHER
OPERATING
APPLICATION 196 PROGRAM 1971PROGRAM 198
195 PROGRAMS
SYSTEM
MODULES
DATA

,0

en

o1
"'T

r

,:t:o
o
o
o
o
......
......
o
CO
CO
CD

rJj

=....

~
~

.....

o
....,
~

114

1;1

a..4Il'""\r
IVlvDEM

WIDE
AREA
NETWORK

KEYBOARD

10

~

00

MOUSE

I

o
-I

NETWORK
INTERFACE
112

> __

--

LOCAL AREA NETWORK

SERIAL
PORT
INTERFACE

198

I

...-/'--/

OPTICAL
DRIVE
INTERFACE

FIG. 1

APPLICATION
PROGRAMS

111

-..~

.......

'-0
~

U.
w
~

Cd

N

u.s. Patent

Sep.14,2004

Sheet 2 of 4

US 6,791,536 B2

202

/

204

201

~\ - - - - - - - - 1 - - - - - - - - - - - - - - - - - - - - - - -J

203

FIG. 2

MS-MOTO_SDFLA_00000770890

u.s.

Patent

Sep.14,2004

US 6,791,536 B2

Sheet 3 of 4

314
LEFT DRAG
>---------------YES--------------~

304

313

FIG. 3

MS-MOTO_SDFLA_00000770891

u.s. Patent

Sep.14,2004

US 6,791,536 B2

Sheet 4 of 4

400

\

401

402

FIG. 4

MS-MOTO_ SDFLA_00000770892

US 6,791,536 B2

1

2

operahility as well. The user may generate other gestures
using the mouse such as by clicking the left or right button
of the mouse and moving the mouse while the button is
depressed (a gesture commonly referred to as either a "left
CROSS-REFERENCE TO RELATED
5 drag" or a "right drag," respectively).
APPLICATIONS
While such mouse gestures are of course easily done with
an actual mouse, it is not always convenient for a user to use
The present application claims priority to u.s. provisional
a mouse with a computer even though the computer may
patent application Serial No. 60/247,844, entitled Simulathave a GUI system. For instance, when using a portable
ing Gestures of a Mouse Using a Stylus and Providing
Feedback Thereto, filed Nov. 10, 2000, which is hereby 10 system there may not always be a surface available upon
which to place a mouse. Some systems utilize a touchincorporated by reference as to its entirety. The present
sensitive display screen with a stylus as an input device. The
application is also related to application Ser. No. 09/760,
user places the stylus tip upon the touch-sensitive display
631, entitled Method and Apparatus For Improving the
screen at various locations to control objects displayed on
Appearance of Digitally Represented Handwriting, filed
Nov. 10,2000; to application Ser. No. 60/247,841, entitled 15 the display screen. A problem with this type of system is that
the stylus is limited in the types of actions that may be easily
Highlevcl Active Pen Matrix, filed Nov. 10, 2000; to appliperformed as compared with a mouse. For instance, the
cation Ser. No. 60/247/973, entitled Selection Handles in
typical stylus can perform only three types of movements:
Editing Electronic Documents, filed Nov. 10, 200; to appliplacing the stylus tip onto the screen, moving the stylus tip
cation Ser. No. 60,247,842, entitled Insertion Point Bungee
Space Tool, filed Nov. 10, 2000; to application Ser. No. 20 across the screen, and removing the stylus tip from the
screen. While a user may operate a remote toolbar to select
60/247,400, entitled System and Method for Accepting
a different tool, the excessive movement renders the control
Disparate Types of User Input, filed Nov. 10, 2000; to
of the system laborious, inconvenient, and slow. Using the
application Ser. No. 60/247,972, entitled In Air Gestures,
tip of the stylus-type device, there is presently no convenient
filed Nov. 10, 2000; to application Ser. No. 60/247,831,
entitled Mouse Input Panel Windows Class List, filed Nov. 25 way to simulate or distinguish between, for example, a right
click as opposed to a left click as interpreted by the operating
10,2000; to application Ser. No. 60/247,843, entitled Mouse
system.
Input Panel and User Interface, filed Nov. 10, 2000; to
In order to make the full range of interface features
application Ser. No. 60/247,479, entitled System and
accessible to users of such a stylus-based computer, there is
Method for Inserting Implicit Page Breaks, filed Nov. 10,
2000; to application Ser. No. 09/750,288, entitled 30 a need for an intuitive way of simulating mouse gestures
with the stylus. Although not necessary, it is preferable and
Classifying, Anchoring, and Transforming Ink, filed Dec.
more useful that any new way of simulating mouse gestures
29, 2000; to application Ser. No. 09/730,170, entitled
be compatible with existing software applications that conHighlevel Active Pen Matrix, filed Dec. 15, 2000; and to
ventionally are used with a mouse.
application Ser. No. 09/741,107, entitled Mode Hinting/
Switching, filed Dec. 21, 2000; each of which is incorpo- 35
There is also a need for providing helpful feedback to the
rated by reference herein as to their entireties.
user of the stylus to indicate whether gestures made with the
stylus are those that are intended by the user. This is useful
FIELD OF THE INVENTION
with a stylus-type input device since there is conventionally
The present invention is directed generally to apparatus 40 little to no natural tactile or auditory feedback when using a
stylus than there is when using a mouse (e.g., the buttons of
and methods for simulating various gestures of a pointing
a mouse often have a definitive mechanical "click" feel and
device with multiple switches, such as a two button mouse
sound when pressed). As will be seen, such feedback is even
in a computer system, and for providing feedhack thereto,
more useful when provided in conjunction with various
and more specifically to simulating gestures such as the left
click of a mouse, the right click of a mouse, and mouse 45 gesture simulation techniques of the present invention.
dragging by manipulation of a stylus in conjunction with a
SUMMARY OF THE INVENTION
touch-sensitive computer display, as well as generating
appropriate visual or other feedback in response to certain
As discussed in the various copending patent applications
gestures.
incorporated herein by reference, aspects of the present
50 invention are directed to a tablet-like computer that may be
BACKGROUND OF THE INVENTION
used for directly writing on a touch-sensitive display surface
using a stylus. The computer may allow the user to write and
'lypical computer systems, especially computer systems
to edit, manipulate, and create objects through the use of the
using graphical user interface (GUI) systems such as
stylus. Many of the features discussed in these copending
Microsoft WINDOWS, are optimized for accepting user
input from two discrete input devices: a keyboard for 55 applications are even more useful in combination with
various aspects of the present invention discussed herein.
entering text, and a pointing device such as a mouse with
An aspect of the present invention is directed to methods
two or more buttons or other selectors for driving the user
and apparatus for simulating gestures of a pointing device,
interface. Virtually all software applications designed to run
on Microsoft WINDOWS are optimized to accept user input
such as a mouse, trackhall, or joystick, hy use of a stylus on
in the same manner. For instance, many applications make 6U a touch-sensitive display surface. For example, a left click of
extensive use of a primary switch of the pointing device such
a mouse may be simulated, e.g., where the user holds the
as the left button of a mouse (a gesture commonly referred
stylus down on the touch-sensitive display surface without
to as a "left click" of a mouse) to select an object, and a
substantial movement and then removes the stylus from the
secondary switch such as the right button of the mouse (a
display surface before the expiration of a threshold amount
gesture commonly referred to as a "right click" of a mouse) 65 of time. A right click of a mouse may be simulated, e.g.,
to display context-sensitive command menus. It is noted that
where the user waits until at least the threshold amount of
time before removing the stylus. Where the stylus is moved
other operating systems incorporate left and right click

SIMUIATING GESTURES OF A POINTING
DEVICE USING A STYLUS AND PROVIDING
FEEDBACK THERETO

MS-MOTO_ SDFLA_00000770893

US 6,791,536 B2

3

4

along the display surface during certain stylus gestures, left
implement various aspects of the invention. Tn this
embodiment, computer 100 includes a processing unit 110,
or right dragging may thereby be simulated.
a system memory 120, and a system bus 130 that couples
Another aspect of the present invention is directed to
various system components including the system memory to
methods and apparatus for providing feedback to a user in a
stylus-based touch-sensitive display surface computer sys- 5 the processing unit 110. The system hus 130 may he any of
several types of bus structures including a memory bus or
tem. The user may accordingly be given some indication that
memory controller, a peripheral bus, and a local bus using
the user has performed a particular stylus gesture, e.g., by
any of a variety of bus architectures. The system memory
displaying a visual object on the display surface to indicate
includes read only memory (ROM) 140 and random access
a particular status. Such feedback may be responsive to
certain stylus gestures such as pressing and holding the 10 memory (RAM) 150.
A basic input/output system 160 (BIOS), containing the
stylus against the display surface for at least a certain
basic routines that help to transfer information between
minimum amount oftime. More generally, the feedback may
elements within the computer 100, such as during start-up,
be in the form of a "state change indicator," which may be
is stored in ROM 140. Computer 100 also includes a hard
visual on the display surface such as an icon, bitmap, or
other visual object or indicator, and/or auditory such as a 15 disk drive 170 for reading from and writing to a hard disk
(not shown), a magnetic disk drive 180 for reading from or
sound emitted from the computer's speakers. The state
writing to a removable magnetic disk 190, and an optical
change indicator may be animated and or otherwise vary
disk drive 191 for reading from or writing to a removable
with stylus hold time. Significant advantages of this aspect
optical disk 192 such as a CD ROM or other optical media.
of the invention are realized in conjunction with the various
features discussed in the copending patent applications 20 The hard disk drive 170, magnetic disk drive 180, and
optical disk drive 191 are connected to the system bus 130
incorporated herein by reference.
by a hard disk drive interface 192, a magnetic disk drive
These and other features of the invention will be apparent
interface 193, and an optical disk drive interface 194,
upon consideration of the following detailed description of
respectively. The drives and their associated computerpreferred embodiments. Although the invention has been
25 readable media provide nonvolatile storage of computer
defined using the appended claims, these claims are exemreadable instructions, data structures, program modules and
plary in that the invention is intended to include the elements
other data for the personal computer 100. It will be appreand steps described herein in any combination or subcomciated by those skilled in the art that other types of computer
bination. Accordingly, there are any number of alternative
readable media which can store data that is accessible by a
combinations for defining the invention, which incorporate
30 computer, such as magnetic cassettes, flash memory cards,
one or more elements from the specification, including the
digital video disks, Bernoulli cartridges, random access
description, claims, and drawings, in various combinations
memories (RAMs), read only memories (ROMs), and the
or subcombinations. It will be apparent to those skilled in the
like, may also be used in the example operating environrelevant technology, in light of the present specification, that
ment.
alternate combinations of aspects of the invention, either
A number of program modules can be stored on the hard
alone or in combination with one or more elements or steps 35
disk, magnetic disk 190, optical disk 192, ROM 140 or
defim:d herein, may be utiliLed as modifications or alterRAM 150, including an operating system 195, one or more
ations of the invention or as part of the invention. It is
application programs 196, other program modules 197, and
intended that the written description of the invention conprogram data 198. A user can enter commands and infortained herein covers all such modifications and alterations.
40 mation into the computer 100 through input devices such as
BRIEF DESCRIPTION OF THE DRAWINGS
a keyboard 101 and pointing device 102. Other input devices
(not shown) may include a microphone, joystick, game pad,
The foregoing summary of the invention, as well as the
satellite dish, scanner, or the like. These and other input
following detailed description of preferred embodiments, is
devices are often connected to the processing unit 110
hetter understood when read in conjunction with the accompanying drawings, which are included by way of example, 45 through a serial port interface 106 that is coupled to the
system bus, but may be connected by other interfaces, such
and not by way of limitation with regard to the claimed
as a parallel port, game port or a universal serial bus (USB).
invention. In the accompanying drawings, elements are
A monitor 107 or other type of display device is also
labeled with three-digit reference numbers, wherein the first
connected to the system bus 130 via an interface, such as a
digit of a reference number indicates the drawing number in
which the element is first illustrated. The same reference 50 video adapter lOR. Tn addition to the monitor, personal
computers typically include other peripheral output devices
number in different drawings refers to the same element.
(not shown), such as speakers aod printers.
FIG. 1 is a schematic diagram of a general-purpose digital
The computer 100 can operate in a networked environcomputing environment that can be used to implement
ment using logical connections to one or more remote
various aspects of the invention.
FIG. 2 is a plan view of a tablet computer and stylus that 55 computers, such as a remote computer 109. Remote computer 109 can be a server, a router, a network PC, a peer
can be used in accordance with various aspects of the present
device or other common network node, and typically
invention.
includes many or all of the elements described above
FIG. 3 is a flow chart showing an exemplary set of steps
relative to computer 100, although only a memory storage
that may be performed in order to simulate a right click of
6U device 111 has been illustrated in FIG. 1. The logical
a mouse according to aspects of the present invention.
connections depicted in FIG. 1 include a local area network
FIG. 4 is a plan view of an illustrative embodiment of a
(LAN) 112 and a wide area network (WAN) 113. Such
trackball having left and right buttons.
networking environments are commonplace in offices,
DETAILED DESCRIPTION OF PREFERRED
enterprise-wide computer networks, intranets and the InterEMBODIMENTS
65 net.
When used in a LAN networking environment, the comFIG. 1 shows an exemplary embodiment of a generalpurpose digital computing environment that can be used to
puter 100 is coonected to the local network 112 through a

MS-MOTO_ SDFLA_00000770894

US 6,791,536 B2

5

6

network interface or adapter 114. When used in a WAN
milliseconds. However, the threshold amount oftime may he
networking environment, the personal computer 100 typiany amount of time, such as between 300 and 600
cally includes a modem 115 or other means for establishing
milliseconds, or between 600 milliseconds and 1 second, or
a communications over the wide area network 113, such as
greater than 1 second. If the computer 201 detects that the
the Internet. The modem 115, which may he internal or 5 stylus moves prior to the timeout condition occurring (step
external, is connected Lo the system bus 130 via the serial
302), then in response the computer 201 may gennate a
port interface 106. In a networked environment, program
standard Microsoft WINDOWS LcftMouseBultonDown
modules depicted relative to the personal computer 100, or
event (or other event that represents the primary switch of
the pointing device being activated) in the original position
portions thereof, may be stored in the remote memory
10 where the stylus 204 was first brought down (step 314),
storage device.
which IIIay be sent tu the suftware applicatiun presently in
It will be appreciated that the network connections shown
use. At this point, any subsequent movement of the stylus
are example and other means of establishing a communica204 across the display surface 202 may be detected by the
tions link between the computers can be used. The existence
computer 201 (step 315), and in response the computer 201
of any of various well-known protocols such as TCP/IP,
Ethernet, FTP, HTTP and the like is presumed, and the 15 may generate one or more Microsoft WINDOWS MouseMove events (step 316) (or other events representing movesystem can be operated in a client-server configuration to
ment of the pointing device). When the stylus 204 is
permit a user to retrieve web pages from a web-based server.
eventually removed from the display surface 202 (hereafter
Any of various conventional web browsers can be used to
referred to as "bringing the stylus up") (step 317), the
display and manipulate data on web pages.
FIG. 2 illustrates a tablet PC 201 that can be used in 20 computer 201 may detect this and in response may generate
a Microsoft WINDOWS LeftMouseButlonUp event (step
accordance with various aspects of the present invention.
318) (or other event that represents the primary switch heing
Any or all of the features, subsystems, and functions in the
deactivated). Thus, via steps 301, 302, and 314-318, a left
system of FIG. 1 can be included in the computer of FIG. 2.
drag of a mouse is simulated.
Tablet PC 201 includes a large touch-sensitive display
To simulate a left dick of a mouse without dragging, if the
surface 202, e.g., a digitizing flat panel display, for example, 25
computer 201 detects that the stylus 204 is not moved prior
a liquid crystal display (LCD) screen, on which a plurality
to the timeout condition (step 302), and if instead the stylus
of windows 203 is displayed. Using a stylus 204, a user can
204 is brought up prior to the timeout condition (step 303),
select, highlight, and write on the digitizing display area.
then in response the computer 201 may generate first a
Examples of suitable digitizing display panels include electromagnetic pen digitizers, such as the Mutoh or Wacom pen 30 LeftMouseButtonDown event (or other event that represents
the primary switch being activated), then optionally delay
digitizers. Other types of pen digitizers, e.g., optical
for a period of time (step 320) such as about 20 milliseconds
digitizers, may also be used. Tablet PC 201 interprets marks
or some other amount of time, and then generate a Lcftmade using stylus 204 in order to manipulate data, enter text,
MouseButtonUp event (steps 304 and 318) (or other event
and execute conventional computer application tasks such as
spreadsheets, word processing programs, and the like. The 35 that represents the primary switch being deactivated). Thus,
via steps 301-304, 318, and 320, a slightly delayed left click
touch-sensitive display surface 202 may also be proximityof a mouse is simulated.
sensitive such that the tip of the stylus 204 may be sensed by
the display surface 202 when proximate thereto without
To simulate a right click of a mouse without dragging, if
actually contacting the display surface 202.
the computer 201 detects that the stylus 204 is not moved
The stylus 204 may be equipped with or without buttons 40 either prior to the timeout condition or after the timeout
condition (steps 302 and 305), and if the computer 201
or other features to augment its selection capabilities. In one
further detects that the stylus 204 is not brought up until after
embodiment, the stylus 204 may be implemented as a
the timeout condition (steps 3U3 and 306), and thus that the
"pencil" or "pen", in which one end constitutes a writing, or
stylus 204 has been held down for at least the threshold
tip, portion and the other end constitutes an "eraser" end
that, when moved across the display indicates portions of the 45 amount of time, then in response the computer 201 may
generate first a Microsoft WINDOWS RightMouseButtondisplay that are to be erased. The term "stylus" as used
Down event (step 307) (or other event that represents the
herein is intended to have a broad definition that includes,
secondary switch of the pointing device being activated).
e.g., a uscr's own finger, a pen, a pencil, a stick, a rod, and/or
The computer 201 may, further in response, optionally delay
any other similar device suitable for pointing at a particular
location on the display surface 202. Other types of pointing 50 for a period of time (step 308), in one embodiment, about 20
milliseconds, and then generate a Microsoft WINDOWS
devices, such as a mouse, trackball, or the like may be used
RightMouseBullonUp event (skp 313) (ur other event that
with the tablet PC 201 in addition to or conjunction with the
represents the secondary switch being deactivated). The
stylus 204.
computer 201 may also generate and display a state change
Referring now to FIG. 3, an example of how a left drag
of a mouse is simulated is now discussed. Any or all of steps 55 indicator (step 319) that indicates to the user that the stylus
204 has been placed down for a certain minimum amount of
set forth herein and in FIG. 3 may be performed by special
time, as will be discussed below in more detail.
software, hardware, and/or firmware that runs in conjunction
It is noted that steps 319 may be in a variety of places
with, or is part of, the operating system of the computer 20l.
within the flowchart of FIG. 3 other than the one shown,
To simulate a left drag, the user may touch the stylus 204 or
other similar user pointing device to the display surface 202, 6U such as between steps 305 and 306. Steps 301-303,
305-308, and 313, thus simulate a delayed right click of a
an action referred to hereafter as "placing the stylus down"
mouse.
(step 301). Responsive to the computer 201 detecting the
To simulate a right drag of a mouse, if the computer 201
stylus being placed down, the computer 201 may begin
counting time, e.g., by using a timer, up to at least a
detects that the stylus 204 is not moved until after the
threshold amount of time. A timeout condition is defined 65 timeout condition (step 305), then in response the computer
responsive to the threshold amount of time having passed. In
201 may generate a RightMouseButtonDown event (step
one embodiment, the threshold amount of time is about 600
309) (or other event that represents the secondary switch

MS-MOTO_ SDFLA_00000770895

US 6,791,536 B2

7

8

heing activated). Upon the computer 201 detecting any
feel the click of a mouse hutton to ensure that a click has
substqut:nt movemmt of the stylus 204 along tht: display
bet:n pt:rformed, tht: ust:r of the stylus 204 may not rt:ceive
surface 202 (step 310), the computer 201 may in response
clear enough feedback that the intended gesture has been
generate one or more MouseMove events (step 311) (or
made properly. Also, it is desirable to indicate to the user that
other events representing movement of the pointing device). 5 a particular requested mode has really heen switched to,
Wht:n the computer dett:cts that the stylus 204 is brought up
such as sekction mode. Further, wht:re the user must hold
(312), in rcsponse the computer 201 may generate a Rightdown the stylus 204 for at least a ccrtain thrcshold amount
of time to simulate, for example, a right click, feedback to
MouseButtonUp event (step 313) (or other event that repthe user indicating that the threshold time has passed would
resents the secondary switch being deactivated). This
simulates, via steps 301-303,305, and 309-313, a right drag 10 be useful.
of a mouse.
Such feedback may be in the form of a state change
indicator, which may be visual (e.g., a graphical icon, a
As a further feature of the right click simulation, a special
symbol, a bitmapped or other image, etc., displayed on the
hold event may be defmed wherein the stylus 204 is held
display surface 202 such as state change indicator 205 in
against the display screen 202 without substantial movement
for at least another threshold amount of time, say about 700 15 FIG. 2), auditory (e.g., a click, a beep, tone, etc. emanating
from the computer 201 and/or stylus 204), and/or tactile
milliseconds, or between 0.5 and 1 second, or between about
(e.g., vibration of the computer 201 and/or stylus 204). For
1 and 3 seconds or more, the computer 201 may detect such
example, where the user selects the pen icon in the applia hold event and in response act by immediately interpreting
cation toolbar (as described in at least one of the co-pending
any subsequent holding of the stylus 204 as holding the left
mouse button down (or, more generally, as activating the 20 patent applications incorporated herein by reference), the
user may drag the stylus 204 over a document displayed on
primary switch of the pointing device). Thus, the computer
the display surface 202 such that digital ink is generated on
201 may, after 600 milliseconds of stylus holding, go on to
the display surface 202. At this point, the user may switch to
steps 305 and 319, and where the stylus 204 is released prior
selection mode by pressing and holding the stylus 204 for at
to, say, 5 seconds or some other time limit, then steps
306-308 and 313 may be implemented. However, if instead 25 kast a thrt:shold amount of timt: that may bt: the same as the
threshold amount of time or that may be different. For
the stylus 204 is further held at least 5 seconds, then the
instance, the threshold minimum amount of time that the
special subroutine may, in response, immediately generate a
user must hold the stylus 204 against the display surface 202
LeftMouseButtonDown event (or other event that represents
may be, in one embodiment, about 400 milliseconds, or
the primary switch of the pointing device being activated).
Then, when the stylus is eventually released, a LeftMouse- 30 between about 400 milliseconds and about 1 second, or more
than about 1 second. Responsive to the user holding without
ButtonUp event (or other event that represents the primary
substantial movement the stylus 204 against the display
switch being deactivated) would be generated by the special
surface 202 for the minimum threshold amount of time, the
subroutine. If the stylus 204 is moved along the display
computer 201 may generate a state change indicator on the
surface 202, then the computer 201 may immediately in
response generate one or more MouseMove events (or other 35 display surface 202 which may be located at a location
depending upon the location of the tip of the stylus 204. For
events representing movement of the pointing device).
example, the state change indicator may be located at or near
When the stylus 204 is eventually removed from the display
the location of the tip of tht: stylus 204.
surface 202, all pending RightMouse events (or other events
that rt:present action by the stcomlary switch of the pointing
In some t:mbodiments, tht: statt change indicator may be
device) that would normally have been generated in accor- 40 animated and/or may otherwise change over time. For
instance, the state change indicator 205 is shown as a
dance with the method shown in FIG. 3 would be canceled.
partially completed clock-like loop that may become more
In the above and remaining discussion, it is assumed that
complete over time in, e.g., a clockwise direction. In such
a Microsoft WINDOWS operating system is being used with
embodiments with animation, the state change indicator may
the computer 201. However, any operating system, especially a GUI operating system, may be used in accordance 45 actually begin prior to the threshold amount of time passing,
as an indication to the user that the threshold amount of time
with the present invention. Further, although the above
will soon pass. For instance, where a first threshold amount
embodiments are sample methods for simulating mouse
of time is 1 second, holding the stylus 204 against the
gcstures using a stylus, thcre arc many variations that are
display surface 202 for 500 milliseconds may cause an icon
encompassed by the present invention. For instance, the
stylus gestures required for simulation a left elick and a right 50 (a state change indicator) in the shape of a wand tip to appear
on the display surface 202 at or near the location of the stylus
click might be reversed, or a user may not need to always
204. As the stylus 204 is continued to be held, tht: icon may
keep the stylus 204 down on the display surface 202 during
animate and begin to glow to indicate that a gesture is
the entire time that a drag is executed, such that a first
beginning to form. At the first threshold amount of time, the
touch/touch-and-hold of the stylus 204 at a first location on
the display surface 202, and a second touch/touch-and-hold 55 animation may have reached the point where the wand tip is
fully glowing. Moreover, at the passing of the first threshold
at a second location, may thereby simulate a drag from the
of time, the user may thereby be placed in selection mode,
first location to the second location.
which is indicated by the look of the state change indicator.
As discussed in the co-pending patent applications incorAs a further possibility in connection with the above
porated hy reference herein, the computer 201 according to
embodiments of the present invention may have the ability 6U example, at a second threshold amount of time of, e.g., 2
seconds, the wand tip may further animate and/or change to
to select ink when in pen mode. A user can press and hold,
indicate that a right click will now be simulated when the
then drag over ink, text, or other objects, within a document
stylus 204 is removed from the display surface 202.
to select those objects. A press and hold operation may
switch into selection mode. Because the user is using the
Where the user removes the stylus 204 from the display
stylus 204 or other similar user pointing device, there are 65 surface 202 prior to the threshold of time passing, then any
different considerations as compared with when a mouse is
subsequent stylus movement may be interpreted by the
used. For instance, while the user of a mouse may hear and
computer 201 as normal stylus input. Also, in this situation

MS-MOTO_ SDFLA_00000770896

US 6,791,536 B2

9

10

any state change indicator and/or animation thereof may he
heing removed from the touch-sensitive display surface
immediately canceled and removed from the display surface
before the threshold amount of time as detected in the step
202.
of Hrst detecting.
8. The method of claim 7, further including generating at
While exemplary systems and methods embodying the
prt:sent invention are shown by way of example, it will be 5 least one event representing the primary switch of the
pointing device being deactivated after the event representunderstood, of course, that the invention is not limited to
ing the primary switch of the pointing device being actithese embodiments. Modifications may be made by those
vated.
skilled in the art, particularly in light of the foregoing
9. The method of claim 1, further including the steps of:
teachings. For example, each of the elements of the aforesixth detecting whether the stylus moves along the touchmentioned embodiments may be utilized alone or in com- 10
sensitive display surface prior to the threshold amount
bination with elements of the other embodiments. Also,
of time having passed; and
although the above exemplary embodiments were described
with regard to simulating a mouse, the simulation of other
generating at least one event representing the primary
pointing devices (e.g., a trackball 400, such as shown in FIG.
switch of the pointing pointing device being activated
4, having a left button 401 and a right button 402) falls 15
responsive to the stylus being moved along the touchwithin the scope and spirit of the invention described herein.
sensitive display surface as detected in the step of sixth
We claim:
detecting.
1. In a computer, a method for simulating at least one
10. The method of claim 9, further including the steps of:
gesture of a pointing device having at least a primary switch
seventh detecting whether the stylus is moved along the
and a secondary switch, the method comprising the steps of: 20
touch-sensitive display surface after the step of sixth
first detecting whether a stylus is being held down on a
detecting; and
touch-sensitive display surface for at least a threshold
responsive to the stylus being moved as detected in the
amount of time;
step of seventh detecting, generating at least one event
representing movement of the pointing device.
second detecting whether the stylus is then removed from 25
the touch-sensitive display surface after at least the
11. The method of claim 10, further including the steps of:
threshold amount of time; and
eighth detecting whether the stylus is removed from the
touch-sensitive display surface after the step of seventh
responsive to the stylus being removed as detected in the
detecting; and
step of second detecting, generating at least one event
responsive to the stylus being removed as detected in the
representing the secondary switch of the pointing 30
device being activated.
step of eighth detecting, generating at least one event
2. The method of claim 1, wherein the step of generating
representing the primary switch of the pointing device
being activated.
further includes generating at least one event representing
12. The mcthod of claim 1, whcrein the pointing device
the secondary switch being deactivated after the event
representing the secondary switch of the pointing device 35 comprises a mouse, the primary switch comprises a left
being activated.
button of the mouse, and the secondary switch comprises a
3. The method of claim 1, further including generating a
right hutton of the mouse.
state change indicator responsive to the step of first detect13. The method of claim 1, wherein the pointing device
ing.
comprises a trackball, the primary switch comprises a left
4. The method of claim 1, further including the steps of: 40 button of the trackball, and the secondary switch comprises
a right button of the trackball.
third detecting whether the stylus is moved along the
14. A computer configured to simulate at least one gesture
touch-sensitive display surface prior to being removed
of a pointing device having a primary switch and a secondfrom the touch-sensitive display surface but after the
ary switch responsive to stylus input, the computer comthreshold amount of time has passed; and
responsive to the stylus being moved as detected in the 45 prising:
a touch-sensitive display surface; and
step of third detecting, generating at least one event
a processor coupled to the touch-sensitive display surface
representing the secondary switch of the pointing
and configured to detect whether the stylus is held
device being activated.
against the touch-sensitive display surface for at !cast a
5. The method of claim 4, further including the steps of:
threshold amount of time, and in response to the stylus
fourth detecting whether the stylus is moved along the 50
being held against the touch-sensitive display surface
touch-sensitive display surface after the step of third
for at least the threshold amount of time generating at
detecting; and
least one event representing an activation of the secresponsive to the stylus heing moved as detected in the
ondary switch of tht: pointing devict:, and in rt:sponse
step of fourth detecting, generating at least one event 55
to the stylus being removed from the touch-sensitive
represt:nting movement of the pointing device.
display surface before the threshold amount of time
6. The method of claim 5, further including the steps of:
generating at least one event representing an activation
fifth detecting whether the stylus is removed from the
of the primary switch of the pointing device.
touch-sensitive display surface after the step of fourth
15. The computer of claim 14, wherein the processor is
detecting; and
6U further configured to generate at least one event representing
the secondary switch of the pointing device being deactiresponsive to the stylus being removed as detected in the
step of fifth detecting, generating at least one event
vated after the event representing the secondary switch of
the pointing device being activated.
representing the secondary switch of the pointing
device being deactivated.
16. The method of claim 14, wherein the pointing device
7. The method of claim 1, further including generating at 65 comprises a mouse, the primary switch comprises a left
least one event representing the primary switch of the
button of the mouse, and the secondary switch comprises a
pointing device being activated responsive to the stylus
right button of the mouse.

MS-MOTO_ SDFLA_00000770897

US 6,791,536 B2
11

12

17. The method of claim 14, wherein the pointing device
responsive to the stylus heing moved as detected in the
step of fourth detecting, generating at kast one event
comprises a trackball, the primary switch comprises a left
representing a secondary switch drag of the pointing
bullon of the trackball, amI the secondary switch comprises
device.
a right button of the trackball.
25. The method of claim 22, wherein the first threshold
18. A computer-readable medium having computer- 5
amount of time is about 600 milliseconds.
executable instructions for performing steps for simulating
26. The method of claim 22, wherein the first threshold
at least one gesture of a pointing device having a primary
amount of time is between 300 and 600 milliseconds.
switch and a secondary switch, the steps comprising:
27. The method of claim 22, wherein the first threshold
first detecting whether a stylus is being held on a touchamount of time is between 600 milliseconds and 1 second.
sensitive display surface for at least a threshold amount 10
28. The method of claim 22, further including the steps of:
of time;
fifth detecting whether the stylus remains held down on
second detecting whether the stylus is then removed from
the touch-sensitive display surface for at least a second
the touch-sensitive display surface after at least the
threshold amount of time later in time than the first
threshold amount of time; and
threshold amount of time; and
15
responsive to the stylus remaining held down as detected
responsive to the stylus being removed as detected in the
in the step of fifth detecting, generating at least one
step of second detecting, generating event representing
event representing the secondary switch of the pointing
the secondary switch of the pointing device being
device being activated.
activated.
29. The method of claim 28, wherein the second threshold
19. The computer-readable medium of claim 18, further 20
of time is about 700 milliseconds.
having computer-executable instructions for performing the
30. The method of claim 22, further including generating
step of generating at least one event representing the seca state change indicator responsive to the stylus being held
ondary switch of the pointing device being deactivated after
down for at least a th.ird threshold amount of time.
the event representing the secondary switch of the pointing
31. The method of claim 30, wherein the third threshold
25 amount of time is equal to the first threshold amount of time.
device being activated.
20. The method of claim 18, wherein the pointing device
32. The method of claim 30, wherein the step of genercomprises a mouse, the primary switch comprises a left
ating the state change indicator includes generating a visual
button of the mouse, and the secondary switch comprises a
state change indicator on the touch-sensitive display surface
right button of the mouse.
that changes over time.
21. The method of claim 18, wherein the pointing device 30
33. The method of claim 32, wherein the third threshold
comprises a trackball, the primary switch comprises a left
amount of time is less than the first threshold amount of
button of the trackball, and the secondary switch comprises
time.
a right button of the trackball.
34. TIle method of claim 30, wherein the step of gener22. In a computer, a method for simulating at least one
ating the state change indicator includes generating a visual
gesture of a pointing device having a primary switch and a 35 state change indicator on the touch-sensitive display surface
secondary switch, the method comprising the steps of:
at a location depending upon a location of the stylus.
first detecting whether a stylus is heing held down on a
35. The method of claim 22, wherein the pointing device
touch-sensitive display surface for at least a first threshcomprises a mouse, the primary switch comprises a left
old amount of time;
bUllon of the mouse, amI the secondary switch comprises a
second detecting whether the stylus is then removed from 40 right button of the mouse.
36. The method of claim 22, wherein the pointing device
the touch-sensitive display surface before or after the
comprises a trackball, the primary switch comprises a left
first threshold amount of time;
button of the trackball, and the secondary switch comprises
responsive to the stylus being removed at a time before
a right button of the trackball.
the first threshold amount of time has passed, generat37. In a computer, a method for providing feedback
ing at least one event representing one of the primary 45
responsive to use of a stylus on a touch-sensitive display
switch of the pointing device being activated or the
surface, the method comprising the steps of:
secondary switch of the pointing device being actidetecting whether a stylus is being hcld down on a
vated; and
touch-sensitive display surface for at least a threshold
responsive to the stylus being removed at a time after the
amount of time; and
first threshold amount of time has passed, generating at 50
generating a state change indicator responsive to the
least one event representing the other of the primary
stylus being held down for at least the threshold amount
switch of the pointing device being activated and the
of time.
secondary switch of the pointing device being acti38. The method of claim 37, wherein the step of genervated.
23. The method of claim 22, further including the steps of: 55 ating includes generating a visual state change indicator on
the touch-sensitive display surface.
third detecting whether the stylus is moved along the
39. The method of claim 37, wherein the step of genertouch-sensitive display surface prior to the threshold
ating includes generating a visual state change indicator at a
amount of time having passed; and
location on the touch-sensitive display surface depending
responsive to the stylus being moved as detected in the 6U upon a location of the stylus.
step of third detecting, generating at least one event
40. TIle method of claim 37, wherein the step of generrepresenting a primary switch drag of the pointing
ating includes generating an animated visual state change
device.
indicator on the touch-sensitive display surface.
24. The method of claim 23, further including the steps of:
41. The method of claim 37, wherein the pointing device
fourth detecting whether the stylus is moved along the 65 comprises a mouse, the primary switch comprises a left
touch-sensitive display surface after to the first threshbutton of the mouse, and the secondary switch comprises a
right button of the mouse.
old amount of time having passed but not before; and

MS-MOTO_ SDFLA_00000770898

US 6,791,536 B2

13

14

42. The method of claim 37, wherein the pointing device
44. The apparatus of claim 43, wherein the processor is
comprises a trackball, the primary switch comprises a left
running a program, the program being configured to be
bUllon of the trackball, amI the secondary switch comprises
responsive to the event generated by the processor.
a right button of the trackball.
45. The apparatus of claim 43, wherein the event corn43. An apparatus comprising:
s prises one of a LdtMouseBullonDown tvtnt and a Righta touch-sensitive display surface; and
MouseButtonDoWIl event.
46. The method of claim 43, wherein the pointing device
a processor coupled to the touch-sensitive display surface
comprises a mouse having a primary switch corresponding
and configured to:
responsive to detecting that a stylus is brought down on
to the left click and a secondary switch corresponding to the
the touch-sensitive display surface, count at least a 10 right click, the primary switch comprising a left button of the
threshold amount of time and determine whether the
mouse, and the secondary switch comprising a right button
stylus is held down on the touch-sensitive display
of the mouse.
surface for at least the threshold amount of time, and
47. The method of claim 43, wherein the pointing device
comprises a trackball, having a primary switch correspondresponsive to detecting that the stylus is later removed
from the touch-sensitive display surface, generate at 15 ing to the left click and a secondary switch corresponding to
least one event representing one or the other of a left
the right click, the primary switch comprising a left button
click of a pointing device and a right click of a
of the trackball, and the secondary switch comprising a right
pointing device depending upon whether the stylus is
button of the trackball.
removed before or after the threshold amount of time
has passed.
* * * * *

MS-MOTO_ SDFLA_00000770899