Motorola Mobility, Inc. v. Microsoft Corporation

Filing 125

NOTICE by Microsoft Corporation re 124 Defendant's MOTION Claim Construction of Filing Exhibits (Attachments: # 1 Appendix (Index to Exhibits), # 2 Exhibit 101, # 3 Exhibit 102, # 4 Exhibit 201, # 5 Exhibit 301, # 6 Exhibit 302, # 7 Exhibit 304, # 8 Exhibit 305, # 9 Exhibit 501, # 10 Exhibit 502, # 11 Exhibit 504, # 12 Exhibit 601, # 13 Exhibit 801 (Part 1), # 14 Exhibit 801 (Part 2), # 15 Exhibit 901, # 16 Exhibit 902, # 17 Exhibit 903, # 18 Exhibit 904, # 19 Exhibit 905, # 20 Exhibit 1101, # 21 Exhibit 1102, # 22 Exhibit 1105, # 23 Exhibit 1110, # 24 Exhibit 1114, # 25 Exhibit 1115, # 26 Exhibit 1116, # 27 Exhibit 1201, # 28 Exhibit 1301, # 29 Exhibit 1302, # 30 Exhibit 1303, # 31 Exhibit 1304, # 32 Exhibit 1305, # 33 Exhibit 1401, # 34 Exhibit 1402, # 35 Exhibit 1403, # 36 Exhibit 1404, # 37 Exhibit 1405, # 38 Exhibit 1406, # 39 Exhibit 1407, # 40 Exhibit 1408, # 41 Exhibit 1409)(Miner, Curtis)

Exhibit 101 111111 1111111111111111111111111111111111111111111111111111111111111 US006791536B2 (54) United States Patent (10) Keely et a1. (12) (45) SIMUIATING GESTURES OF A POINTING DEVICE USING A STYLUS AND PROVIDING FEEDBACK THERETO (75) Notice: (21) Filed: 4,899,138 5,404,458 5,592,566 5,602,570 5,000,113 5,864,635 5,880,411 0,049,329 6,262,719 6,266,050 A * A * A * A * A * A * A * A * B1 * B1 * 2/1990 4/1995 1/1997 2/1997 9/1997 1/1999 3/1999 4/2000 7/2001 7/2001 Araki et al. ................ 345/175 Zetts 710/73 Pagallo et al. .............. 382/187 Capps ct al. ................ 345/173 Logan ......................... 341/34 Zetts et al. ................. 382/187 Gillespie et al. ......... 178/18.01 Zetts et al. ................. 345/179 Bi et al. ..................... 345/179 Oh et al. .................... 345/173 App!. No.: 09/815,272 (22) References Cited U.S. PATENT DOCUMENTS Assignee: Microsoft Corporation, Redmond, WA (US) (*) (56) Inventors: Leroy B. Keely, Portola Valley, CA (US); David F. Jones, Redmond, WA (US); David Switzer, Redmond, WA (US); Michael Hin-cheung Tsang, Bellevue, WA (US); William Hong Vong, Seattle, WA (US) (73) Patent No.: US 6,791,536 B2 Date of Patent: Sep.14,2004 * cited by examiner Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.c. 154(b) by 69 days. Primary Examiner-Henry N. Tran (74) Attorney, Agent, or Firm-Banner & Witcoff, Ltd. (57) (65) Mar. 23, 2001 Prior Publication Data US 2002/0057263 A1 May 16, 2002 Related U.S. Application Data (60) Provisional application No. 60/247,844, filed on Nov. 10, 2000. (51) (52) Int. CI.7 .................................................. G09G 5/00 U.S. CI........................ 345/173; 345/161; 345/163; 345/168; 345/173; 345/179; 345/863 Field of Search ................................. 345/173, 156, 345/161, 163, 167, 168, 179, 863; 178/19.01, 19.04,20.01; 703/17, 19, 21-24 (58) ABSTRACT A method and apparatus for simulating at least one gesture of a pointing device such as a mouse. A left click, right click, left drag, right drag, and/or mouse movement may be simulated using a stylus in conjunction with a touchsensitive display surface. For example, a computer having the display surface may detect whether a stylus is being held down on a touch-sensitive display surface for at least a threshold amount of time. The computer may further detect whether the stylus is then removed from the touch-sensitive display surface after at least the threshold amount of time. Responsive to the stylus being removed, the computer may generate at least one event representing a right mouse button being pressed. 47 Claims, 4 Drawing Sheets MS-MOTO_ SDFLA_00000770888 e . rJJ. . 166 _S~~T§~ ~§.~O!~ _I (ROM) 140 I 1~~ BIOS ~ ~ 100 1120 ~ """'" I DIGITIZER ,-- I -------(RAM) 150 IOPERATING 195 SYSTEM PROCESSING UNIT 110 VIDEO ADAPTER I 108 APPLICATION 196' , PROGRAMS I I I 165 16 107 = """'" rJj ~ '!=i .... ~~ N o o ~ OTHER PROGRAM MODULES 197 MAGNETIC DISK DRIVE INTERFACE HARD DISK INTERFACE PROGRAM DATA I ~170 /...-/ s:: en s:: ~- /// .,.../', ~80 "..,./...... t """ ; - 0 e 190 , 11 0 ", b ~9111 t 192 OTHER OPERATING APPLICATION 196 PROGRAM 1971PROGRAM 198 195 PROGRAMS SYSTEM MODULES DATA ,0 en o1 "'T r ,:t:o o o o o ...... ...... o CO CO CD rJj =.... ~ ~ ..... o ...., ~ 114 1;1 a..4Il'""\r IVlvDEM WIDE AREA NETWORK KEYBOARD 10 ~ 00 MOUSE I o -I NETWORK INTERFACE 112 > __ -- LOCAL AREA NETWORK SERIAL PORT INTERFACE 198 I ...-/'--/ OPTICAL DRIVE INTERFACE FIG. 1 APPLICATION PROGRAMS 111 -..~ ....... '-0 ~ U. w ~ Cd N u.s. Patent Sep.14,2004 Sheet 2 of 4 US 6,791,536 B2 202 / 204 201 ~\ - - - - - - - - 1 - - - - - - - - - - - - - - - - - - - - - - -J 203 FIG. 2 MS-MOTO_SDFLA_00000770890 u.s. Patent Sep.14,2004 US 6,791,536 B2 Sheet 3 of 4 314 LEFT DRAG >---------------YES--------------~ 304 313 FIG. 3 MS-MOTO_SDFLA_00000770891 u.s. Patent Sep.14,2004 US 6,791,536 B2 Sheet 4 of 4 400 \ 401 402 FIG. 4 MS-MOTO_ SDFLA_00000770892 US 6,791,536 B2 1 2 operahility as well. The user may generate other gestures using the mouse such as by clicking the left or right button of the mouse and moving the mouse while the button is depressed (a gesture commonly referred to as either a "left CROSS-REFERENCE TO RELATED 5 drag" or a "right drag," respectively). APPLICATIONS While such mouse gestures are of course easily done with an actual mouse, it is not always convenient for a user to use The present application claims priority to u.s. provisional a mouse with a computer even though the computer may patent application Serial No. 60/247,844, entitled Simulathave a GUI system. For instance, when using a portable ing Gestures of a Mouse Using a Stylus and Providing Feedback Thereto, filed Nov. 10, 2000, which is hereby 10 system there may not always be a surface available upon which to place a mouse. Some systems utilize a touchincorporated by reference as to its entirety. The present sensitive display screen with a stylus as an input device. The application is also related to application Ser. No. 09/760, user places the stylus tip upon the touch-sensitive display 631, entitled Method and Apparatus For Improving the screen at various locations to control objects displayed on Appearance of Digitally Represented Handwriting, filed Nov. 10,2000; to application Ser. No. 60/247,841, entitled 15 the display screen. A problem with this type of system is that the stylus is limited in the types of actions that may be easily Highlevcl Active Pen Matrix, filed Nov. 10, 2000; to appliperformed as compared with a mouse. For instance, the cation Ser. No. 60/247/973, entitled Selection Handles in typical stylus can perform only three types of movements: Editing Electronic Documents, filed Nov. 10, 200; to appliplacing the stylus tip onto the screen, moving the stylus tip cation Ser. No. 60,247,842, entitled Insertion Point Bungee Space Tool, filed Nov. 10, 2000; to application Ser. No. 20 across the screen, and removing the stylus tip from the screen. While a user may operate a remote toolbar to select 60/247,400, entitled System and Method for Accepting a different tool, the excessive movement renders the control Disparate Types of User Input, filed Nov. 10, 2000; to of the system laborious, inconvenient, and slow. Using the application Ser. No. 60/247,972, entitled In Air Gestures, tip of the stylus-type device, there is presently no convenient filed Nov. 10, 2000; to application Ser. No. 60/247,831, entitled Mouse Input Panel Windows Class List, filed Nov. 25 way to simulate or distinguish between, for example, a right click as opposed to a left click as interpreted by the operating 10,2000; to application Ser. No. 60/247,843, entitled Mouse system. Input Panel and User Interface, filed Nov. 10, 2000; to In order to make the full range of interface features application Ser. No. 60/247,479, entitled System and accessible to users of such a stylus-based computer, there is Method for Inserting Implicit Page Breaks, filed Nov. 10, 2000; to application Ser. No. 09/750,288, entitled 30 a need for an intuitive way of simulating mouse gestures with the stylus. Although not necessary, it is preferable and Classifying, Anchoring, and Transforming Ink, filed Dec. more useful that any new way of simulating mouse gestures 29, 2000; to application Ser. No. 09/730,170, entitled be compatible with existing software applications that conHighlevel Active Pen Matrix, filed Dec. 15, 2000; and to ventionally are used with a mouse. application Ser. No. 09/741,107, entitled Mode Hinting/ Switching, filed Dec. 21, 2000; each of which is incorpo- 35 There is also a need for providing helpful feedback to the rated by reference herein as to their entireties. user of the stylus to indicate whether gestures made with the stylus are those that are intended by the user. This is useful FIELD OF THE INVENTION with a stylus-type input device since there is conventionally The present invention is directed generally to apparatus 40 little to no natural tactile or auditory feedback when using a stylus than there is when using a mouse (e.g., the buttons of and methods for simulating various gestures of a pointing a mouse often have a definitive mechanical "click" feel and device with multiple switches, such as a two button mouse sound when pressed). As will be seen, such feedback is even in a computer system, and for providing feedhack thereto, more useful when provided in conjunction with various and more specifically to simulating gestures such as the left click of a mouse, the right click of a mouse, and mouse 45 gesture simulation techniques of the present invention. dragging by manipulation of a stylus in conjunction with a SUMMARY OF THE INVENTION touch-sensitive computer display, as well as generating appropriate visual or other feedback in response to certain As discussed in the various copending patent applications gestures. incorporated herein by reference, aspects of the present 50 invention are directed to a tablet-like computer that may be BACKGROUND OF THE INVENTION used for directly writing on a touch-sensitive display surface using a stylus. The computer may allow the user to write and 'lypical computer systems, especially computer systems to edit, manipulate, and create objects through the use of the using graphical user interface (GUI) systems such as stylus. Many of the features discussed in these copending Microsoft WINDOWS, are optimized for accepting user input from two discrete input devices: a keyboard for 55 applications are even more useful in combination with various aspects of the present invention discussed herein. entering text, and a pointing device such as a mouse with An aspect of the present invention is directed to methods two or more buttons or other selectors for driving the user and apparatus for simulating gestures of a pointing device, interface. Virtually all software applications designed to run on Microsoft WINDOWS are optimized to accept user input such as a mouse, trackhall, or joystick, hy use of a stylus on in the same manner. For instance, many applications make 6U a touch-sensitive display surface. For example, a left click of extensive use of a primary switch of the pointing device such a mouse may be simulated, e.g., where the user holds the as the left button of a mouse (a gesture commonly referred stylus down on the touch-sensitive display surface without to as a "left click" of a mouse) to select an object, and a substantial movement and then removes the stylus from the secondary switch such as the right button of the mouse (a display surface before the expiration of a threshold amount gesture commonly referred to as a "right click" of a mouse) 65 of time. A right click of a mouse may be simulated, e.g., to display context-sensitive command menus. It is noted that where the user waits until at least the threshold amount of time before removing the stylus. Where the stylus is moved other operating systems incorporate left and right click SIMUIATING GESTURES OF A POINTING DEVICE USING A STYLUS AND PROVIDING FEEDBACK THERETO MS-MOTO_ SDFLA_00000770893 US 6,791,536 B2 3 4 along the display surface during certain stylus gestures, left implement various aspects of the invention. Tn this embodiment, computer 100 includes a processing unit 110, or right dragging may thereby be simulated. a system memory 120, and a system bus 130 that couples Another aspect of the present invention is directed to various system components including the system memory to methods and apparatus for providing feedback to a user in a stylus-based touch-sensitive display surface computer sys- 5 the processing unit 110. The system hus 130 may he any of several types of bus structures including a memory bus or tem. The user may accordingly be given some indication that memory controller, a peripheral bus, and a local bus using the user has performed a particular stylus gesture, e.g., by any of a variety of bus architectures. The system memory displaying a visual object on the display surface to indicate includes read only memory (ROM) 140 and random access a particular status. Such feedback may be responsive to certain stylus gestures such as pressing and holding the 10 memory (RAM) 150. A basic input/output system 160 (BIOS), containing the stylus against the display surface for at least a certain basic routines that help to transfer information between minimum amount oftime. More generally, the feedback may elements within the computer 100, such as during start-up, be in the form of a "state change indicator," which may be is stored in ROM 140. Computer 100 also includes a hard visual on the display surface such as an icon, bitmap, or other visual object or indicator, and/or auditory such as a 15 disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or sound emitted from the computer's speakers. The state writing to a removable magnetic disk 190, and an optical change indicator may be animated and or otherwise vary disk drive 191 for reading from or writing to a removable with stylus hold time. Significant advantages of this aspect optical disk 192 such as a CD ROM or other optical media. of the invention are realized in conjunction with the various features discussed in the copending patent applications 20 The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 incorporated herein by reference. by a hard disk drive interface 192, a magnetic disk drive These and other features of the invention will be apparent interface 193, and an optical disk drive interface 194, upon consideration of the following detailed description of respectively. The drives and their associated computerpreferred embodiments. Although the invention has been 25 readable media provide nonvolatile storage of computer defined using the appended claims, these claims are exemreadable instructions, data structures, program modules and plary in that the invention is intended to include the elements other data for the personal computer 100. It will be appreand steps described herein in any combination or subcomciated by those skilled in the art that other types of computer bination. Accordingly, there are any number of alternative readable media which can store data that is accessible by a combinations for defining the invention, which incorporate 30 computer, such as magnetic cassettes, flash memory cards, one or more elements from the specification, including the digital video disks, Bernoulli cartridges, random access description, claims, and drawings, in various combinations memories (RAMs), read only memories (ROMs), and the or subcombinations. It will be apparent to those skilled in the like, may also be used in the example operating environrelevant technology, in light of the present specification, that ment. alternate combinations of aspects of the invention, either A number of program modules can be stored on the hard alone or in combination with one or more elements or steps 35 disk, magnetic disk 190, optical disk 192, ROM 140 or defim:d herein, may be utiliLed as modifications or alterRAM 150, including an operating system 195, one or more ations of the invention or as part of the invention. It is application programs 196, other program modules 197, and intended that the written description of the invention conprogram data 198. A user can enter commands and infortained herein covers all such modifications and alterations. 40 mation into the computer 100 through input devices such as BRIEF DESCRIPTION OF THE DRAWINGS a keyboard 101 and pointing device 102. Other input devices (not shown) may include a microphone, joystick, game pad, The foregoing summary of the invention, as well as the satellite dish, scanner, or the like. These and other input following detailed description of preferred embodiments, is devices are often connected to the processing unit 110 hetter understood when read in conjunction with the accompanying drawings, which are included by way of example, 45 through a serial port interface 106 that is coupled to the system bus, but may be connected by other interfaces, such and not by way of limitation with regard to the claimed as a parallel port, game port or a universal serial bus (USB). invention. In the accompanying drawings, elements are A monitor 107 or other type of display device is also labeled with three-digit reference numbers, wherein the first connected to the system bus 130 via an interface, such as a digit of a reference number indicates the drawing number in which the element is first illustrated. The same reference 50 video adapter lOR. Tn addition to the monitor, personal computers typically include other peripheral output devices number in different drawings refers to the same element. (not shown), such as speakers aod printers. FIG. 1 is a schematic diagram of a general-purpose digital The computer 100 can operate in a networked environcomputing environment that can be used to implement ment using logical connections to one or more remote various aspects of the invention. FIG. 2 is a plan view of a tablet computer and stylus that 55 computers, such as a remote computer 109. Remote computer 109 can be a server, a router, a network PC, a peer can be used in accordance with various aspects of the present device or other common network node, and typically invention. includes many or all of the elements described above FIG. 3 is a flow chart showing an exemplary set of steps relative to computer 100, although only a memory storage that may be performed in order to simulate a right click of 6U device 111 has been illustrated in FIG. 1. The logical a mouse according to aspects of the present invention. connections depicted in FIG. 1 include a local area network FIG. 4 is a plan view of an illustrative embodiment of a (LAN) 112 and a wide area network (WAN) 113. Such trackball having left and right buttons. networking environments are commonplace in offices, DETAILED DESCRIPTION OF PREFERRED enterprise-wide computer networks, intranets and the InterEMBODIMENTS 65 net. When used in a LAN networking environment, the comFIG. 1 shows an exemplary embodiment of a generalpurpose digital computing environment that can be used to puter 100 is coonected to the local network 112 through a MS-MOTO_ SDFLA_00000770894 US 6,791,536 B2 5 6 network interface or adapter 114. When used in a WAN milliseconds. However, the threshold amount oftime may he networking environment, the personal computer 100 typiany amount of time, such as between 300 and 600 cally includes a modem 115 or other means for establishing milliseconds, or between 600 milliseconds and 1 second, or a communications over the wide area network 113, such as greater than 1 second. If the computer 201 detects that the the Internet. The modem 115, which may he internal or 5 stylus moves prior to the timeout condition occurring (step external, is connected Lo the system bus 130 via the serial 302), then in response the computer 201 may gennate a port interface 106. In a networked environment, program standard Microsoft WINDOWS LcftMouseBultonDown modules depicted relative to the personal computer 100, or event (or other event that represents the primary switch of the pointing device being activated) in the original position portions thereof, may be stored in the remote memory 10 where the stylus 204 was first brought down (step 314), storage device. which IIIay be sent tu the suftware applicatiun presently in It will be appreciated that the network connections shown use. At this point, any subsequent movement of the stylus are example and other means of establishing a communica204 across the display surface 202 may be detected by the tions link between the computers can be used. The existence computer 201 (step 315), and in response the computer 201 of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed, and the 15 may generate one or more Microsoft WINDOWS MouseMove events (step 316) (or other events representing movesystem can be operated in a client-server configuration to ment of the pointing device). When the stylus 204 is permit a user to retrieve web pages from a web-based server. eventually removed from the display surface 202 (hereafter Any of various conventional web browsers can be used to referred to as "bringing the stylus up") (step 317), the display and manipulate data on web pages. FIG. 2 illustrates a tablet PC 201 that can be used in 20 computer 201 may detect this and in response may generate a Microsoft WINDOWS LeftMouseButlonUp event (step accordance with various aspects of the present invention. 318) (or other event that represents the primary switch heing Any or all of the features, subsystems, and functions in the deactivated). Thus, via steps 301, 302, and 314-318, a left system of FIG. 1 can be included in the computer of FIG. 2. drag of a mouse is simulated. Tablet PC 201 includes a large touch-sensitive display To simulate a left dick of a mouse without dragging, if the surface 202, e.g., a digitizing flat panel display, for example, 25 computer 201 detects that the stylus 204 is not moved prior a liquid crystal display (LCD) screen, on which a plurality to the timeout condition (step 302), and if instead the stylus of windows 203 is displayed. Using a stylus 204, a user can 204 is brought up prior to the timeout condition (step 303), select, highlight, and write on the digitizing display area. then in response the computer 201 may generate first a Examples of suitable digitizing display panels include electromagnetic pen digitizers, such as the Mutoh or Wacom pen 30 LeftMouseButtonDown event (or other event that represents the primary switch being activated), then optionally delay digitizers. Other types of pen digitizers, e.g., optical for a period of time (step 320) such as about 20 milliseconds digitizers, may also be used. Tablet PC 201 interprets marks or some other amount of time, and then generate a Lcftmade using stylus 204 in order to manipulate data, enter text, MouseButtonUp event (steps 304 and 318) (or other event and execute conventional computer application tasks such as spreadsheets, word processing programs, and the like. The 35 that represents the primary switch being deactivated). Thus, via steps 301-304, 318, and 320, a slightly delayed left click touch-sensitive display surface 202 may also be proximityof a mouse is simulated. sensitive such that the tip of the stylus 204 may be sensed by the display surface 202 when proximate thereto without To simulate a right click of a mouse without dragging, if actually contacting the display surface 202. the computer 201 detects that the stylus 204 is not moved The stylus 204 may be equipped with or without buttons 40 either prior to the timeout condition or after the timeout condition (steps 302 and 305), and if the computer 201 or other features to augment its selection capabilities. In one further detects that the stylus 204 is not brought up until after embodiment, the stylus 204 may be implemented as a the timeout condition (steps 3U3 and 306), and thus that the "pencil" or "pen", in which one end constitutes a writing, or stylus 204 has been held down for at least the threshold tip, portion and the other end constitutes an "eraser" end that, when moved across the display indicates portions of the 45 amount of time, then in response the computer 201 may generate first a Microsoft WINDOWS RightMouseButtondisplay that are to be erased. The term "stylus" as used Down event (step 307) (or other event that represents the herein is intended to have a broad definition that includes, secondary switch of the pointing device being activated). e.g., a uscr's own finger, a pen, a pencil, a stick, a rod, and/or The computer 201 may, further in response, optionally delay any other similar device suitable for pointing at a particular location on the display surface 202. Other types of pointing 50 for a period of time (step 308), in one embodiment, about 20 milliseconds, and then generate a Microsoft WINDOWS devices, such as a mouse, trackball, or the like may be used RightMouseBullonUp event (skp 313) (ur other event that with the tablet PC 201 in addition to or conjunction with the represents the secondary switch being deactivated). The stylus 204. computer 201 may also generate and display a state change Referring now to FIG. 3, an example of how a left drag of a mouse is simulated is now discussed. Any or all of steps 55 indicator (step 319) that indicates to the user that the stylus 204 has been placed down for a certain minimum amount of set forth herein and in FIG. 3 may be performed by special time, as will be discussed below in more detail. software, hardware, and/or firmware that runs in conjunction It is noted that steps 319 may be in a variety of places with, or is part of, the operating system of the computer 20l. within the flowchart of FIG. 3 other than the one shown, To simulate a left drag, the user may touch the stylus 204 or other similar user pointing device to the display surface 202, 6U such as between steps 305 and 306. Steps 301-303, 305-308, and 313, thus simulate a delayed right click of a an action referred to hereafter as "placing the stylus down" mouse. (step 301). Responsive to the computer 201 detecting the To simulate a right drag of a mouse, if the computer 201 stylus being placed down, the computer 201 may begin counting time, e.g., by using a timer, up to at least a detects that the stylus 204 is not moved until after the threshold amount of time. A timeout condition is defined 65 timeout condition (step 305), then in response the computer responsive to the threshold amount of time having passed. In 201 may generate a RightMouseButtonDown event (step one embodiment, the threshold amount of time is about 600 309) (or other event that represents the secondary switch MS-MOTO_ SDFLA_00000770895 US 6,791,536 B2 7 8 heing activated). Upon the computer 201 detecting any feel the click of a mouse hutton to ensure that a click has substqut:nt movemmt of the stylus 204 along tht: display bet:n pt:rformed, tht: ust:r of the stylus 204 may not rt:ceive surface 202 (step 310), the computer 201 may in response clear enough feedback that the intended gesture has been generate one or more MouseMove events (step 311) (or made properly. Also, it is desirable to indicate to the user that other events representing movement of the pointing device). 5 a particular requested mode has really heen switched to, Wht:n the computer dett:cts that the stylus 204 is brought up such as sekction mode. Further, wht:re the user must hold (312), in rcsponse the computer 201 may generate a Rightdown the stylus 204 for at least a ccrtain thrcshold amount of time to simulate, for example, a right click, feedback to MouseButtonUp event (step 313) (or other event that repthe user indicating that the threshold time has passed would resents the secondary switch being deactivated). This simulates, via steps 301-303,305, and 309-313, a right drag 10 be useful. of a mouse. Such feedback may be in the form of a state change indicator, which may be visual (e.g., a graphical icon, a As a further feature of the right click simulation, a special symbol, a bitmapped or other image, etc., displayed on the hold event may be defmed wherein the stylus 204 is held display surface 202 such as state change indicator 205 in against the display screen 202 without substantial movement for at least another threshold amount of time, say about 700 15 FIG. 2), auditory (e.g., a click, a beep, tone, etc. emanating from the computer 201 and/or stylus 204), and/or tactile milliseconds, or between 0.5 and 1 second, or between about (e.g., vibration of the computer 201 and/or stylus 204). For 1 and 3 seconds or more, the computer 201 may detect such example, where the user selects the pen icon in the applia hold event and in response act by immediately interpreting cation toolbar (as described in at least one of the co-pending any subsequent holding of the stylus 204 as holding the left mouse button down (or, more generally, as activating the 20 patent applications incorporated herein by reference), the user may drag the stylus 204 over a document displayed on primary switch of the pointing device). Thus, the computer the display surface 202 such that digital ink is generated on 201 may, after 600 milliseconds of stylus holding, go on to the display surface 202. At this point, the user may switch to steps 305 and 319, and where the stylus 204 is released prior selection mode by pressing and holding the stylus 204 for at to, say, 5 seconds or some other time limit, then steps 306-308 and 313 may be implemented. However, if instead 25 kast a thrt:shold amount of timt: that may bt: the same as the threshold amount of time or that may be different. For the stylus 204 is further held at least 5 seconds, then the instance, the threshold minimum amount of time that the special subroutine may, in response, immediately generate a user must hold the stylus 204 against the display surface 202 LeftMouseButtonDown event (or other event that represents may be, in one embodiment, about 400 milliseconds, or the primary switch of the pointing device being activated). Then, when the stylus is eventually released, a LeftMouse- 30 between about 400 milliseconds and about 1 second, or more than about 1 second. Responsive to the user holding without ButtonUp event (or other event that represents the primary substantial movement the stylus 204 against the display switch being deactivated) would be generated by the special surface 202 for the minimum threshold amount of time, the subroutine. If the stylus 204 is moved along the display computer 201 may generate a state change indicator on the surface 202, then the computer 201 may immediately in response generate one or more MouseMove events (or other 35 display surface 202 which may be located at a location depending upon the location of the tip of the stylus 204. For events representing movement of the pointing device). example, the state change indicator may be located at or near When the stylus 204 is eventually removed from the display the location of the tip of tht: stylus 204. surface 202, all pending RightMouse events (or other events that rt:present action by the stcomlary switch of the pointing In some t:mbodiments, tht: statt change indicator may be device) that would normally have been generated in accor- 40 animated and/or may otherwise change over time. For instance, the state change indicator 205 is shown as a dance with the method shown in FIG. 3 would be canceled. partially completed clock-like loop that may become more In the above and remaining discussion, it is assumed that complete over time in, e.g., a clockwise direction. In such a Microsoft WINDOWS operating system is being used with embodiments with animation, the state change indicator may the computer 201. However, any operating system, especially a GUI operating system, may be used in accordance 45 actually begin prior to the threshold amount of time passing, as an indication to the user that the threshold amount of time with the present invention. Further, although the above will soon pass. For instance, where a first threshold amount embodiments are sample methods for simulating mouse of time is 1 second, holding the stylus 204 against the gcstures using a stylus, thcre arc many variations that are display surface 202 for 500 milliseconds may cause an icon encompassed by the present invention. For instance, the stylus gestures required for simulation a left elick and a right 50 (a state change indicator) in the shape of a wand tip to appear on the display surface 202 at or near the location of the stylus click might be reversed, or a user may not need to always 204. As the stylus 204 is continued to be held, tht: icon may keep the stylus 204 down on the display surface 202 during animate and begin to glow to indicate that a gesture is the entire time that a drag is executed, such that a first beginning to form. At the first threshold amount of time, the touch/touch-and-hold of the stylus 204 at a first location on the display surface 202, and a second touch/touch-and-hold 55 animation may have reached the point where the wand tip is fully glowing. Moreover, at the passing of the first threshold at a second location, may thereby simulate a drag from the of time, the user may thereby be placed in selection mode, first location to the second location. which is indicated by the look of the state change indicator. As discussed in the co-pending patent applications incorAs a further possibility in connection with the above porated hy reference herein, the computer 201 according to embodiments of the present invention may have the ability 6U example, at a second threshold amount of time of, e.g., 2 seconds, the wand tip may further animate and/or change to to select ink when in pen mode. A user can press and hold, indicate that a right click will now be simulated when the then drag over ink, text, or other objects, within a document stylus 204 is removed from the display surface 202. to select those objects. A press and hold operation may switch into selection mode. Because the user is using the Where the user removes the stylus 204 from the display stylus 204 or other similar user pointing device, there are 65 surface 202 prior to the threshold of time passing, then any different considerations as compared with when a mouse is subsequent stylus movement may be interpreted by the used. For instance, while the user of a mouse may hear and computer 201 as normal stylus input. Also, in this situation MS-MOTO_ SDFLA_00000770896 US 6,791,536 B2 9 10 any state change indicator and/or animation thereof may he heing removed from the touch-sensitive display surface immediately canceled and removed from the display surface before the threshold amount of time as detected in the step 202. of Hrst detecting. 8. The method of claim 7, further including generating at While exemplary systems and methods embodying the prt:sent invention are shown by way of example, it will be 5 least one event representing the primary switch of the pointing device being deactivated after the event representunderstood, of course, that the invention is not limited to ing the primary switch of the pointing device being actithese embodiments. Modifications may be made by those vated. skilled in the art, particularly in light of the foregoing 9. The method of claim 1, further including the steps of: teachings. For example, each of the elements of the aforesixth detecting whether the stylus moves along the touchmentioned embodiments may be utilized alone or in com- 10 sensitive display surface prior to the threshold amount bination with elements of the other embodiments. Also, of time having passed; and although the above exemplary embodiments were described with regard to simulating a mouse, the simulation of other generating at least one event representing the primary pointing devices (e.g., a trackball 400, such as shown in FIG. switch of the pointing pointing device being activated 4, having a left button 401 and a right button 402) falls 15 responsive to the stylus being moved along the touchwithin the scope and spirit of the invention described herein. sensitive display surface as detected in the step of sixth We claim: detecting. 1. In a computer, a method for simulating at least one 10. The method of claim 9, further including the steps of: gesture of a pointing device having at least a primary switch seventh detecting whether the stylus is moved along the and a secondary switch, the method comprising the steps of: 20 touch-sensitive display surface after the step of sixth first detecting whether a stylus is being held down on a detecting; and touch-sensitive display surface for at least a threshold responsive to the stylus being moved as detected in the amount of time; step of seventh detecting, generating at least one event representing movement of the pointing device. second detecting whether the stylus is then removed from 25 the touch-sensitive display surface after at least the 11. The method of claim 10, further including the steps of: threshold amount of time; and eighth detecting whether the stylus is removed from the touch-sensitive display surface after the step of seventh responsive to the stylus being removed as detected in the detecting; and step of second detecting, generating at least one event responsive to the stylus being removed as detected in the representing the secondary switch of the pointing 30 device being activated. step of eighth detecting, generating at least one event 2. The method of claim 1, wherein the step of generating representing the primary switch of the pointing device being activated. further includes generating at least one event representing 12. The mcthod of claim 1, whcrein the pointing device the secondary switch being deactivated after the event representing the secondary switch of the pointing device 35 comprises a mouse, the primary switch comprises a left being activated. button of the mouse, and the secondary switch comprises a 3. The method of claim 1, further including generating a right hutton of the mouse. state change indicator responsive to the step of first detect13. The method of claim 1, wherein the pointing device ing. comprises a trackball, the primary switch comprises a left 4. The method of claim 1, further including the steps of: 40 button of the trackball, and the secondary switch comprises a right button of the trackball. third detecting whether the stylus is moved along the 14. A computer configured to simulate at least one gesture touch-sensitive display surface prior to being removed of a pointing device having a primary switch and a secondfrom the touch-sensitive display surface but after the ary switch responsive to stylus input, the computer comthreshold amount of time has passed; and responsive to the stylus being moved as detected in the 45 prising: a touch-sensitive display surface; and step of third detecting, generating at least one event a processor coupled to the touch-sensitive display surface representing the secondary switch of the pointing and configured to detect whether the stylus is held device being activated. against the touch-sensitive display surface for at !cast a 5. The method of claim 4, further including the steps of: threshold amount of time, and in response to the stylus fourth detecting whether the stylus is moved along the 50 being held against the touch-sensitive display surface touch-sensitive display surface after the step of third for at least the threshold amount of time generating at detecting; and least one event representing an activation of the secresponsive to the stylus heing moved as detected in the ondary switch of tht: pointing devict:, and in rt:sponse step of fourth detecting, generating at least one event 55 to the stylus being removed from the touch-sensitive represt:nting movement of the pointing device. display surface before the threshold amount of time 6. The method of claim 5, further including the steps of: generating at least one event representing an activation fifth detecting whether the stylus is removed from the of the primary switch of the pointing device. touch-sensitive display surface after the step of fourth 15. The computer of claim 14, wherein the processor is detecting; and 6U further configured to generate at least one event representing the secondary switch of the pointing device being deactiresponsive to the stylus being removed as detected in the step of fifth detecting, generating at least one event vated after the event representing the secondary switch of the pointing device being activated. representing the secondary switch of the pointing device being deactivated. 16. The method of claim 14, wherein the pointing device 7. The method of claim 1, further including generating at 65 comprises a mouse, the primary switch comprises a left least one event representing the primary switch of the button of the mouse, and the secondary switch comprises a pointing device being activated responsive to the stylus right button of the mouse. MS-MOTO_ SDFLA_00000770897 US 6,791,536 B2 11 12 17. The method of claim 14, wherein the pointing device responsive to the stylus heing moved as detected in the step of fourth detecting, generating at kast one event comprises a trackball, the primary switch comprises a left representing a secondary switch drag of the pointing bullon of the trackball, amI the secondary switch comprises device. a right button of the trackball. 25. The method of claim 22, wherein the first threshold 18. A computer-readable medium having computer- 5 amount of time is about 600 milliseconds. executable instructions for performing steps for simulating 26. The method of claim 22, wherein the first threshold at least one gesture of a pointing device having a primary amount of time is between 300 and 600 milliseconds. switch and a secondary switch, the steps comprising: 27. The method of claim 22, wherein the first threshold first detecting whether a stylus is being held on a touchamount of time is between 600 milliseconds and 1 second. sensitive display surface for at least a threshold amount 10 28. The method of claim 22, further including the steps of: of time; fifth detecting whether the stylus remains held down on second detecting whether the stylus is then removed from the touch-sensitive display surface for at least a second the touch-sensitive display surface after at least the threshold amount of time later in time than the first threshold amount of time; and threshold amount of time; and 15 responsive to the stylus remaining held down as detected responsive to the stylus being removed as detected in the in the step of fifth detecting, generating at least one step of second detecting, generating event representing event representing the secondary switch of the pointing the secondary switch of the pointing device being device being activated. activated. 29. The method of claim 28, wherein the second threshold 19. The computer-readable medium of claim 18, further 20 of time is about 700 milliseconds. having computer-executable instructions for performing the 30. The method of claim 22, further including generating step of generating at least one event representing the seca state change indicator responsive to the stylus being held ondary switch of the pointing device being deactivated after down for at least a th.ird threshold amount of time. the event representing the secondary switch of the pointing 31. The method of claim 30, wherein the third threshold 25 amount of time is equal to the first threshold amount of time. device being activated. 20. The method of claim 18, wherein the pointing device 32. The method of claim 30, wherein the step of genercomprises a mouse, the primary switch comprises a left ating the state change indicator includes generating a visual button of the mouse, and the secondary switch comprises a state change indicator on the touch-sensitive display surface right button of the mouse. that changes over time. 21. The method of claim 18, wherein the pointing device 30 33. The method of claim 32, wherein the third threshold comprises a trackball, the primary switch comprises a left amount of time is less than the first threshold amount of button of the trackball, and the secondary switch comprises time. a right button of the trackball. 34. TIle method of claim 30, wherein the step of gener22. In a computer, a method for simulating at least one ating the state change indicator includes generating a visual gesture of a pointing device having a primary switch and a 35 state change indicator on the touch-sensitive display surface secondary switch, the method comprising the steps of: at a location depending upon a location of the stylus. first detecting whether a stylus is heing held down on a 35. The method of claim 22, wherein the pointing device touch-sensitive display surface for at least a first threshcomprises a mouse, the primary switch comprises a left old amount of time; bUllon of the mouse, amI the secondary switch comprises a second detecting whether the stylus is then removed from 40 right button of the mouse. 36. The method of claim 22, wherein the pointing device the touch-sensitive display surface before or after the comprises a trackball, the primary switch comprises a left first threshold amount of time; button of the trackball, and the secondary switch comprises responsive to the stylus being removed at a time before a right button of the trackball. the first threshold amount of time has passed, generat37. In a computer, a method for providing feedback ing at least one event representing one of the primary 45 responsive to use of a stylus on a touch-sensitive display switch of the pointing device being activated or the surface, the method comprising the steps of: secondary switch of the pointing device being actidetecting whether a stylus is being hcld down on a vated; and touch-sensitive display surface for at least a threshold responsive to the stylus being removed at a time after the amount of time; and first threshold amount of time has passed, generating at 50 generating a state change indicator responsive to the least one event representing the other of the primary stylus being held down for at least the threshold amount switch of the pointing device being activated and the of time. secondary switch of the pointing device being acti38. The method of claim 37, wherein the step of genervated. 23. The method of claim 22, further including the steps of: 55 ating includes generating a visual state change indicator on the touch-sensitive display surface. third detecting whether the stylus is moved along the 39. The method of claim 37, wherein the step of genertouch-sensitive display surface prior to the threshold ating includes generating a visual state change indicator at a amount of time having passed; and location on the touch-sensitive display surface depending responsive to the stylus being moved as detected in the 6U upon a location of the stylus. step of third detecting, generating at least one event 40. TIle method of claim 37, wherein the step of generrepresenting a primary switch drag of the pointing ating includes generating an animated visual state change device. indicator on the touch-sensitive display surface. 24. The method of claim 23, further including the steps of: 41. The method of claim 37, wherein the pointing device fourth detecting whether the stylus is moved along the 65 comprises a mouse, the primary switch comprises a left touch-sensitive display surface after to the first threshbutton of the mouse, and the secondary switch comprises a right button of the mouse. old amount of time having passed but not before; and MS-MOTO_ SDFLA_00000770898 US 6,791,536 B2 13 14 42. The method of claim 37, wherein the pointing device 44. The apparatus of claim 43, wherein the processor is comprises a trackball, the primary switch comprises a left running a program, the program being configured to be bUllon of the trackball, amI the secondary switch comprises responsive to the event generated by the processor. a right button of the trackball. 45. The apparatus of claim 43, wherein the event corn43. An apparatus comprising: s prises one of a LdtMouseBullonDown tvtnt and a Righta touch-sensitive display surface; and MouseButtonDoWIl event. 46. The method of claim 43, wherein the pointing device a processor coupled to the touch-sensitive display surface comprises a mouse having a primary switch corresponding and configured to: responsive to detecting that a stylus is brought down on to the left click and a secondary switch corresponding to the the touch-sensitive display surface, count at least a 10 right click, the primary switch comprising a left button of the threshold amount of time and determine whether the mouse, and the secondary switch comprising a right button stylus is held down on the touch-sensitive display of the mouse. surface for at least the threshold amount of time, and 47. The method of claim 43, wherein the pointing device comprises a trackball, having a primary switch correspondresponsive to detecting that the stylus is later removed from the touch-sensitive display surface, generate at 15 ing to the left click and a secondary switch corresponding to least one event representing one or the other of a left the right click, the primary switch comprising a left button click of a pointing device and a right click of a of the trackball, and the secondary switch comprising a right pointing device depending upon whether the stylus is button of the trackball. removed before or after the threshold amount of time has passed. * * * * * MS-MOTO_ SDFLA_00000770899