Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 1056

Administrative Motion to File Under Seal Reply Declaration of Marc J. Pernick in Support of Apple's Motion to Strike Portions of Samsung's Expert Reports filed by Apple Inc.. (Attachments: #1 Declaration of Marc J. Pernick in Support of Reply re Apple's Motion to Strike Portions of Samsung's Expert Reports, #2 Exhibit 1, #3 Exhibit 2, #4 Exhibit 3, #5 Exhibit 4, #6 Exhibit 5, #7 Exhibit 6, #8 Exhibit 7, #9 Exhibit 8, #10 Exhibit 9, #11 Exhibit 10, #12 Exhibit 11)(Jacobs, Michael) (Filed on 6/7/2012)

Exhibit 2

EXHIBIT J-1
SAMSUNG’S INVALIDITY CLAIM CHARTS FOR MERL DIAMOND TOUCH TABLE SYSTEM
1.

2.
3.
4.
5.
6.
7.
8.

Materials cited:
Chris Forlines, C., Esenther, A., Shen, C., Wigdor, D., and Ryall, K. Multi-user, Multi-display Interaction with a Single-user, Single-display Geospatial Application. UIST
’06 (ACM Oct. 15-18 2006) printed in original at pp. 273-276, reprinted as Mitsubishi Electronic Research Laboratories (“MERL”) Technical Report No. TR2006-083 in
October 2006.1 (“TR2006-083”)
P.H. Dietz and Leigh, D. DiamondTouch: a multi-user touch technology. Proc ACM UIST, (ACM 2001) printed in original at pp. 219-226, reprinted as MERL Technical
Report No. TR2003-125. (“TR2003-125”)
Alan Esenther, Cliff Forlines, Kathy Ryall, Sam Shipman. DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications, (MERL 2002), printed as MERL
Technical Report No. TR2002-48 (“TR2002-48”)
Edward Tse, Chia Shen, Saul Greenberg, Clifton Forlines: Enabling interaction with single user applications through speech and gestures on a multi-user tabletop (AVI
2006), pp. 336-343, reprinted as MERL Technical Report No. TR2005-130 (“TR2005-130”).
Alan Esenther and Kent Wittenburg, Multi-User Multi-Touch Games on DiamondTouch with the DTFlash Toolkit , printed as MERL Technical Report No. TR2005-105
(2005).
http://www.youtube.com/watch?v=t35HXAjNW6s (“MERL video”)
Edward Tse exhibiting Diamond Touch (2006) - http://video.google.com/videoplay?docid=6420668728353654549 (“TSE video”)
DiamondTouch II 88cm Engineering Prototype (DT Physical Device)2.

U.S. Patent No. 7,844,915
Claim 1
[1p] A machine
implemented method for
scrolling on a touch-

MERL DIAMOND TOUCH TABLE SYSTEM
The Diamond Touch Table System discloses a machine implemented method for scrolling on a touch-sensitive display of a device. For
example:

1

Samsung is informed and believes that a standard business practice of Mitsubishi Electronic Research Laboratories was to reprint and make available on its website
(http://www.merl.com/publications/?pub_type=Technical+Reports) what it considered to be “major publications” by members of the MERL staff to which MERL owned the
copyright. According to MERL’s website, as of October 7, 2011, such a report would be considered “major” if it appeared in a “refereed journal, a refereed conference
proceeding or some other significant publication…” (Id.)
2

Outside counsel for Samsung is currently in possession of this device, and will make it available for inspection by counsel at a time convenient for them, given reasonable

notice.

03450.51900/4387870.1

1

U.S. Patent No. 7,844,915
sensitive display of a
device comprising:

MERL DIAMOND TOUCH TABLE SYSTEM
“ABSTRACT
DiamondTouch [1] is a multi-touch input technology that supports multiple, simultaneous users; it can distinguish who is touching where.
We present the DiamondTouch SDK; it provides support for the development of applications that utilize DiamondTouch’s capabilities to
implement computer supported collaboration and rich input modalities (such as gestures). Our first demo illustrates the basic utilities and
functionality of our system.”
(TR2002-48 at 2).

03450.51900/4387870.1

2

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM

(TR2002-48 at 2).

“Our infrastructure is illustrated in Fig. 1. A standard Windows computer drives our infrastructure software, as described below.
The table is a 42” MERL Diamond Touch surface [6] with a 4:3 aspect ratio; a digital projector casts a 1280x1024 pixel image on the table’s
surface. This table is multi-touch sensitive, where contact is presented through the DiamondTouch SDK as an array of horizontal and

03450.51900/4387870.1

3

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
vertical signals, touch points and bounding boxes (Fig. 1, row 5). The table is also multi-user, as it distinguishes signals from up to four
people. While our technology uses the Diamond Touch, the theoretical motivations, strategies developed, and lessons learnt should apply to
other touch/vision based surfaces that offer similar multi user capabilities.”
(TR2005-130 at 2).

(Tse at 3, Fig. 1.)
4.1 GoogleEarth
Google Earth is a free desktop geospatial application that allows one to search, navigate, bookmark, and annotate satellite imagery of the
entire planet using a keyboard and mouse. Its database contains detailed satellite imagery with layered geospatial data (e.g., roads, borders,
accommodations, etc). It is highly interactive, with compelling real time feedback during panning, zooming and ‘flying’ actions, as well as
the ability to tilt and rotate the scene and view 3D terrain or buildings. . . . Table 1 provides a partial list of how we mapped Google Earth
onto our multimodal speech and gesture system, while Fig. 2 illustrates Google Earth running on our multimodal, multi user table.
(TR2005-130 at 4.)

03450.51900/4387870.1

4

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM

(TR2005-130 at 4.)

(TR2005-130 at 4.)

[1a]receiving a user input,
the user input is one or
more input points applied
to the touch-sensitive
display that is integrated
with the device;

03450.51900/4387870.1

The Diamond Touch Table System discloses receiving a user input, the user input is one or more input points applied to the touch-sensitive
display that is integrated with the device. For example:
“To interact with existing single user applications, we first use the GroupLab WidgetTap toolkit [8] to determine the location and size of the
GUI elements within it. We then use the Microsoft Send Input facility to relay the gesture and speech input actions to the locations of the
mapped UI elements (Fig. 1, rows 1, 2 and 3). Thus speech and gestures are mapped and transformed into one or more traditional GUI
actions as if the user had performed the interaction sequence via the mouse and keyboard. The consequence is that the application appears to

5

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
directly understand the spoken command and gestures.”
(TR2005-130 at 5)
“Mapping of Gestures. Many systems rely on abstract gestures to invoke (i.e., mode change into) commands. For example, a two fingered
gesture invokes an ‘Annotate’ mode in Wu’s example application [25]. Yet our behavioral foundations state that people working over a table
should be able to easily understand other people’s rich gestural acts and hand postures as both consequential communication and as
communicative acts. This strongly suggests that our vocabulary of postures and dynamics must reflect people’s natural gestures as much as
possible (a point also advocated in [25][26]).
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth. Pointing to indicate deictic references, and using the sides of two hands to select a group of
objects in Warcraft III is also well understood [17][5][3]. Because most of these acts work over a location, gaze awareness becomes highly
meaningful. However, the table’s input constraints can restrict what we would like to do. For example, an upwards hand tilt movement
would be a natural way to tilt the 3D map of Google Earth, but this posture is not recognized by the DiamondTouch table. Instead, we resort
to a more abstract one hand / five finger gesture set to tilt the map up and down (Table 1).”
(TR2005-130 at 6)
DiamondTouch uses an array of antennas embedded into a surface, with each antenna transmitting a unique signal. Each user has their own
receiver, generally attached to their chair. When a person touches the surface, energy from nearby antennas is coupled through the user to
their receiver. Using this mechanism, the system determines who is touching where.
(TR2002-48 at 3)

03450.51900/4387870.1

6

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM

(TR2002-48 at 2).
[1b]creating an event
object in response to the
user input;

03450.51900/4387870.1

The Diamond Touch Table system discloses creating an event object in resposne to the user input. For example:
“Our third demo illustrates the mouse emulation capabilities of DiamondTouch. There are two aspects to this demo. First, this capability
allows traditional software to be used with DiamondTouch. We currently have several mouse modes (one- touch, two-touch, etc) and are
experimenting to determine how best to implement a fully-functioning mouse with DiamondTouch. Our mouse emulator works with
7

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
traditional software. Second, under Windows the DiamondTouch SDK can generate ‘augmented mouse events’ that contain the user ID info
in an extra field. In this manner C++ Windows code using a Win32 API call can get the ID information for each mouse event as it is
processed by the application. The application then needs to store the state for the different users, and also some mouse information. We have
a sample drawing program (a modified MFC demo) to illustrate the use of ‘augmented mouse events’ for writing applications for
DiamondTouch.”
(TR2002-48 at 3.)
“Gesture Engine. Since recognizing gestures from multiple people on a table top is still an emerging research area [25][26], we could not
use existing 3rd party gesture recognizers. Consequently, we developed our own Diamond Touch gesture recognition engine to convert the
raw touch information produced by the DiamondTouch SDK into a number of rotation and table-size independent features (Fig. 1, rows 4+5
middle). Using a Univariate Gaussian clustering algorithm, features from a single input frame are compared against a number of pre-trained
hand and finger postures. By examining multiple frames over time, we capture dynamic information such as a hand moving up or two
fingers moving closer together or farther apart. This allows applications to be developed that understand both different hand postures and
dynamic movements over the Diamond Touch.”
(TR2005-130 at 5)

“The DiamondTouch surface consists of overlapping vertical and horizontal arrays of antennas. The hardware periodically produces frames
of data containing scalar values that mea- sure the proximity of the user’s finger(s) to each antenna.
“The DiamondTouch Library (dtlib) reads these data frames from the DiamondTouch device and affords access to the raw data and to
various abstractions and interpretations of that data, such as the location of the maximum proximity (the touch point) and the bounding box
of the area touched. Other abstractions are possible and are the subject of ongoing research. A weighted interpolation algorithm increases the
effective resolution to 2500 x 1500. Median filtering, hysteresis, and adaptive touch thresholding are used to improve robustness in the face
of RF interference and other environ- mental variables.
“The SDK consists of dtlib (ANSI C), jdt (a Java interface layer), merldt (a Windows application providing mouse emu- lation, projector
calibration, and various diagnostic displays), and a simple multi-user application example.”
(TR2002-48 at 1-2)

03450.51900/4387870.1

8

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
“To interact with existing single user applications, we first use the GroupLab WidgetTap toolkit [8] to determine the location and size of the
GUI elements within it. We then use the Microsoft Send Input facility to relay the gesture and speech input actions to the locations of the
mapped UI elements (Fig. 1, rows 1, 2 and 3). Thus speech and gestures are mapped and transformed into one or more traditional GUI
actions as if the user had performed the interaction sequence via the mouse and keyboard. The consequence is that the application appears to
directly understand the spoken command and gestures.”
(TR2005-130 at 5)

[1c] determining whether
the event object invokes a
scroll or gesture operation
by distinguishing between
a single input point applied
to the touch-sensitive
display that is interpreted
as the scroll operation and
two or more input points
applied to the touchsensitive display that are
interpreted as the gesture
operation;

The Diamond Touch Table System discloses determining whether the event object invokes a scroll or gesture operation by distinguishing
between a single input point applied to the touch-sensitive display that is interpreted as the scroll operation and two or more input points
applied to the touch-sensitive display that are interpreted as the gesture operation. For example:

(TR2005-130 at 4, Fig. 2.)

03450.51900/4387870.1

9

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.”
(TR2005-130 at 6.)

[1d] issuing at least one
scroll or gesture call based
on invoking the scroll or
gesture operation;

The Diamond Touch Table System discloses issuing at least one scroll or gesture call based on invoking the scroll or gesture operation. For
example:

“Fortunately, both Google Earth and Warcraft III are highly interactive, immediately responding to all user commands in a very visual and
often compelling manner. Panning in both produces an immediate response, as does zooming or issuing a ‘Fly to’ command in Google
Earth.”
(TR2005-130 at 5.)
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.”
(TR2005-130 at 5.)
“Our third demo illustrates DiamondTouch’s ability to run with existing applications by providing a mouse emulation mode. DiamondTouch
is well-suited to shared-display applications. It is suitable for front-projected video of the computer display, which facilitates direct
manipulation of user interface elements and provides a shared focus of attention for collaborating users. Possible applications include
command-and control command posts, control rooms, business or technical meetings, and a variety of casual applications in the home, at
schools, and in retail settings.”
(TR2002-48 at 1)

“DIAMONDTOUCH SDK

03450.51900/4387870.1

10

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
The DiamondTouch surface consists of overlapping vertical and horizontal arrays of antennas. The hardware periodically produces frames
of data containing scalar values that measure the proximity of the user’s finger(s) to each antenna.
The DiamondTouch Library (dtlib) reads these data frames from the DiamondTouch device and affords access to the raw data and to various
abstractions and interpretations of that data, such as the location of the maximum proximity (the touch point) and the bounding box of the
area touched. Other abstractions are possible and are the subject of ongoing research. A weighted interpolation algorithm increases the
effective resolution to 2500 x 1500. Median filtering, hysteresis, and adaptive touch thresholding are used to improve robustness in the face
of RF interference and other environmental variables.
“The SDK consists of dtlib (ANSI C), jdt (a Java interface layer), merldt (a Windows application providing mouse emulation, projector
calibration, and various diagnostic displays), and a simple multi-user application example.”
(TR2002-48 at 1-2.)
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.”
(TR2005-130 at 6.)
Our third demo illustrates the mouse emulation capabilities of DiamondTouch. There are two aspects to this demo. First, this capability
allows traditional software to be used with DiamondTouch. We currently have several mouse modes (onetouch, two-touch, etc) and are
experimenting to determine how best to implement a fully-functioning mouse with DiamondTouch. Our mouse emulator works with
traditional software. Second, under Windows the DiamondTouch SDK can generate ‘augmented mouse events’ that contain the user ID info
in an extra field. In this manner C++ Windows code using a Win32 API call can get the ID information for each mouse event as it is
processed by the application. The application then needs to store the state for the different users, and also some mouse information. We have
a sample drawing program (a modified MFC demo) to illustrate the use of ‘augmented mouse events’ for writing applications for
DiamondTouch.
(TR2002-48 at 4.)

[1e] responding to at least
one scroll call, if issued, by

03450.51900/4387870.1

The Diamond Touch Table System discloses responding to at least one scroll call, if issued, by scrolling a window having a view associated
with the event object based on an amount of a scroll with the scroll stopped at a predetermined position in relation to the user input. For
11

U.S. Patent No. 7,844,915
scrolling a window having
a view associated with the
event object based on an
amount of a scroll with the
scroll stopped at a
predetermined position in
relation to the user input;
and

MERL DIAMOND TOUCH TABLE SYSTEM
example:
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.”
(TR2005-130 at 6.)

(TR2005-130 at Table 1.)
See also Tse video – single-finger scroll and two-finger zoom seen at 0:17 to 0:29 under the title “Zoom and Pan”; narrator indicates that
“We’ve added the ability to do the panning action with one hand and it’s also possible to use the zooming action by spreading two fingers
apart or moving to fingers together.” As shown in the Tse video, a move gesture stops the scroll at a predetermined position in relation to
the user input (specifically, the position at which the user input ends).
[1f] responding to at least

03450.51900/4387870.1

12

U.S. Patent No. 7,844,915
one gesture call, if issued,
by scaling the view
associated with the event
object based on receiving
the two or more input
points in the form of the
user input.

MERL DIAMOND TOUCH TABLE SYSTEM
4.1 GoogleEarth
Google Earth is a free desktop geospatial application that allows one to search, navigate, bookmark, and annotate satellite imagery of the
entire planet using a keyboard and mouse. Its database contains detailed satellite imagery with layered geospatial data (e.g., roads, borders,
accommodations, etc). It is highly interactive, with compelling real time feedback during panning, zooming and ‘flying’ actions, as well as
the ability to tilt and rotate the scene and view 3D terrain or buildings. . . . Table 1 provides a partial list of how we mapped Google Earth
onto our multimodal speech and gesture system, while Fig. 2 illustrates Google Earth running on our multimodal, multi user table.
(TR2005-130 at 4.)
“Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.”
(TR2005-130 at 6.)

(TR2005-130 Fig. 2.)

03450.51900/4387870.1

13

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
Video – single-finger scroll and two-finger zoom seen at 0:17 to 0:29 under the title “Zoom and Pan”; narrator indicates that “We’ve added
the ability to do the panning action with one hand and it’s also possible to use the zooming action by spreading two fingers apart or moving
to fingers together.”
As shown in the video, the view of the map is scaled based on receiving two input points from two fingers. As the fingers move closer
together, the view of the map decreases its size. As the fingers move further apart, the view of the map increases its size.

Claim 2
[2] The method as in claim
1, further comprising:
rubberbanding a scrolling
region displayed within the
window by a
predetermined maximum
displacement when the
scrolling region exceeds a
window edge based on the
scroll.

The Diamond Touch Table System discloses rubberbanding a scrolling region displayed within the window by a predetermined maximum
displacement when the scrolling region exceeds a window edge based on the scroll.
“3 DTFlash Authoring Environment
“Our previous research revealed significant shortcomings of traditional tools and development environments for DiamondTouch
applications. The DiamondTouch SDK [10], released with DiamondTouch prototypes by MERL, provides a low-level C API for accessing
data to determine which users are touching a surface at which places. Early research into building on top of this SDK focused on providing
an API based on a general purpose programming environment such as Java [7] or .NET [2]. DTFlash takes a different direction by
leveraging the Macromedia Flash authoring environment to emphasize authoring over programming. For example, the standard Flash
authoring tool can be used to create arbitrary shapes or objects which can then simply be marked with our extensions as being draggable or
rotatable. Literally no coding is needed, yet the new content is “multi-toucher-aware,” allowing multiple people to interact with different
shapes or objects at the same time.
“Through our earlier work, we found that a rapid prototyping tool for multi-user/multi-touch applications requires fundamental low-level
support for a variety of items: simultaneous users; multiple points of input from each user; an authoring environment for creating "multitouch aware" content; multimedia support; the ability to simulate multiple touchers and touch points with a mouse and keyboard; and debugmode overlays for visualizing toucher information. DTFlash provides all these capabilities, in part by defining primitive touch events,
enhanced primitive events, and methods for semantic operations.
“Also of note, DTFlash applications can also work as regular web pages, allowing for simple deployment and ushering in a new dimension
of multi-user enabled web pages that eliminate the need to take turns with the mouse. Flash is also based on vector graphics and optimized
for small downloads, so DTFlash applications have a small memory footprint. But it is the reliance on weak static typing and it's

03450.51900/4387870.1

14

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
"expressiveness" which make Flash particularly well-suited for exploring drastic changes without breaking existing applications and for
facilitating the creation of complex and novel visual interfaces.”
(TR2005-105 at 5-6).
Interaction with the DT Physical Device, and in particular the DTFlash toolkit thereon, showed a sample DTFlash application in the form of
a single-picture flash-based web page comprising a scrolling region. This scrolling region, when scrolled up or down past the natural edge
of the window boundary, continued scrolling into a “new” copy of the web page image. When the scroll was released, the scrolling region
was rubberbanded back by a predetermined maximum displacement amount to place the scrolling region back within its natural boundary
with the window edge exactly coinciding with the edge of the scrolling region.

Claim 3
[3] The method as in claim
1, further comprising:
attaching scroll indicators
to a content edge of the
window.

The Diamond Touch Table System discloses attaching scroll indicators to the content edge of the window. For example:
See MERL video at 0:36 – 0:46, 0:54 – 1:04.
“Our third demo illustrates the mouse emulation capabilities of DiamondTouch. There are two aspects to this demo. First, this capability
allows traditional software to be used with DiamondTouch. We currently have several mouse modes (one- touch, two-touch, etc) and are
experimenting to determine how best to implement a fully-functioning mouse with DiamondTouch. Our mouse emulator works with
traditional software. Second, under Windows the DiamondTouch SDK can generate ‘augmented mouse events’ that contain the user ID info
in an extra field. In this manner C++ Windows code using a Win32 API call can get the ID information for each mouse event as it is
processed by the application. The application then needs to store the state for the different users, and also some mouse information. We have
a sample drawing program (a modified MFC demo) to illustrate the use of ‘augmented mouse events’ for writing applications for
DiamondTouch.”
(TR2002-48 at 3.)
Interaction with the DT Physical Device revealed any number of Windows XP programs native to the clean installation of the operating
system (for example, Internet Explorer) comprised a scroll indicator attached to the content edge of the window.

Claim 4

03450.51900/4387870.1

15

U.S. Patent No. 7,844,915
[4] The method as in claim
1, further comprising:
attaching scroll indicators
to the window edge.

MERL DIAMOND TOUCH TABLE SYSTEM
The Diamond Touch Table System discloses attaching scroll indicators to the window edge. For example:
See MERL video at 0:36 – 0:46, 0:54 – 1:04.
“Our third demo illustrates the mouse emulation capabilities of DiamondTouch. There are two aspects to this demo. First, this capability
allows traditional software to be used with DiamondTouch. We currently have several mouse modes (one- touch, two-touch, etc) and are
experimenting to determine how best to implement a fully-functioning mouse with DiamondTouch. Our mouse emulator works with
traditional software. Second, under Windows the DiamondTouch SDK can generate ‘augmented mouse events’ that contain the user ID info
in an extra field. In this manner C++ Windows code using a Win32 API call can get the ID information for each mouse event as it is
processed by the application. The application then needs to store the state for the different users, and also some mouse information. We have
a sample drawing program (a modified MFC demo) to illustrate the use of ‘augmented mouse events’ for writing applications for
DiamondTouch.”
(TR2002-48 at 3.)
Interaction with the DT Physical Device revealed any number of Windows XP programs native to the clean installation of the operating
system (for example, Internet Explorer) comprised a scroll indicator attached to the window edge.

Claim 5
[5] The method as in claim
1, wherein determining
whether the event object
invokes a scroll or gesture
operation is based on
receiving a drag user input
for a certain time period.

The Diamond Touch Table System discloses attaching scroll indicators to the window edge. For example:
“Gesture Engine. Since recognizing gestures from multiple people on a table top is still an emerging research area [25][26], we could not
use existing 3rd party gesture recognizers. Consequently, we developed our own Diamond Touch gesture recognition engine to convert the
raw touch information produced by the DiamondTouch SDK into a number of rotation and table-size independent features (Fig. 1, rows 4+5
middle). Using a Univariate Gaussian clustering algorithm, features from a single input frame are compared against a number of pre-trained
hand and finger postures. By examining multiple frames over time, we capture dynamic information such as a hand moving up or two
fingers moving closer together or farther apart. This allows applications to be developed that understand both different hand postures and
dynamic movements over the Diamond Touch.
(TR2005-130 at 3.)
Because we reserve gestures for spatial manipulations, very little learning is needed: panning by dragging one’s finger or hand across the
surface is easily understood by others, as is the surface stretching metaphor used in spreading apart or narrowing two fingers to activate
discrete or continuous zooming in Google Earth.

03450.51900/4387870.1

16

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
(TR2005-130 at 3)

Claim 6
[6] The method as in claim
1, further comprising:
responding to at least one
gesture call, if issued, by
rotating a view associated
with the event object based
on receiving a plurality of
input points in the form of
the user input.

The Diamond Touch Table System discloses responding to at least one gesture call, if issued, by rotating a view associated with the event
object based on receiving a plurality of input points in the form of the user input. For example:

“4.1 GoogleEarth
Google Earth is a free desktop geospatial application that allows one to search, navigate, bookmark, and annotate satellite imagery of the
entire planet using a keyboard and mouse. Its database contains detailed satellite imagery with layered geospatial data (e.g., roads, borders,
accommodations, etc). It is highly interactive, with compelling real time feedback during panning, zooming and ‘flying’ actions, as well as
the ability to tilt and rotate the scene and view 3D terrain or buildings. . . . Table 1 provides a partial list of how we mapped Google Earth
onto our multimodal speech and gesture system, while Fig. 2 illustrates Google Earth running on our multimodal, multi user table.”
(TR2005-130 at 4.)
“Gesture Engine. Since recognizing gestures from multiple people on a table top is still an emerging research area [25][26], we could not
use existing 3rd party gesture recognizers. Consequently, we developed our own Diamond Touch gesture recognition engine to convert the
raw touch information produced by the DiamondTouch SDK into a number of rotation and table-size independent features (Fig. 1, rows 4+5
middle). Using a Univariate Gaussian clustering algorithm, features from a single input frame are compared against a number of pre-trained
hand and finger postures. By examining multiple frames over time, we capture dynamic information such as a hand moving up or two
fingers moving closer together or farther apart. This allows applications to be developed that understand both different hand postures and
dynamic movements over the Diamond Touch.”
(TR2005-130 at 5)

Claim 7
[7] The method as in claim The Diamond Touch Table System discloses a device that is one of: a data processing device, a portable device, a portable data processing
1, wherein the device is one device, a multi touch device, a multi touch portable device, a wireless device, and a cell phonet. For example:
of: a data processing
device, a portable device, a
03450.51900/4387870.1

17

U.S. Patent No. 7,844,915
portable data processing
device, a multi touch
device, a multi touch
portable device, a wireless
device, and a cell phone.

MERL DIAMOND TOUCH TABLE SYSTEM
“ABSTRACT
DiamondTouch [1] is a multi-touch input technology that supports multiple, simultaneous users; it can distinguish who is touching where.
We present the DiamondTouch SDK; it provides support for the development of applications that utilize DiamondTouch’s capabilities to
implement computer supported collaboration and rich input modalities (such as gestures). Our first demo illustrates the basic utilities and
functionality of our system.”
(TR2002-48 at 2).

03450.51900/4387870.1

18

U.S. Patent No. 7,844,915

MERL DIAMOND TOUCH TABLE SYSTEM
(TR2002-48 at 2).

“Our infrastructure is illustrated in Fig. 1. A standard Windows computer drives our infrastructure software, as described below.
The table is a 42” MERL Diamond Touch surface [6] with a 4:3 aspect ratio; a digital projector casts a 1280x1024 pixel image on the table’s
surface. This table is multi-touch sensitive, where contact is presented through the DiamondTouch SDK as an array of horizontal and
vertical signals, touch points and bounding boxes (Fig. 1, row 5). The table is also multi-user, as it distinguishes signals from up to four
people. While our technology uses the Diamond Touch, the theoretical motivations, strategies developed, and lessons learnt should apply to
other touch/vision based surfaces that offer similar multi user capabilities.”
(TR2005-130 at 2).

(Tse at 3, Fig. 1.)
Claim 8
[8p] A machine readable
storage medium storing

03450.51900/4387870.1

See [1p]

19

U.S. Patent No. 7,844,915
executable program
instructions which when
executed cause a data
processing system to
perform a method
comprising:
[8a] receiving a user input,
the user input is one or
more input points applied
to a touch-sensitive display
that is integrated with the
data processing system;
[8b] creating an event
object in response to the
user input;
[8c] determining whether
the event object invokes a
scroll or gesture operation
by distinguishing between
a single input point applied
to the touch-sensitive
display that is interpreted
as the scroll operation and
two or more input points
applied to the touchsensitive display that are
interpreted as the gesture
operation;
[8d] issuing at least one
scroll or gesture call based
on invoking the scroll or
gesture operation;
03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM

See [1a]

See [1b]

See [1c]

See [1d]

20

U.S. Patent No. 7,844,915
[8e] responding to at least
one scroll call, if issued, by
scrolling a window having
a view associated with the
event object; and
[8f] responding to at least
one gesture call, if issued,
by scaling the view
associated with the event
object based on receiving
the two or more input
points in the form of the
user input.
Claim 9
[9] The medium as in claim
8, further comprising:
rubberbanding a scrolling
region displayed within the
window by a
predetermined maximum
displacement when the
scrolled region exceeds a
window edge based on the
scroll.
Claim 10
[10] The medium as in
claim 8, further
comprising: attaching scroll
indicators to a content edge
of the view.
Claim 11

03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM
See [1e]

See [1f]

See [2]

See [3]

21

U.S. Patent No. 7,844,915
[11] The medium as in
claim 8, further
comprising: attaching scroll
indicators to a window
edge of the view.
Claim 12
[12] The medium as in
claim 8, wherein
determining whether the
event object invokes a
scroll or gesture operation
is based on receiving a drag
user input for a certain time
period.
Claim 13
[13] The medium as in
claim 8, further
comprising: responding to
at least one gesture call, if
issued, by rotating a view
associated with the event
object based on receiving a
plurality of input points in
the form of the user input.
Claim 14
[14] The medium as in
claim 8, wherein the data
processing system is one
of: a data processing
device, a portable device, a
portable data processing

03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM
See [4]

See [5]

See [6]

See [7]

22

U.S. Patent No. 7,844,915
device, a multi touch
device, a multi touch
portable device, a wireless
device, and a cell phone.
Claim 15
[15p] An apparatus,
comprising:
[15a] means for receiving,
through a hardware device,
a user input on a touchsensitive display of the
apparatus, the user input is
one or more input points
applied to the touchsensitive display that is
integrated with the
apparatus;
[15b] means for creating an
event object in response to
the user input;
[15c] means for
determining whether the
event object invokes a
scroll or gesture operation
by distinguishing between
a single input point applied
to the touch-sensitive
display that is interpreted
as the scroll operation and
two or more input points
applied to the touchsensitive display that are
03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM

See [1p]
See [1a]

See [1b]

See [1c]

23

U.S. Patent No. 7,844,915
interpreted as the gesture
operation;
[15d] means for issuing at
least one scroll or gesture
call based on invoking the
scroll or gesture operation;
[15e] means for responding
to at least one scroll call, if
issued, by scrolling a
window having a view
associated with the event
object; and
[15f] means for responding
to at least one gesture call,
if issued, by scaling the
view associated with the
event object based on
receiving the two or more
input points in the form of
the user input.
Claim 16
[16] The apparatus as in
claim 15, further
comprising: means for
rubberbanding a scrolling
region displayed within the
window by a
predetermined maximum
displacement when the
scrolling region exceeds a
window edge based on the
scroll.
03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM

See [1d]

See [1e]

See [1f]

See [2]

24

U.S. Patent No. 7,844,915
Claim 17
[17] The apparatus as in
claim 15, further
comprising: means for
attaching scroll indicators
to a content edge of the
window.
Claim 18
[18] The apparatus as in
claim 15, further
comprising: means for
attaching scroll indicators
to the window edge.
Claim 19
[19] The apparatus as in
claim 15, wherein
determining whether the
event object invokes a
scroll or gesture operation
is based on receiving a drag
user input for a certain time
period.
Claim 20
[20] The apparatus as in
claim 15, further
comprising: means for
responding to at least one
gesture call, if issued, by
rotating a view associated
with the event object based
on receiving a plurality of

03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM
See [3]

See [4]

See [5]

See [6]

25

U.S. Patent No. 7,844,915
input points in the form of
the user input.
Claim 21
[21] The apparatus as in
claim 15, wherein the
apparatus is one of: a data
processing device, a
portable device, a portable
data processing device, a
multi touch device, a multi
touch portable device, a
wireless device, and a cell
phone.

03450.51900/4387870.1

MERL DIAMOND TOUCH TABLE SYSTEM

See [7]

26