Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 933

Declaration of Bogue in Support of #930 Administrative Motion to File Under Seal Samsung's Motion for Summary Judgment filed bySamsung Electronics Co. Ltd.. (Attachments: #1 Exhibit 1, #2 Exhibit 2, #3 Exhibit 3, #4 Exhibit 4, #5 Exhibit 5)(Related document(s) #930 ) (Maroulis, Victoria) (Filed on 5/17/2012)

EXHIBIT 2

MITSUBISHI ELECTRIC RESEARCH LABORATORIES
http://www.mefl.com

Multi-User Multi-Touch Games on
DiamondTouch with the DTFlash Toolkit

Alan Esenther and Kent Wittenburg

TR2005-105 December 2005

Abstract
Games and other forms of tabletop electronic entertainment are a natural application of the new
multi-user multi-touch tabletop technology DiamondTouch [3]. Electronic versions of familiar tabletop games such as ping-pong or air hockey require simultaneous touch events that can
be uniquely associated with different users. Multi-touch two-handed gestures useful for, e.g.,
rotating, stretching, capturing, or releasing also have natural applications for entertainment applications built on electronic tabletops. Here we show a set of games that are illustrative of the
capabilities of an underlying authoring toolkit we cal DTFlash. DTFlash is designed so that
those familiar with Macromedia Flash authority tools can add multi-user multi-touch gestures
and behaviors to web-enabled games and other applications for the DiamondTouch table.
Intelligent Technologies for Interactive Entertainment 2005

This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part
without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include
the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an aclcnowledgment of
the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or
republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All
rights reserved.
Copyright @ Mitsubishi Electric Research Laboratories, Inc., 2005
201 Broadway, Cambridge, Massachusetts 02139

SAIqNDCA00035 993

SAMNDCA00035994

Multi-User Multi-Touch Games on DiamondTouch
with the DTFlash Toolkit
Alan Esenther and Kent Wittenburg
Mitsubishi Electric Research Laboratories, Inc.
201 Broadway
Cambridge, MA 02139 USA
{Esenther, Wittenburg} @merl.com

Abstract. Games and other forms of tabletop electronic entertainment are a natural application of the
new- multi-user multi-touch tabletop technology DiamondTouch [3]. Electronic versions of familiar
tabletop games such as ping-pong or air hockey require simultaneous touch events that can be uniquely
associated with different users. Multi-touch two-handed gestures useful for, e.g., rotating, stretching,
capturing, or releasing also have natural applications for entertainment applications built on electronic
tabletops. Here we show- a set of games that are illustrative of the capabilities of an underlying
authoring toolkit we call DTFlash. DTFlash is designed so that those familiar with Macromedia Flash
authoring tools can add multi-user multi-touch gestures and behaviors to web-enabled games and other
applications for the DiamondTouch table.

1 Introduction
DiamondTouch [3] is a touch technology suitable for electronic tabletops that affords simultaneous
multi-user multi-touch events. Unlike other competing touch technologies, it can reliably associate touch
events, including multi-finger or multi-handed events, with specific users. The way it works is that a twodimensional antenna array is embedded in a surface emitting a small electric signal. Receivers embedded in
chairs, table edges, floor sections, or batons receive the signal when a user "completes the circuit." A user
touching the surface that is also making contact with a receiver causes an increase in capacitive signal
strength that allows registration of the touched areas with the associated receiver. A computer display is
projected onto the surface, providing visual content that is calibrated with the touches.
Previous research on tabletop computing has emphasized new forms of gestures and layouts as well as
how such technology can be used for collaborative applications such as design, command and control, and
story-telling [1, 4-6, 8-9]. Here we illustrate how tabletop games can take advantage of DiamondTouch and
also introduce our new authoring environment called DTFlash.

2 Tabletop Games
Tabletop games, certainly competitive ones, on a touch surface require that the system distinguishes one
participant’s actions from another. One could of course associate a specific area of the surface with a
specific player or else structure the interaction so that turn-taking was required. However, such constraints
clearly inhibit the possible design space. Users interacting with multi-user single-display environments
such as electronic tabletops naturally expect to be able to interact at the same time on the same or different
areas of the surface. Here we illustrate a set of simple games that demonstrate some of the interaction
possibilities enabled by our toolkit.

Ballpit (Figure 1) is a game in which players drag or flick balls out of the way to be the first to find a
striped ball hiding underneath one of them. This was inspired by "Where’s Waldo" puzzles. It is a simple
illustration of the use of widgets with unique user IDs [5] that afford simultaneous interaction.

SAMNDCA00035995

Fig. 1. Ballpit game players drag or flick balls out of the way to reveal the hidden object.

SpaceBalls (Figure 2) is a more complex application where players use either their finger tip or the
rectangle created by multiple fingers to delete colorful 3-D balls that bounce off the sides of the screen.
There are a few "super" balls which can only be deleted if two or more players touch it at the same time.
Interestingly, a super ball can also be deleted if one player touches the ball and then "high fives" another
player, since the cross-coupling between the players is detected. Note that users touching each other as one
of them touches the table completes a circuit that is distinguishable from the circuit created by a single user.

Fig. 2. SpaceBalls--players touch or swipe to delete colored balls.

The Flick Documents application (Figure 3) illustrates the capability for users to drag or flick objects on
the table that require specific orientations. A player could pass a document or an image to another player
through flicking or dragging. It’s possible to specify whether content should always face the center of the
table or a particular side.
~
F ............

,

¥ t.’~

Fig. 3. Flick Documents players drag or flick objects that can have varying orientation.
The Collaborative Rotating application (Figure 4) lets players simultaneously manipulate the same
object. One player’s finger determines the location of the upper left comer of the image and the other

SAMNDCA00035996

player’s finger determines the location of the lower left corner. The image is rapidly rotated and resized
accordingly.

r

Fig. 4. Collaborative Rotating--players simultaneously interact with the same object.

Ice-scraper (Figure 5) is an interactive image un-masking application where players use multi-finger
touches to unmask ("scrape the ice ofF’) an image underneath. The distance between the fingers determines
the width of a tool’s effective area. This functionality was illustrated also in SquiggleDraw from U. Calgary
[21.

Fig. 1. Ice-scraper players use multi-finger touches to reveal an image underneath.

The interactions and gestures shown above are only just a glimpse at the possibilities of multi-touch
multi-user interactions that are enabled by DTFlash. Of course the games we show are very simple. Their
primary purpose is to illustrate the interaction possibilities afforded by our toolkit.

3 DTFlash Authoring Environment
Our previous research revealed significant shortcomings of traditional tools and development
environments for DiamondTouch applications. The DiamondTouch SDK [10], released with
DiamondTouch prototypes by MERL, provides a low-level C API for accessing data to determine which
users are touching a surface at which places. Early research into building on top of this SDK focused on
providing an API based on a general purpose programming environment such as Java [7] or .NET [2].
DTFlash takes a different direction by leveraging the Macromedia Flash authoring environment to

SAMNDCA00035 997

emphasize authoring over programming. For example, the standard Flash authoring tool can be used to
create arbitrary shapes or objects which can then simply be marked with our extensions as being draggable
or rotatable. Literally no coding is needed, yet the new content is "multi-toucher-aware," allowing multiple
people to interact with different shapes or objects at the same time.
Through our earlier work, we found that a rapid prototyping tool for multi-user/multi-touch applications
requires fundamental low-level support for a variety of items: simultaneous users; multiple points of input
from each user; an authoring environment for creating "multi-touch aware" content; multimedia support;
the ability to simulate multiple touchers and touch points with a mouse and keyboard; and debug-mode
overlays for visualizing toucher information. DTFlash provides all these capabilities, in part by defining
primitive touch events, enhanced primitive events, and methods for semantic operations.
Also of note, DTFlash applications can also work as regular web pages, allowing for simple
deployment and ushering in a new dimension of multi-user enabled web pages that eliminate the need to
take turns with the mouse. Flash is also based on vector graphics and optimized for small downloads, so
DTFlash applications have a small memory footprint. But it is the reliance on weak static typing and it’s
"expressiveness" which make Flash particularly well-suited for exploring drastic changes without breaking
existing applications and for facilitating the creation of complex and novel visual interfaces.

4 Conclusion
An electronic tabletop is a natural for games and other entertainment applications. Many such
applications can take advantage of the multi-touch multi-user capabilities that DiamondTouch affords.
DTFlash is the latest toolkit to emerge for building applications on DiamondTouch, and we hope that it will
make DiamondTouch development accessible to a larger audience of authors, rather than programmers,
who are already familiar with Macromedia Flash.

References
1. Cappelletti A.; Gelmini, G.; Pianesi, F., Rossi, F.; and Zancanaro, M.: Enforcing Cooperative Storytelling: First
Studies, In Proceedings of International Conference on Advanced Learning Technologies (ICALT), September
2004.
2. Diaz-Marino, R.A., Tse, E, and Greenberg, S. (2003) Programming for Multiple Touches and Multiple Users: A
Toolkit for the DiamondTouch Hardware. Demonstration in Companion Proceedings of ACM User Interface
Software & Technology (UIST), 2003, ACM Press.
3. Dietz, P.H.; Leigh, D.L., "DiamondToueh: A Multi-User Touch Technology", ACM Symposium on User Interface
Software and Technology (UIST), pp. 219-226, November 2001, ACM Press.
4. Rekimoto, J.: SmartSkin: an Infrastructure for Freehand Manipulation on Interactive Surfaces, Proceedings of ACM
Conference on Computer Human Interaction (CHI), pp. 113-120, 2002, ACM Press.
5. Ryall, K.; Esenther, A.; Everitt, K.; Forlines, C.; Ringel Morris, M.; Shen, C.; Shipman, S.; and Vernier, F.:
iDwidgets: Parameterizing Widgets by User Identity. In Proceedings of Interact, Rome, Italy, September 2005.
6. Shen, C.; Lesh, N.B.; Vernier, F.; Forlines, C.; Frost, J., "Building and Sharing Digital Group Histories", ACM
Conference on Computer Supported Cooperative Work (CSCW), pp. 3, November 2002, ACM Press.
7. Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table
Interaction", ACM Conference on Human Factors in Computing Systems (CHI), pp. 167-174, April 2004, ACM
Press.
8. Tandler, P.; Prante, T.; Mueller-Tomfelde, C., Streitz, N.; and Steinmetz, R.: Connectables: Dynamic Coupling of
Displays for the Flexible Creation of Shared Workspaces, ACM Symposium on User Interface Software and
Technology (UIST), pp. 11-20, November 2001, ACM Press.
9. Wu, M.; Balakrishnan, R.: Multi-Finger and Whole Hand Gestural Interaction Techniques for Multi-User Tabletop
Displays, ACM Symposium on User Interface Software and Technology (UIST), November 2003, pp. 193-202,
ACM Press.
10. Esenther, A.;Forlines, C.; Ryall, K.; Shipman, S.: DiamondToueh SDK: Support for Multi-User Multi-Touch
Applications, ACM Conference on Computer Supported Cooperative Work (CSCW), November 2002.

SAMNDCA00035998