Apple Inc. v. Samsung Electronics Co. Ltd. et al

Filing 559

Declaration of Deok Keun Matthew Ahn in Support of #558 Response ( Non Motion ), filed byApple Inc.. (Attachments: #1 Exhibit 1.01, #2 Exhibit 1.02, #3 Exhibit 1.03, #4 Exhibit 1.04, #5 Exhibit 1.05, #6 Exhibit 1.06, #7 Exhibit 1.07, #8 Exhibit 1.08, #9 Exhibit 1.09, #10 Exhibit 1.10, #11 Exhibit 1.11, #12 Exhibit 1.12, #13 Exhibit 1.13)(Related document(s) #558 ) (Jacobs, Michael) (Filed on 12/29/2011)

EXHIBIT 1.05

9

5,202,961

lected flow velocity and the selected flow direction
being selected in a one of a tentative set mode and
a fixed set mode, in the tentative set mode the se-

10

9. A controller as recited in claim 1, wherein said
control icon image includes a first image when said
control icon is operating in said tentative set mode and

lected flow velocity and the selected flow direction
wherein said control icon image includes a second
being continuously variable according to the se- 5 image when said control icon is operating in said fixed
lected position of the control icon image while the
set mode,

button is in a one of two button states, in the fixed
10. A controller as recited in claim 9, wherein said
set mode the selected flow velocity and the seslide indicator includes at least one interactive control
lected flow direction being independent of control
dial for indicating a range of gradational flow velocities
icon image position, said tentative set mode an said 10 for said sequential information in a predetermined flow
fixed set mode being alternatively selectable
direction and includes at least one interactive control
through operation of said control device; and
button for indicating a predetermined flow velocity for
means for detecting the presence of said control icon
said sequential information in said predetermined flow
image at said selected position and for generating
direction, wherein said control icon image is operative
said control signals corresponding to said selected 15 to be moved to said selected position in response to said
flow direction and said selected flow velocity for

control dial being selected by said control device, and

use in regulating the flow of said sequential inforwherein said control icon image is operative to be
mation.
moved to said selected position in response to said con2. A controller as recited in claim 1, wherein said
trol button being selected by said control device.
slide indicator includes at least one interactive control 20 11. A controller as recited in claim 10, wherein said
dial for indicating a range of gradational flow velocities
slide indicator image has a first region for indicating a
for said seguential information in a predetermined flow

zero flow velocity for said sequential information, and a

direction and includes at least one interactive control
second region contiguous to said first region for indicatbutton for indicating a predetermined flow velocity for
ing a forward flow velocity of said sequential informasaid sequential information in said predetermined flow 25 tion.
direction, wherein said control icon image is operative
12. A controller as recited in claim 11, and further
to be moved to said selected position in response to said
including a third region contiguous to said first region
control dial being selected by said control device, and
for indicating a backward flow velocity of said sequenwherein said control icon image is operative to be
tial information.
moved to said selected position in response to said con- 30 13. A controller as recited in claim 12, wherein said
trol button being selected by said control device.

3. A controller as recited in claim 2, wherein
slide indicator includes a plurality of said control
tons, and wherein said control buttons are radio
tons
4. A controller as recited in claim 1, wherein

first region includes one of said interactive control but-

said
butbut-

tons, wherein said second region includes one of said
control dials, and wherein said third region includes one
of said control dials.
35
14. A controller for producing control signals for use
said
in regulating the flow of sequential information, com-

slide indicator image has a first region for indicating a

zero flow velocity for said sequential information, and a
second region contiguous to said first region for indicating a forward flow velocity of said sequential informa- 40
tion
5. A controller as recited in claim 4, and further including a third region contiguous to said first region for
indicating a backward flow velocity of said sequential
information.
45
6. A controller as recited in claim 5, wherein said
slide indicator includes at least one interactive control
dial for indicating a range of gradational flow velocities
for said sequential information in a predetermined flow
direction and includes at least one interactive control 50
button for indicating a predetermined flow velocity for
said sequential information in said predetermined flow
direction, wherein said control icon image is operative
to be moved to said selected position in response to said

prising:

a first means for indicating a range of selectable flow
directions and flow velocities for said sequential
information;
a second means responsive to a user for choosing a
selection position from said range of selectable
flow directions and flow velocities, said selection
position corresponding to a selected flow direction
and a selected means for operating in a first mode
for tentatively maintaining said selection position,
means for operating in a second mode for fixedly
maintaining said selection position, and means for
switching between said first and second modes
while maintaining said selection position; and
a third means responsive to said second means for
detecting said selection position and producing said
control signals for regulating the flow of said sequential information as selected by said second

control dial being selected by said control device, and 55

means in said selected flow direction and at said

wherein said control icon image is operative to be
moved to said selected position in response to said control button being selected by said control device.
7. A controller as recited in claim 6, wherein said first
region includes one of said interactive control buttons, 60
wherein said second region includes one of said control
dials, and wherein said third region includes one of said
control dials.
8. A controller as recited in claim 7, wherein said
control buttons are radio buttons, wherein said second 65
re on further includes a plurality of said radio buttons,
an wherein said third region further includes a plurality of said radio buttons.

selected flow velocity.
15. A controller as recited in claim 14, wherein said
first means includes a slide mdicator having at least one
interactive control dial for indicating a range of gradational flow velocities for said sequential information m a
predetermined flow direction and having at least one
interactive control button for indicating a predetermined flow velocity for said sequential information m
said predetermined flow direction.
16. A controller as recited in claim 15, wherein said
slide indicator includes a plurality of said control buttons, and wherein said control buttons are radio buttons.

APLNDC00028552

11

5,202,961

12

17. A controller as recited in claim 14, wherein said
27. A controller as recited in claim 26, wherein said
first means includes a slide indicator having a first refirst means includes at least one interactive control dial
gion for indicating a zero flow velocity for said sequenfor indicating a range of gradational flow velocities for
tial information, and a second region contiguous to said
said sequential information in a predetermined flow
first region for indicating a forward flow velocity for 5 direction and includes one or more mteractive control
said sequential information.
buttons for indicating a predetermined flow velocity for
18. A controller as recited in claim 17, and further
said sequential information in said predetermined flow
including a third region contiguous to said first region
direction.
for indicating a backward flow velocity of said sequen- 10 first region includesas recited in interactivewherein but28. A controller one of said claim 27, control said
tial information.
19. A controller as recited in claim 18, wherein said
slide indicator includes at least one interactive control

tons, wherein said second region includes one of said
control dials, and wherein said third region includes one

dial for indicating a range of gradational flow velocities
of said control dials.
for said sequential information in a predetermined flow
29. A controller for use in combmation with a display
direction and includes at least one interactive control 15 system and a user operable control device for producing
button for indicating said predetermined flow velocity
control signals for regulating the flow direction and the
for said sequential information in a predetermined flow
flow velocity of sequential information, said control
device including means for selecting and manipulating
dr20
controller as recited in claim 19, wherein said
images on a display of said display system, comprising:
first region includes one of said interactive control buta first means for generating an interactive controller
tons, wherein said second region includes one of said
image for display on said display, said controller
control dials and a plurality of said control buttons, and
image including a plurality of selectable markers
wherein said third region includes one of said control
and a slide indicator, each said marker corresponddials and a plurality of said control buttons.
25
ing to a predetermined flow velocity for said se21. A controller as recited in claim 17, wherein said
quential information in a predetermined flow direccontrol buttons are radio buttons.
22. A controller as recited in claim 14, wherein said
controller is used in combination with a display system
and a user operable control device including means for
selecting and manipulating images on a display of said
display system, wherein said first means includes means
for generating an interactive slide indicator image portraying said range on said display, and wherein said

tion, said slide indicator corresponding to a range
of gradational flow velocities for said sequential
information in one or more predetermined flow
directions;
a second means responsive to a user for selecting a
selection position along said controller image corresponding to a selected flow direction and a selected flow velocity for said sequential informa-

second means further includes means for generating an 35
interactive control icon image on said display visually
roximate to said slide indicator image, said control
con image being operative to be moved by said control

don; means responsive to said second means for
a third and
detecting said selection position and generatmg
said control signals for regulating the flow of said

device to said selected position along said slide indica-

sequential information as selected by said second

tor image, said third means being operative to detect the y
sition of said control icon image along said slide indi-

means in said selected flow direction and at said
selected flow velocity, said second means having a

tor image as said selection position and to generate

first operating mode, a second operating mode, and

said control signals corresponding to said selected flow
direction and said selected flow velocity for use in regulating the flow of said sequential information45

a third operating mode, said third means bemg
operative to tentatively generate said control signals in response to said first operating mode, said

23. A controller as recited in claim 22, wherein said
control icon image has a first image when said second
means is operating in said first mode and wherein said
control icon image has a second image when said sec-

third means being operative to fixedly generate
said control signals in response to said second oper-

on A co ro err as ecm din
2
herein said y
first means includes at least one interactive control dial
for indicating a range of gradational flow velocities for
d
ential information in a predetermined flow
ec n and includes at least one interactive control 55
button for indicating a predetermined flow velocity for
said sequential information in said predetermined flow

ating mode, said third operating mode being operative to switch between said first o ratm m e
w n u t en temsni contreol signals.
30. A controller as recited in claim 29, wherein said
slide indicator includes at least one interactive control
dial representing said range of gradational flow velocities, and wherein each of said markers includes at least
one interactive control button representing a predetermined flow velocity and a predetermined flow direc-

da25œ.
ntroller as recited in claim 22, wherein said
. A controller as recited in claim 30, wherein said
slide indicator image has a first region for indicating a 60 slide indicator includes a first region corresponding to a
zero flow velocity for said sequential information, and a
zero flow velocity for said sequential information, and a
second region contiguous to said first region for indicatsecond region contiguous to said first region corrg
ing a forward flow velocity for said sequential informasponding to a forward range of gradational flow velocaties for said sequential information.
tio . A controller as recited in claim 25, and further 65 32. A controller as recited in claim 31, and further
clud
a third region contiguous to said first region
including a third region contiguous to said first region
r ind ting a backward flow velocity of said sequencorresponding to a backward range of gradational flow
velocities for said sequential information.
tial information.

APLNDC00028553

5,202,961

13

14

33. A controller as recited in claim 32, wherein said
mined flow velocity for said sequential information in
first region includes one of said interactive control butone of said predetermined flow directions.
tons, wherein said second region includes one of said
41. A controller as recited in claim 40, wherein said
control dials and a plurality of said control buttons, and
first region includes one of said interactive control butwherein said third region includes one of said control -5 tons, wherein said second region includes one of said
dials and a plurality of said control buttons.
control dials, and wherein said third region includes one
34. A controller as recited in claim 33, wherein said
of said control dials.

control buttons are radio buttons.
42. A method for producing control signals for con35. A controller as recited in claim 29, wherein said
trolling the flow of sequential information, comprising
second means further includes means for generating an 10 the steps of:
interactive control icon for display on said display visuselecting a flow operation mode for the flow of the
ally proximate to said slide indicator, said control icon
sequential information, said flow operation mode
be
rative to be moved by said control device to
including a tentative flow operation mode for consai se tion position along said slide indicator, said
trolling the flow of said sequential information
third means being operative to detect the position of 15
according to a position of a control icon image
said control icon along said slide indicator as said selection position and to generate said control signals corresponding to said selected flow direction and said se-

while a selecting means of a control device is in a
one of two states, on a temporary basis and a fixed
flow operation mode for maintaining a fixed flow

lected flow velocity for use in regulating the flow of
ial inf rmation
20
said sequ troller as recited in claim 35, wherein said
control icon has a first image when said second means is

of said sequential information;
selecting a selected flow direction and a selected flow
velocity for the sequential information; and
producing control signals for regulating the flow of

said first mode and wherein said control
c nm as amsecond image when said second means is

the sequential information in accordance with e
selected flow direction and the selected flow veloc-

o
e t 1 esr
ec e ein claim 35, wherein said 25
it e method of claim 42, and further including the
slide indicator includes at least one interactive control
steps of
n

e

30

di

38. A controller as recited in claim 35, wherein to a
slide indicator includes a first region corresponding said 35 44.velocity.
The method of claim 43, wherein said control icon
zero flow velocity for said sequential information, and a
has a first image when said tentative flow operation
"""!..

°"ro°°"":"°"'s.'°r'ida ne ion

odse

edmí

eew

e

te

ti forAsai
uential inf rma
aim 38, and further 40 of selecting said first image to select said tentative flow
including a third region contiguous to said first region
operation mode and selecting said second image to
corresponding to a backward range of gradational flow
seleect d fmix flow ope ion mdheerein said step of

ve i esc r 11 rœa r
d c
, wherein said
selecting said flow direction and said flow velocity
slid indicator includes at least one interactive control 45 includes the step of moving said control icon to a seer
se e ti 1ri f rm to
noe of d pred
termined flow directions and includes one or more in-

fl w-dir t onna dæs d flo vbe oc y ano d
flow of the sequential information accordingly.

teractive control buttons representing said predeter50

55

60

65

APLNDC00028554

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllIl
USOO5428730A
United States Patent (19]
Baker et al•
[54) MULTIMEDIA SYSTEM HAVING
SOFTWARE -· -ANISM PROVIDING
STANDARDIZED INTERFACES AND
CONTROLS FOR THE OPERATION OF
MULTIMEDIA DEVICES
[75]

Inventors: Randolph S. Baker, Atlanta, Ga.;
Richard E. DuLaney, Lighthouse

[11]
[45]

Warinner, Woodstock, both of Ga.;
Scott J. Broussard, Boca Raton, Fla.
[73]

Assignee:

International Business Machines
Corporation, Armonk, N.Y.

[21] Appl. No.: 991,034
[22] Filed:

Dec. 15, 1992

[51] Int. CI.6 .............................................. GO6F 15/00
[52] U.S. CI.
....... 395/154; 395/161
[58] Field of Search ............... 395/154, 155, 157, 159,
395/161
[56]

References Cited

U.S. PA a sa x DOCUMENTS
4,845,644 7/ 989 Anthias et al. ...................... 395/157
4,866,638 9/ 989 Cosentino et al. .................. _95/155
4,901,221 2/ 990 Kodosky et al. ................... .95/159

4,914,568 4/ 990 Kodosky et al. ......... ....... ¯95/275
5,040,131 8/ 99
5,041,992 8/ 99

Torres ----..---........ 195/156
Cunningham et al. ............. 295/155

5,060,140 10/ 99

Brown et al. ....................... 364/200

5,065,345 11/199

Knowles et al. .................... 395/154

5,109,515 4/ 992 Laggis et al- ..................-.... 395/725
5,291,587 3/ 994 Kodosky et al. ................... 395/500
5,293,476 3/ 994 Wolber et al. ------................ 395/ 59

5,309,556 5/ 994 Sismilich ............................. 395/ 61
5,313,574 5/ 994 Beethe ..--.....----..-................. 395/ 59
5,333,299 7/ 994 Koval et al. .--..................... 395/550

OTHER PUBLICATIONS
Cohen, "LabVIEW Controls DOS, Tape Systems",
MacWEEK, v. 6, No. 42, Nov. 23, 1992, p. 12 (1).

5,428,730
Jun. 27, 1995

Rizzo, "LabVIEW 2", MacUSER, v. 7, No. 5, May
1991, p. 58 (2).
Santori, "An Instrument That Isn't Really", IEEE
Spectrum, v. 27, No. 8, Aug. 1990, pp. 36-39.
Schmalzel, et al., "Instrumentation Curriculum:.From
ATE to VXI," IEEE IMTC '92, May 1992, pp.
265-267.

Point; Patricia A. Esack, Boca
Raton, both of Fla.; Stephen R.
Henson, Atlanta, Ga.; Scott R. Swix,
Half Moon Bay, Calif.; George E.
Wagener, Marietta; Frank D.

Patent Number:
Date of Patent:

(List continued on next page.)
Primary Examiner--Heather R. Herndon
Assistant Examiner-Joseph R. Burwell
Attorney, Agent, or Firm-Bernard D. Bogdon; Stephen
A. Becker

[57]

--•«CT

A multimedia system includes a multimedia device con-

trol program having a plurality of application program
selectable controls for controlling operation of a like
plurality of different classes of multimedia devices.
Each class has a different set of controllable device
functions or attributes. In response to selection of a class
by an application program, a unique control panel is
created in a user interface for controlling operation of a
device. The selected control performs all of the processing actions necessary to control the device in response
to user selections on the interface, without involvement
of the application program in such actions. The multi-

media control program includes a plurality of panel
templates from which control panels are created in a
control screen. Each panel has a plurality of controllers
that can manipulated by a user to control a corresponding function of a multimedia device. The templates
provide a consistent interfaces having a common look
and feel. The multimedia control program also includes
a plurality of class control programs one or more of
which are selected by an application program to be
attached to a like number of multimedia devices for
controlling operation of the devices in response to user
selections on the user interface, thereby relieving the
application program of performing the necessary control functions.

20 Claims, 7 Drawing Sheets

APLNDC00028555

5,428,730
Page 2

O-·---- PUBLICATIONS
Petzold, "An Introduction to the Media Control Interface", PC Magazine, v. 11, No. 1, Jan. 14, 1992, p. 407
(5).
Petzold, "MCI Command Strings: The Easy Multimedia Interface", PC Magazine, v. I I, No. 2, Jan. 28, 1992,
p. 329 (5).
Varney, "Multimedia Development Kit Now Runs
Under Motif", Digital News & Review, v. 9, No. 17,
Sep. 14, 1992, p. 15 (1).
Arnett, "Workstations-a Software Advantage?", Multimedia Computing & Presentations, v. 4, No. 7, Apr. 30,
1992, p. 1 (3).
Volkman, "The Digital Video Interface for Windows
Multimedia", Windows-DOS Developer's Journal, v.
3, No. 9, Sep. 1992, p. 5 (9).
"Microsoft Ships Toolkit for Multimedia Windows",
PC Magazine, v. 10, No. 19, Nov, 12, 1991, p. 61 (1).

Seltzer, "05/2 PDK a Boom for Developers", PC
Week, v. 9, No. 39, Sep. 28, 1992, p. 1 (2).
Thomsen, "A Software Architecture for VXI Bus Systems", Autotestcon '89 Conference Record, 1989, pp.
327-334.
Southerton, Programmer's Guide to Presentation Man-

ager, 1989, Addison-Wesley, pp. 32-36, 75-78, 181,
205-211, 312-313.
Hamilton et al., "The ATTC Laboratory Automation
System", National Association of Broadcasters, Apr.
1992.
Pyxis Corp. (Fairfax, Va.) News Release, "New VI
Libraries Expand LabView TM Capabilities for Macintosh", Nov. 6, 1992.
Pyxis Corp. (Fairfax, Va.), "Multimedia VIs Application Note", 1992.
Letter from Scott Hamilton, President, Pyxis Corp.
(Fairfax, Va.), Sep. 12, 1994.
IBM TDB vol. 34, No. 11, Apr. 1992, p. 29.

APLNDC00028556

U.S. Patent

June 27, 1995

12

MICRO
PROCESSOR

10 A

16

Sheet 1 of 7

MATH
COPROCESSOR

5,428,730

,

20

BC

22

14D
45

37

BUFFER

102
98

OP A

6

SCSI HD

DATA

MAIN
MEMORY
APPUCATION
-PROGRAMS
_ MMAP

100

NVRAM
38
J
40

ROM
POST

32

ROM
2
FLOPPY

73

PRINTER

BIOS

74

KEYBOARD

76

46
42

G

lOC

- MMPM
- MMDCP

SPEAKER
SERIAL

44

48
VSP

60

7Û

cr

AUX.
VIDEO
110

AIC

112

MONITOR

CABLE
8g

49

90
n?P

\

108
VIDEO DISC
PLAYER

68

i

82 79
MOUSE

MONITOR

02

TEXT AND
GRAPHICS
VRAM

78
80

114
DAC

MUX

88

a

24

I

39
3

ADAPTER

41

AUDIO

\

INST.
RAM
84Á

----

DATA
RAM
86
-

50

.

/
92

94

ADAPTER
EC

HEADPHONE
MICROPHONE
LINE IN
LINE OUT
SPEAKER
MIDI

CARD

96
MMD

APLNDC00028557

U.S. Patent

June 27, 1995

Sheet 2 of 7

5,428,730

FIG. 2
200

204 )

213

21L - Audio

202 )

212

Sample VIDPLAY

22o)

206

21s il
2

231

Mute

Freeze

22 N

226

--232

Volume

227

222-

-233

Balance
223-

3°

File Coritrols S_ource Image Help

'j'

.

Video -

-

Contrast

21e

-

2 6

234

Player Control
Treble

24

Color
44

LJ
Frarrie

242 243

224-

i i i i
Bass

236

Tint

249"

245

i

246

\247 \248

Sh ness

APLNDC00028558

U.S. Patent

June 27, 1995

126

Sheet 3 of 7

USER INTERFACE (Uly

102

106

MULTIMEDIA
APPLICATION
PROGRAM
(MMAP)

AUDIO CONTROL

103

OS/2
MULTIMEDIA
PRESENTATION
MANAGER
(MMPM)

VfDEO CONTROL

105
107
10

OS/2
PRESENTATION
MANAGER (PM)

FIG.3

/ MULTIMEDIA DEVICE CONTROL
PROGRAM (MMDCP)
101

104

5,428,730

PLAYER CONTROL

€ONTROL iNSTANCE(S)
- PANEL TEMPLATES

111%
113¾

INTERFACE LOGIC
-

DEVICE LOGIC

OS/2 KERNEL
120

OS/2
DEVICE
DRIVERS
122
DEVICE
COMMUNICATIONS

SERIAL, PARALLEL,
SYSTEM BUS
42

BIOS
124

m

DEVICES

108
VIDEODISC
VIDEOTAPE DIGITAL AUD1O

CD

MMD 96

DIGITAL VIDEO
ETC.

APLNDC00028559

U.S. Patent

June 27, 1995

Sheet 4 of 7

5,428,730

FIG. 4 ( PRIOR ART)
126

128

USER INTERFACE (UI)
*
130
Ý
132
A
OTHER
k UI SELECTIONS
k CONTROL PANEL DISPLAY
APPLICATION
(BUTTONS, SL1DERS,
& UPDATE, CHANGE IN
INTERACTIONS
DIALS, ETC.)
STATUS, ENABLE/DISABLE, ETC.

MMAP
13

CREATE/DESTROY CONTROL DISPLAY WITH SELECTION COMPONENTS

136

UI DETECTION
Ul EVALUATION- WHAT WAS SELECTED?
SELECTION EVALUATION- VALID/INVALID
DEVICE COMMANDS NECESSARY
COMMAND RESPONSE EVALUATION
UI UPDATE TO REFLECT NEW CONDITIONS
ERROR PROCESSING
TIMED EVENT NOTIFICATION PROCESSING

138

140

CONTINUOUS STATUS NOTIFICATIONS
COMMAND RESULTS
TIMED EVENT NOTIFICATIONS

STATUS REQUESTS
104

Y
DEVICE OPENICLOSE
DEVICE COMMANDS
COMMAND TERMINATES

DEVICE ERROR NOTIFICATIONS

w

MMPM
120

DEVlCE DRIVER
'

DEVICE COMMUNICATIONS
'S

DEVICE

APLNDC00028560

U.S. Patent

June 27, 1995

Sheet 5 of 7

5,428,730

FIG. 5
126

128

102

USER INTERFACE (Ul)
A
130
Ý
132
OTHER
k Ul SELECTIONS
i CONTROL PANEL DISPLAY
APPLICATION
(BUTTONS, SLIDERS,
& UPDATE, CHANGE IN
INTERACTIONS
DIALS, ETC.)
STATUS, ENABLE/DISABLE, ETC.
106
A

MMAP

,l
150
CREATE

102/

CREATE WINDOW
OPEN DEVICE

CREATE CONTROL

107

156' BUILD CONTROL DISPLAY
y
O NIBol

DESTROY 158
UI DETECTION
om amur
<152
- U1 EVALUATION- WHAT WAS SE wm 2
DEVICE ID >
SELECTION EVALUATION- VALID/lNVALID
DEVICE COMMANDS NECESSARY
MISC.

/DENTIFY DEVICE

MSGS·
>
(O ) 160s
154

1623 w
DEVICE OPEN/CLOSE

104

MMDCP

COMMAND RESPONSE EVALUATION
UI UPDATE TO REFLECT NEW CONDITIONS
ERROR PROCESSING
TIMED EVENT PROCESSING

164
CONTINUOUS STATUS NOTIFICATIONS
i COMMAND RESULTS

TIMED EVENT NOTIFICATIONS
DEVICE ERROR NOTIFICATIONS
DEVICE COMMANDS
COMMAND TERMINATES
STATUS REQUESTS

o

MMPM
120

DEVICE DRIVER
122

DEVICE COMMUNICATIONS
124

DEVICE

APLNDC00028561

U.S. Patent

June 27, 1995

Sheet 6 of 7

5,428,730

FIG. 6
126'

USER INTERFACE (Ul)
130-1

132-1

-

130-2

107P

102

107A

I DU- I

PLAYER
CONTROL
INSTANCE

7

U-2'

<-

AUDIO
CONTROL
INSTANCE

MMAP

*
152-1

164-1

- 132-2

152-2

162-1

162-2

164-2

104

MMPM
120-1

120-2

LAYER DEVICE DRIVER

AUDIO DEVlCE DRIVER

122-1

122-2

DEVICE COMMUNICATIONS

DEVlCE COMMUNICATIONS

124-1

SERIAL PORT

PLAYER DEVICE

124-2

SYSTEM BUS

AUDIO DEVICE

APLNDC00028562

U.S. Patent

170

June 27, 1995

172

APPLICATION
INPUT
CREATE

FIG. 7

USER
INTERFACE
INPUT

DESTROY
DEVICE ID
MISC.

Sheet 7 of 7

SELECTIONS

5,428,730

c 17

OS INPUT
MOVE
SHOW
HIDE
PAINT
ETC.

176

DEVICE
INPUT
(VIA MMPM)
ERRORS
TIMED EVENTS

106
MMDCP CONTROL M

^GE PROCESSING

178

CREATE MESSAGES FROM APPLICATION
ACQUIRE RESOURCES (MEMORY, ETC.)
MAKE CONTROL PANEL
PLACE IN WINDOW

180

DESTROY MESSAGES FROM APPLICATION
RELEASE RESOURCES (MEMORY, ETC.)
DISCARD WINDOWS, CONTROL PANEL

182

DEVICE ID MESSAGES FROM APPLICATION
CHECK ID VALIDITY
ATTACH CONTROL INSTANCE TO DEVICE
DETERMINE DEVICE CAPABILITIES
QUERY DEVICE STATUS
UPDATE CONTROL PANEL (ENABLE/DISABLE COMPONENTS,
CHANGE CURRENT SETTINGS)

184

MISCELLANEOUS APPLICATION MESSAGES
TOGGLE APPLICATION NOTIFY ONIOFF
TOGGLE DEVICE COMMUNICATION ONIOFF
USE CUSTOM CONTROL PANEL LAYOUT

186

OPERATING SYSTEM MESSAGES
MOVE, PAINT, SHOW, HIDE, ETC. THE WINDOW
UPDATE THE CONTROL PANEL AS NECESSARY

788

DEVICE NOTIFICATION MESSAGES
ERROR? IF SO, A l I vlPT RECOVERY OR DISABLE COMPONENTS
UPDATE CONTROL PANEL AS NECESSARY

190

UI SELECTION MESSAGES
/ CHECK CURRENT CONTROL PANEL/DEVICE COly i s i
DETERMINE ACTIONS NECFSSARY, INCLUDING:
CANCEL CURRENT OPERATION(S)
SEND DEVICE COMMAND(S)
EXAMINE DEVICE COMMAND RESPONSES
IF ERROR, ATTEMPT RECOVERY
UPDATE CONTROL PANEL AS NECESSARY

APLNDC00028563

1

5,428,730

MULTIMEDIA SYSTEM HAVING SOFTWARE
ME· ••-=SMPROVIDING STANDARDIZED
INTERFACES AND CONTROLS FOR THE
OPERATION OF MUL • = - = ·IA DEVICES

2

gram having a plurality of application program select-

able controls for a like plurality of different classes of
multimedia devices. Each class has a different set of
controllable device functions or attributes. In response
5 to selection of a control by an application program, a
unique control panel is created in a user interface for
BACKGROUND OF THE Iix veis1ION
controlling operation of a device. The selected control
performs all of the processing actions necessary to conThis invention relates to the field of data processing,
trol the device in response to user selections on the
and, more particularly, to an improved multimedia system having a software mechanism that provides stan- 10 interface, without involvement of the application pro-

dardized interfaces and controls for the operation of
multimedia devices.

gram in such actions.
In accordance with another aspect of the invention,

A multimedia system is designed to present various
multimedia materials in various combinations of text,

the multimedia application program includes a plurality
of predetermined templates defining different control
graphics, video, image, animation, sound, etc. Such a 15 panels which contain user manipulable controllers. The
system is a combination of hardware and software. The
application program sends a message to the control
hardware may include a personal computer to which
program indicating its selection and the control provarious multimedia devices can be attached. The hardgram creates the desired control screen from the seware runs under the control of an operating system and
lected template.
multimedia application programs. Recent commercially 20
DRAWINGS
available multimedia systems provide multimedia application developers the capability to control the multimeOther objects and advantages of the invention will be
dia devices via software. In turn, application developers
apparent from the following description taken in concreate programs that allow end users to control a device
nection with the accompanying drawings wherein:
by manipulating simulated controls, such as buttons, 25
FIG. 1 is a schematic diagram of a multimedia data
dials, and other elements, on a computer display screen.
processing system embodying the invention;
The application programmers have been required to
FIG. 2 is a schematic diagram of an exemplary concreate their own custom designed device control
trol screen on a display;
screens as well as handle all user interactions, device
FIG. 3 is a schematic diagram of the major compocommunications, and screen changes. Such custom 30
nents of the software mechanism included in the system
development has resulted in inconsistent user interfaces,

requires knowledge of the device and device control
methods, and duplicates effort for multiple applications.

shown in FIG. 1;
FIG. 4 is a schematic diagram illustrating communi-

Another object of the invention is to provide a multimedia control program that interacts with a multimedia
application program to control a multimedia device
without requiring the application program developer to
have an in-depth knowledge of the multimedia device
to be controlled.
A further object of the invention is to provide a multimedia control program which provides common functions available for use by multimedia application programs, so as to relieve application programmers of duplicate development effort and to thereby increase multimedia programmer productivity.
Still another object of the invention is to provide a
multimedia control program which creates user interfaces comprising display screens having a common look
and feel, the screens containing graphic controllers that
can be manipulated by users to control multimedia devices.
A still further object of the invention is to provide a
multimedia device control program which creates a
control panel with minimum involvement of an application program and which thereafter controls operation
of the device without intervention of the application
program.
Briefly, in accordance with the invention, a multimedia system includes a multimedia device control pro-

FIG. 7 is a block diagram illustrating different inputs
into the MMDCP, and the actions responsive to the
45 inputs.
DET A e av DESCRIPTION
Referring now to the drawings, and first to FIG. 1,

cations between the major components of a prior art
These problems reduce useability and increase development expense.
35 system which the present invention improves;
FIG. 5 is a diagram similar to FIG. 4 but showing the
SUMMARY OF THE I- - · ION
improved processing in accordance with the invention
for multimedia device control and user interface interOne of the objects of the invention is to solve the
action;
problems noted above by providing a multimedia sysFIG. 6 is a simplified diagram similar to FIG. 5 but
tem with a set of standardized multimedia controls by 40
showing processing for controlling a plurality of differwhich a consistent user interface is presented from difent multimedia devices; and
ferent application programs.

there is shown an exemplary multimedia data process50 ing system comprising a personal computer 10 operable
under a multitasking operating system such as OS/2
Version 2.0, to execute application programs. Computer
10 comprises a microprocessor 12 connected to a local
bus 14 which, in turn, is connected to a bus interface
55 controller (BIC) 16, a math coprocessor 18, and a small
computer system interface (SCSI) adapter 20. Microprocessor 12 is preferably one of the family of 80xxx
microp. _
-, such as an 80386 microprocessor, and
local bus 14 includes conventional data, address, and
60 control lines conforming to the architecture of such
processor. Adapter 20 is also connected to a SCSI bus
22 which is connected to a SCSI hard drive (HD) 24
designated as the C:drive, the bus also being connectable to other SCSI devices (not shown). Adapter 20 is
65 also connected to a NVRAM 30 and to a read only
memory (ROM) 32.
BIC 16 performs two primary functions, one being
that of a memory controller for accessing a main mem-

APLNDC00028564

3

5,428,730

ory 36 and a ROM 38. Main memory 16 is a dynamic
random access memory (RAM) that comprises stores

data and programs for execution by microprocessor 12
and math coprocessor 18. ROM 38 stores a POST program 40 and a BIOS 42. POST program 40 performs a 5
standard power-on, self-test of the system when computer 10 is started by turning the power on or by a
keyboard reset. An address and control bus 37 connects
BIC 16 with memory 36 and ROM 38. A data bus 39
connects memory 36 and ROM 38 with a data buffer 41 10
that is further connected to data bus 14D of bus 14.
Control lines 45 interconnect BIC 16 and data buffer 41.
The other primary function of BIC 16 is to interface
between bus 14 and an I/O bus 44 designed in conformance with Micro Channel (MC) architecture. Bus 44 is 15
further connected to an input/output controller (IOC)
46, a video signal processor (VSP) 48, a digital signal
processor (DSP) 49, and a plurality of expansion connectors (EC) or slots 50. VSP 48 is further connected to
a video RAM (VRAM) 60 and a multiplexor (MUX) 62. 20
VRAM 60 stores text and graphic information for controlling what appears on the screen of a monitor 68.
MUX 62 is further connected to a digital to analog
converter (DAC) 66 and to a connector or terminal 70
that is connectable to a video feature bus (VFB). DAC 25
66 is connected to monitor 68 that provides a conventional output screen or display for viewing by a user.
IOC46 controls operation of plurality of I/O devices
including a floppy disc drive 72 designated as the A:drive, a printer 74, and a keyboard 76. Drive 72 com- 30
prises a controller (not shown) and a removable floppy
disc or diskette 73. IOC 46 also is connected to a mouse
connector 78, a serial port connector 80, and a speaker
connector 82, which allow various optional devices to
be connected into the system. A mouse 79 is connected 35
to connector 78.
DSP 49 is further connected to an instruction RAM
84, a data RAM 96, an analog interface controller (AIC)
88, and an audio controller (90). RAMS 84 and 86 respectively hold instructions and data used by DSP 49 40
for processing signals. Audio controller 90 controls
various audio inputs and outputs and is connected to a
plurality of connectors 92 by which various devices can
be connected to the system. Such devices include a
headphone, a microphone, a speaker, a musical instru- 45
ment digitizing interface (MIDI), and devices requiring
an audio line-in and line-out functions. Various other
multimedia devices (MMD) 96 can be also attached to

4

invention is embodied primarily in MMDCP 106 and its
interaction with operating system 100 and the hardware. The combination and interaction of hardware and
software is referred to herein as a "software mechanism".

FIG. 2 illustrates an exemplary control screen 200 in
which a plurality of windows or panels are displayed on
a monitor in response to execution of a sample program
named "VIDPLAY". This program creates a display of
a control screen 200 that includes a video window 202,
an audio control panel 204, a video control panel 206,
and a player control panel 208. Also presented in the
screen is a standard cursor 210 that is manipulated
through use of mouse 79. Window 202 contains a title
bar 212 bearing the title "Sample VIDPLAY", a window control button 213, a sizing button 215, a menu bar
214, and a video viewing area 216. The window itself is
conventional and can be manipulated by means of cursor 210 to change the position and size of the window.
Audio control panel 204 contains a title bar 218 identifying the panel as an "Audio" control panel, and a
control area containing various simulated controllers or
controls 220-224. Control 220 is a mute button, änd
controls 221-224 are dials for respectively controlling
volume, balance, treble, and bass. With reference to
control 221, each dial includes reference marks 228
around the periphery of a circular knob, a movable
pointer 229, a decrease button 226, and an increase
button 227. Buttons 226 and 227 respectively display
graphic symbols "-" and "+" which generally denote
the button function. Also included in control 221 is a
legend "Volume" signifying the function of the device
being controlled. When the cursor 210 is placed on one
of buttons and the mouse is clicked, the pointer appears
to move until the mouse button is released. Such movement would be --panied by signals being sent to the
physical device whose controls are being simulated to
cause the corresponding action to occur. For example,
if the cursor is clicked on button 227, pointer 229 rotates
clockwise and thé actual volume of the audio device
would increase.
Video control panel 206 has a title bar 230 displaying
the legend "Video", a freeze button 231, and dial controls 232-236 for controlling the brightness, contrast,
color, tint, and sharpness, of the video device, e.g.,
monitor 112 (FIG. 1). Player control panel 208 displays
the legend "Player Control" in title bar 240, push buttons 241-248, and a slider control 249. Buttons 241-248

the system through an EC 50 and adapter card 94.
include different symbols symbolic of different player
For purposes of illustrating the invention, system 10 50 functions controlled thereby. Such buttons provide, for
further includes a video disc player 108 having a control
video disc player 108, controls of eject, pause, play,
line connected to connector 80 of the serial port for
play backward, record, repeat, rewind, scan forward,

receiving start/stop signals through the serial port.
scan backward, step backward (frame backward), step
Player 108 is also connected to and transmits video
forward (frame advance), and stop. Slider control 249
signals to an auxiliary video card 110 that has a connec- 55 displays the current device position in time or frame
tion to a monitor 112 and another connection 108 to
based units.
VFB connector 70.
Preferably, each button presents a visual differentiaMemory 36 stores various programs for execution in
tion between actuated and non-actuated conditions.
the system, which programs include application proThis can be accomplished by shadowing the buttons to
grams 100, including multimedia application programs 60 give appearances of being pushed in or sticking out
(MMAP) 102, and an operating system 98 which infrom the plane of the display. Some of the buttons may
cludes OS/2 multimedia presentation manager
also be animated. For example, scan forward and scan
(MMPM) 104 and a multimedia device control program
backward buttons may have a series of arrows that are
(MMDCP) 106. It is to be noted that while FIG. 1
successively highlighted to simulate the direction of
illustrates an exemplary multimedia system, the operat- 65 movement.
ing system is general purpose and is designed to run or
It is to be noted that the above discussion of FIG. 2
control data processing systems having configurations
provides nomenclature facilitating an understanding of
that are different from the one shown in FIG. 1. The
the invention as described below. The contents of con-

APLNDC00028565

5

5,428, 730

trol screen 200 as shown in FIG. 2 is intended to primarily show what samples of different control panels look
like. Quite obviously, in any given system, the type and
number of panels is dependent upon what multimedia
devices are connected to the system, and what controls
are selected to be shown. Further variations in such
control panels and how they are created are described
hereinafter.
FIG. 3 illustrates the layered structure of the software mechanism and the primary programs that are
executed concurrently in a multitasking mode, in accordance with the invention. At the top, a user interface
126 communicates with MMAP 102 and MMDCP 106,
both of which are above MMPM 104. The latter program interacts with the IBM OS/2 Presentation Manager (PM) program 99 which is above the operating
system kernel 98, which sits on top of device drivers
120. PM 99 also communicates with MMAP 102 and
MMDCP 106. A device communications program 122
is between drivers 120 and BIOS 42 which is the lowest
software layer and provides device specific processing
for operating multimedia devices 124. These devices
include video disc player 118, and any MMD 96 such as
video tape, digital audio, compact disc (CD), and digital
video devices.

6

the commands can be sent to the proper device. After
attachment, the control instance performs its primary
function of controlling the particular device, as described in more detail hereinafter.
5
FIG. 4 illustrates various functions 134, 136, and 138
as performed in the prior art by a typical application
program 102' and the interfaces between such program

and user interface (UI) 126' and MMPM 104. The illustration is for a system having a single multimedia device
10 124. The graphics on the screen display together with
the means for manipulating the graphics and making
selections, form the user interface 126' through which
the user interacts with the system to control operation
of the multimedia devices. Input messages 130 from the
15 user into the application program 102' include UI selections made by the user actuating the various buttons,
sliders, dials, etc. Output communications or messages
132 from the application to the user interface include
controlling and updating the panel display, indicating
20 changes in status of the multimedia devices, enabling

and disabling buttons or controls which are or are not
appropriate for the current device state. For example,if
a device is not playing, then the pause button is disabled

because its use is not applicable. Other application inter25 actions 128, such as those not pertinent to the control of

MMAP 102 is a user oriented application program
a MMD, may also occur with the user interface.
providing access to services of multimedia devices apVarious input/output messages 140 and 142 also compropriate to the application, e.g., a video disc browsing
municate between the application and MMPM 104.
program. MMAP 102 interacts with MMDCP 106 for
Application 102' may receive as input messages 142
the purpose of providing a user interface for the appro- 30 continuous notifications on the status of the multimedia
priate device, and with MMPM 104 for opening and
devices, the results of commands, timed event notificaclosing the device and accessing specific features or
tions and device error notifications. The application
performing custom processing not available via the
output messages 140 include device open/close and
more generic MMDCP 106. MMPM 104 implements a
other commands, command terminates, and status rehigh level, device independent interface to multimedia 35 quests.
devices for application use and includes support for
In response to the application program being run or
device context switching, data streaming, and manageexecuted, the program is initially operative to create, by
ment of device contention. PM 99 is used for its screen
function or step 134, a control screen on a monitor

management functions and capability of creating winwhich screen includes one or more controllers. The
dows on a display screen. OS/2 kernel 98 represents the 40 size, appearance, location, contents, etc. of the control
services provided by the IBM OS/2 operating system.
screen are the full responsibility of the application.
These services generally include functions for process
Once the controllers are displayed, the user can thereafand .. ... management. Kernel 98 and PM 99 are
ter manipulate the controls and make selections to conomitted from later drawing figures for simplicity of
trol operation of a MMD. When the application is being
illustration. OS/2 device drivers 120 are device specific 45 ended, a destroy message 134 destroys the control
control programs providing a software interface for
screen initially created. Functions 136 are performed in
controlling individual devices A device driver at this
response to the application program receivmg messages
level generally provides services for a device of a spe130 and include detecting UI actions, evaluating the
cific type and manufacturer. Device communications
selection as to what action was selected, determining if
122 indicates a communication mechanism by which a 50 the selection is valid or invalid, and generating various
device driver __ - an attached device. Some deoutput messages 140. In response to the application

vices are card type adapters installed in the system for

program receivmg messages 142, application 102' evalu-

access by the system bus. External devices are generally

ates responses to commands, updates the UI to reflect

attached to the computer by a cable plugged into the
new conditions, and processes the notifications of timed
parallel port or the serial port.
55 events and device errors. The application also sends
MMDCP 106 comprises an audio control 101, a video
messages 132 as appropriate in response to such processcontrol 103, a player control 105, one or more control
ing. In summary, the messages and interactions exinstances 107, panel templates 109, interface logic 111,
plained with reference to FIG. 4 indicate what an appliand device logic 113. Controls 101, 103, and 105 are
cation program would have to do in the absence of the
separate generic routines or programs for controlling 60 invention. It is to be appreciated that application promultimedia devices having audio, video, and player
grams normally perform functions in addition to those
attributes. In general, an application program first opens
related to MMD control but inasmuch as the invention
a device, and then sends a message to MMDCP to creis concerned with MMD control, the additional funcate a control screen which is done by making an intions have been omitted for simplicity.
stance or copy of the appropriate control and then at- 65
With reference to FIG. 5, in accordance with the
taching the control instance to the device. That is, once
invention, MMAP 102 is greatly simplified and reduced
the instance has been made, it is necessary to inform it of
relative to the equivalent prior art application program
the identity of the device it will be controlling so that
102' because MMDCP 106 performs most of the func-

APLNDC00028566

i

7

5,428,730

8

tions for generating and receiving communications with
display player control panel208 and audio control panel
the user interface and with MMPM 104. When MMAP
204. Once the control instances are attached, each con102 is run, a series of instructions 102I are executed. The
trol instance manages its attached device and configures
first instruction is to create a window and this is done by
itself according to the device capabilities. During operPM 99. The second instruction is to open the device to 5 ation, player control instance 107P receives communibe controlled and this is done by sending by sending a
cations 130-1 for the UI and sends messages 132-1 to the
DEVICE OPEN command 162. The third instruction is
UI, while audio control instance 107A receives mes-

to create a control and this is done by sending a CREsages 130-2 from the UI and sends messages 132-2 to the
ATE message 150. The fourth instruction IDENTIFY
UI. In the prior art, the application program has to
DEVICE sends a DEVICE ID message 152 identifying 10 implement and manage the additional functionality to
the particular device to be controlled to MMDCP. In
address and handle the functional differences between
response to messages 150 and 152, MMDCP 106 creates
player and audio devices.
a control instance 107 and attaches it to open device
FIG. 7 shows further details of control message pro124. After that, control processing including generating
cessing that is done in response to ......Ing messages or
and receiving communications from the UI and the 15 inputs 170, 172, 174, and 176 from four sources, namely,
MMPM, is done by control instance 107 thereby reliev¯
MMAP 102, UI 126, the operating system, and multimeing the application from doing similar functions and
dia device 124 inputs through MMPM 104. In response
being concerned with knowledge of individual devices.
to receiving a DRE A TE control screen command,
Control instance 107 receives input messages 130 from
MMDCP is operative to perform the functions 178 and
UI 126, transmits messages 132 to the UI, and communi- 20 acquire the resources (e.g., memory) for the display,
cates with MMPM 104 via two way communications
make the desired control panels, and place them on the
164. Control instance 107, in response to receiving variscreen. In response to receiving a DESTROY message
ous inputs, performs the functions described below with
from the application, actions 180 release the resources,
reference to FIG. 7. The control screen presents a conand discard the window and any control panels on the
trol interface to the end user, and control instance 107 25
translates actions and selections on the interface into
screen. In response to receipt of a DEVICE ID message
appropriate media control and system commands.
from the application, actions 182 comprise checking the
MMDCP 106 can create three different types of convalidity of the device ID, making a control instance by
trol panels: a video control panel for a device, such as a
copying the control and then attaching the control
graphics/video overlay adapter, having video attri- 30 instance to the open device, determining device capabilbutes; an audio control panel for a device, such as an
ities, querying the device status, and updating the conamp/mixer device, having audio attributes; and a player
trol panels, e.g., by enabling or disabling components
control panel for a device, such as a videodisc player,
and changing current settings. The attaching is simply
having player attributes. A video control panel typinotifying the control instance of the ID of device to be
cally includes a freeze push button, and circular sliders 35 controlled. Actions 184 are functions or actions peror dials to control brightness, contrast, saturation, hue,
formed in response to miscellaneous messages from the
and sharpness. An audio control panel typically inapplication and include enabling or disabling the sendcludes a mute push button and a series of circular sliders
ing of notification messages to the application, toggling
to control volume, balance, treble, and bass. A player
application notify on/off, toggling device communicacontrol panel typically includes push buttons for combi- 40 tion on/off, and using custom control panel layout.
nations of the following player actions: eject, pause,
Action 186 updates control panels as necessary in
play backward,play forward, play reverse, record, re.
response to receiving messages from the operatmg syspeat, rewind, scan backward, scan forward, scan retem. Such messages include hide, show, move, paint,
verse, step backward, step forward, and stop. A player
etc. In response to receiving device notification mescontrol panel might also include a linear slider for posi- 45 sages, actions 188 include updating the control panel as
tion display. It should be obvious that the individual
necessary. If an error has occurred, a recovery attempt
controls depend on the capabilities of the attached mulis made or the component is disabled. Timed events are

timedia device(s) and the configuration of the player
control ...

events which occur at a device asynchronously with
respect to a command issued to the device. For examFIG. 6 illustrates the manner in which MMDCP 106 50 ple, if the device is instructed to play for a certain
amount of time, the completion of the play command is
controls two multimedia operations for a MMO that
a timed event which results in a notification being sent
runs two multimedia devices. The devices might be ,
to MMDCP 106. At this point, MMDCP 106 stops
e.g., a player device 124-1 such as videodisc player 108
animating the play button and changes the appearance
connected to the serial port, and an audio device 124-2
connected to one of the expansion connectors 50 and 55 of the play button to be consistent with a "stop" state.
Other timed events might include position notifications
accessible over the system bus. For such a configuraand cuepoints. When an input 172 is received, actions
tion, MMAP 102 needs to send a first DEVICE OPEN
190 are performed to check the -.-..t control pamessage 162-1, a first DEVICE ID message 152-1, and
nel/device context, and determine what actions are
a first
^TE message 150-1 to create a player control instance 107P, and to then send a second DEVICE 60 necessary. If for example, a device is already playing
and the play button is selected, the context indicates no
OPEN message 162-2, a second DEVICE ID message
152-2, and a second CREATE message 150-2 to create
action is necessary and the selection of the play button

is ignored. The actions to be done include cancelling
an audio control instance 107A. Also, respective player
current operations, sending device commands, examinand audio device drivers 120-1 and 120-2 and respective
device communications 122-1 and 122-2 have to have 65 ing device command responses, attempting error recovbeen loaded into the system, for establishing communiery, and updating the control panel as necessary. The
cations with the respective multimedia devices. With
specific details of the actions described relative to FIG.
two devices, the control screen 200 in FIG. 2 could
7 should be obvious to those skilled in the art and have

APLNDC00028567

9

5,428,730

been omitted to shorten description particularly since
the details are not claimed.
A typical sequence of actions that occur can be illus-

trated by the example of audio control panel 204 and
volume controller 221 (FIG. 2). Suppose the user de- 5

10

The following application code is written for OS/2
operating system with OS/2 multimedia extensions, and
provides an example of the code simplicity of an application program useable with the invention:

sires a volume increase. To accomplish this using the
******** BEGIN EXAMPLE CODE ********
mouse, the mouse is manipulated to position cursor 210
//Open a videodisc device with an alias name and obtain a
on button 227 and then clicked and held until the vol//device ID
ume increases to the desired level. In response to the
OpenDevice ((LPSTR)szVdiscDevName, (LPSTR)szVdiscAlias,
mouse being clicked, the following sequence of actions 10 &wVdiscDevlD);
//Create a window to hold the Player Control Panel on the
occur. First, presentation manager 99 (FIG. 3) deter//screen
mines which application, the cursor is currently being
hwndVPCDIg = WinLoadDIg(HWND_DESKTOP,
operated under, which window the cursor is being ophwndClient,

erated in, and which object in the window has been

pVPCDigProc,

0,
selected. In this example, the object is the audio control- 15
ID_DLG_VPC,
ler volume increase button. PM 99 then sends a user
NULL);
interface message 172 (FIG. 7) to the audio control
//Create the Player Control Panel using MINIMAL
instance which message identifies the object and the
//configuration. "hwndVPC" is the handle uses for any further

control action desired. In response to receiving the

//communication with the player control

functions 190 and determines what actions are necessary. The first is to decide whether to cancel any cur-

WC-PLAYERCTL,

rent operation. If the mute button is on, it would be

WS VISIBLE ] MCS-MINIMAL //Select panel style

audio device to increase the volume by an increment.

HWND-TOP
ID_VPC_WND,

message, the audio control instance then performs the 20 hwndVPC = WincreateWindow
(hwndVPCDig,

//Create Player Control

//Misc OS/2 item

X_MMCTL, Y MMCTL
cancelled to allow audio to be produced by the device.
200,100
Next, a volume increase device command is sent to the 25 hwndVPCDig,

At this point the device responds and if the response
indicates such action has been done without error, the
next action is to update the pointer of the volume dial

and this is done by sending a message through PM 99
and user interface which thereupon redraws the volume
dial pointer to a new position representing an increment
of volume increase. The sequence of actions repeat until
the mouse button is released, or until the maximum
volume is reached. An optional action (not shown)
could be performed before communicating with the
device, to give the application program the opportunity
to handle the user selection. This is useful in situations
where operational customization is done by the application, instead of operating by the control instance. With
this option, the application would send a return code
indicating it will or will not handle the message.
As indicated above, MMDCP 106 includes a variety
of panel templates 109 that are used in making a control
panel in 178. MMDCP provides the application programmer with the options of accepting a default panel,
a minimal panel, a complete panel, or a custom panel. A
complete panel includes all the supported dials, buttons,
sliders, etc. for controlling a particular device. A minimal panel includes only those controllers generally
considered absolutely necessary. Naturally, the minimal
panel occupies less screen area than a complete panel. A
custom panel could be defined by the application and
use any subset of the components of a complete panel,
change relative positions, include other functions, etc.
The templates also include vertically oriented panels,
such as the audio and video panels shown in FIG. 2, and
horizontally oriented panels, for both complete and
minimal panels. Additional options include animated
buttons (such as movement of arrows on a Play button
when a device is playing), and the ability for an application to request that the control only present a user inter-

//Use the window above

30

//Starting screen location
//Size
//Misc OS/2 item

//Misc OS/2 item
//Misc OS/2 item

NULL,
//Misc OS/2 item
NULL);
//Misc OS/2 item
//Inform the Player Control to use the videodisc player that
//was opened earlier

WinSendMsg (hwndVPC, M -

-- -

EVICEALIAS,

szVdiscAlias,

0)

********END SAMPLE CODE ********

35

In response to successful execution of the above code,

a device ID is obtained, a window is created, a player
control instance is created, and the device is set. The
player control instance will display the control screen
40 including the player control panel, handle all user interface actions, communicate with the videodisc player,
etc. No other code in the application program is needed
to perform or process selections from the control panel.
As indicated previously, an application program must
45 first create an instance of a control and second, attach

50

55

60

face but not communicate with the device. This option

may be useful if the application desires to control other
devices while taking advantage of the user interface 65
standards and processing. The designs of the panels
provide a uniform appearance and common look and
feel between different applications.

the instance to an open device before the control instance can assume responsibility for managing the user
interface with the attached device. The first action
involves creating a window of the proper window class.
There is a window class for each type of supported
device to be controlled: audio attributes, video attributes, and player attributes. There are also separate
control programs (which may share a function) for each
window class. In presentation manager (PM), an application program creates a window using the PM call
WinCreateWindow. WinCreateWindow has several
parameters including the window class and window
style.
The window style of the MMDCP specifies, among
other things, the orientation of the control panel (vertical or horizontal) and the degree of desired device control (minimal, complete). The control program uses the
style to select one of four layouts for the initial appearance. For example, audio control 204 in FIG. 2 is a
vertical, complete control. The result thus promotes a
consistent user interface.
The panel templates 109 each specify the type of
components and their positions using a dialog template

APLNDC00028568

11

5,428,730

of PM. By way of example, the dialog template corresponding to the layout of audio control panel 204 is:

12
multimedia device (hereinafter referred to as open
MMD) and for thereafter transmitting signals to

DGLTEMPLATE ID-DLG_AACCOMPLETE_V LOADONCALL MOVEABLE

DI- --· --LE
BEGIN

DIALOG "", ID_DLG_AACCOMPLETE_V, 0, 0, 42, 174, NOT
FS DLGBORDER | FS_NOBYTEALIGN
PRESPARAMS PP_FONTNAMESIZE, "8.Helv"
BEGIN
PUSHBUTTON " ",
ID-MUTE, I, 153, 40, 20, WS-TABSTOP

"Mute", ID_TEXT-MUTE 1,145,40,8
PRESPARAMS PP-FOREGROUNDCOLORINDEX, SYSCLR_WINDOWTEXT
--· -- = - ·· ·N "Volume" ID_VOLUME 1,IO9,40,36,WS-TABSTOP
PUSHBUTTON "Balance", ID_BALANCE 1,73,40,36, WS_TABSTOP
PUSHBUTTON "Treble", ID_TREBLE, 1,37,40,36, WS-TABSTOP
PUSHBUTTON "Bass", ID_BASS 1,1,40,36, WS TABSTOP
END

END

For the purpose of defining the layout of a control
panel, the window class specified in the dialog template 20

is ignored for each of the predefined components such
as the mute button or the volume dial. To create the
control panel display, the MMDCP uses the information in the template to create each of the desired components. The subset of components present influences the 25
subsequent operation of the MMDCP. For example, a
minimal audio control includes only a volume dial and
a mute button. A request to update the control panel
would query the device only for the applicable information, i.e., the volume and mute status. A complete audio 30

control would cause a query of the device for all of the
components, i.e., the mute, volume, bass, and treble
status,
The template mechanism also allows the MMAP to
specify an alternative layout for a control panel. This is 35
done through an optional miscellaneous application

message 184 to USE CUSTOM CONTROL PANEL
LAYOUT. In this case, the MMAP provides the desired template to the MMDCP after the panel is created.
The custom template supersedes any layout selected at 40
MMDCP creation time. If neither minimal nor complete is specified in the style, then the MMDCP assumes
that the MMAP will subsequently provide a layout and
no components are created until this is done.
While the invention has been described in connection 45
with a specific operating system, it should be obvious
that other operating systems could also be used. Further, it should also be obvious that additional classes of
control programs can be added or substituted for the

audio, video and player control programs. It should be 50
further apparent to those skilled in the art that many
changes can be made in the details and arrangements of
steps and parts without departing from the scope of the
invention as defined in the appended claims.
What is claimed is:
55
1. In a multimedia data processing system (MMDPS)
comprismg a processor, a storage system including a
memory for storing at least one multimedia application
program (MMAP) and a multitasking operating system
for execution by said processor, a monitor for display- 60
ing screens of information, and connector means for

said open MMD in response to said first means
receiving device commands, said signals being
operative to control said controllable functions of
said open MMD;
second means comprising a user interface including
user manipulable objects on a display screen of said
monitor;
and multimedia device control means (MMDCM)
layered between said first means and said second
means, said MMDCM comprising a plurality of
application-program-selectable class-specific controls, each control being for a different one of said
classes of multimedia devices;
said MunCM being responsive to a CREATE message from the application program to select one of
said controls (hereinafter referred to as selected
control) having the same class as said open MMD;
said selected control comprising third means for creating a control screen on said monitor in said user
interface, said control screen having a control
panel comprising a plurality of user actuated controllers for controlling different functions of said
open MMD;
said user interface being operative to send a user
selection message in response to a user manipulating one of said controllers;
and, said MMDCM further comprising processing
means operative, in response to receiving said user
selection message, to send a device command to
said first means for changing a function of said
open MMD and to send a message to said user
interface to update the one of said controllers manipulated by the user to reflect the changed function of said open MMD.
2. A MMDPS in accordance with claim 1 wherein:
said selected control is a copy of one of said controls
(which copy is referred to hereinafter as a control
instance);
and said MMDCM is operative to attach said control
instance to said open MMD in response to receiv-

ing an identification of said MMD.
3. A MMDPS in accordance with claim 2 wherein
said control instance is made in response to said CRE-

connecting one or more multimedia devices into said

ATE message and said identification is done by a DEMMDPS, each device having a plurality of controllable
VICE ID message from the application program.
functions, each device being in one class of a plurality of
4. A MMDPS in accordance with claim 3 wherein:
different classes of multimedia devices, the improve- 65
said MMDCM further comprises a plurality of predefined control panels for each class of said controls,
ment comprising:

first means responsive to a DEVICE OPEN command from said application program for opening a

said MMDCM being further operative in response
to said on = ^ TE message to select one of said

APLNDC00028569

13

5,428,730

predefined control panels for display in said control screen;
and said processing means is further operative in
response to receiving said DEVICE ID message to
determine device capabilities and device status, and 5
send messages to said user interface for updating

said control panel in accordance with said device
capabilities and said device status.
5. A MMDPS in accordance with claim 4 wherein
said processing means is operative to selectively enable 10
and disable said controllers in said control screen according to whether or not said MMD has a controllable
function corresponding to the control function of each
controller.
6. A MMDPS in accordance with claim 4 wherein: 15
said MMDCM is further operative in response to
receiving a second CREATE message to create a
second control instance and a second control panel
for a second MMD.
7. A MMDPS in accordance with claim 1 wherein 20
said MMDCM further comprises a plurality of predefined control panels for each class of said controls, said
MMDCM being further operative in response to said
CREATE message to select one of said predefined
control panels for display in said control screen.
25
8. A MMDPS in accordance with claim 7 wherein
said predefined control panels for each class include a
minimal control panel and a complete control panel,
said minimal control panel having a minimum number
of controllers for controlling a minimum number of 30
functions, and said complete control panel having a
plurality of controllers for controlling a maximum number of functions of each class.
9. A MMDPS in accordance with claim 8 wherein:
said predefined control panels further comprise con- 35
trol panels of horizontal and vertical orientations.
10. A MMDPS in accordance with claim 1 wherein:
said first means is operative to send to said control
instance notification messages notifying said control instance of errors occurring in said MMD and 40
timed events occurring in said MMD;
and said processing means is further operative to
process said notification messages.
11. In a multimedia data processing system
(MMDPS) comprising a processor, a storage system 45
including a memory for storing at least one multimedia
application program (MMAP) and a multitasking oper-

14
trols, said templates for each class include a minimal control panel template and a complete control
panel template, said minimal control panel template
defining a minimum number of controllers for con-

trolling a minimum number of functions, and said
complete control panel template defining a plurality of controllers for controlling a maximum number of functions of each class;
said MMDCM being responsive to a CREATE message from the application program to select one of
said templates and send a message to said user mterface to create a control panel in said control screen

in accordance with the definition in the template so
selected;

said control panel in said control screen comprising a
plurality of user actuated controllers for controlling different functions of said open MMD;
said user interface being operative to send a user

selection message to said MMDCM in response to
a user manipulating one of said controllers;
and, said MMDCM further comprising processing
means operative, in response to receiving said user
selection message, to send a device command to
said first means for changing a function of said
open MMD and to send a message to said user
interface to update the one of said controllers manipulated by the user to reflect the changed function of said open MMD.
12. A MMDPS in accordance with claim 11 wherein
said minimal control panel templates and said complete

control panel templates further define control panels of
horizontal and vertical orientations.
13. A MMDPS in accordance with claim 12 wherein
said controllers include buttons, dials, and sliders.
14. The method of operating a multimedia data processing system (MMDPS) comprising a processor, a

storage system including a memory for storing at least
one multimedia application program (MMAP) and a
multitasking operating system for execution by said
processor, a monitor for displaying screens of information, and connector means for connectmg one or more
multimedia devices into said MMDPS, each device
having a plurality of controllable functions, each device
being in one class of a plurality of different classes of
multimedia devices, said method comprising:
(a) opening a multimedia device (hereinafter referred

ating system for execution by said processor, a monitor
for displaying screens of information, and connector
means for connecting one or more multimedia devices 50

to as open MMD);
(b) storing in said storage system a plurality of application-program-selectable class-specific controls, each control being for a different one of said

into said MMDPS, each device having a plurality of
controllable functions, each device being in one class of

classes of multimedia devices;
(c) creating, in response to a CREATE message from

a plurality of different classes of multimedia devices, the
improvement comprising:

the application program, a control instance from
the one of said controls (hereinafter referred to as

first means responsive to a DEVICE OPEN com- 55
mand from said application program for opening a
multimedia device (hereinafter referred to as open
MMD) and for thereafter transmitting signals to
said open MMD in response to said first means
receiving device commands, said signals being 60

selected control) having the same class as said open
MMD and attaching said control instance to said
open MMD;
(d) creating a control screen on said monitor in said
user interface, said control screen including a control panel comprising a plurality of user actuated

operative to control said controllable functions of
said open MMD;
second means comprising a user interface including a
display screen on said monitor;
and multimedia device control means (MMDCM) 65
layered between said first means and said second

controllers for controlling different functions of
said open MMD;
(e) sending a user selection message from said user
interface to said control instance in response to a
user manipulating one of said controllers;
(f) generating, in response to said control instance

means, said MMDCM comprising a plurality of

receiving said user selection message, at least one

control panel templates for each class of said con-

device command for changing a function of said

APLNDC00028570

15

5,428,730

16

.

open MMD, based upon said user selection mes-

accordance with said device capabilities and said

sage and current operational state of said open

device status.
18. The method according to claim 15 comprising:
sending to said control instance notification messages
notifying said control instance of errors occurring
in said MMD and timed events occurring in said

MMD;
(g) transmitting signals to said open MMD in response to said device commands, said signals being 5
operative to change one of said controllable func-

tions of said open MMD;

MMD;

and (h) sending a message to said user interface to

and processing said notification messages.

update the one of said controllers manipulated by
19. The method according to claim 14 further comthe user, to reflect the changed function of said 10 pnsmg:
open MMD.
storing in said storage system a plurality of control
15. The method according to claim 14 further companel templates for each class of said controls, said
pnsmg:
templates for each class include a minimal control
storing in said storage system a plurality of control
panel template and a complete control panel tempanel templates for each class of said controls, said 15
plate, said minimal control panel template defining
templates for each class include a minimal control
a minimum number of controllers for controlling a
panel template and a complete control panel temminimum number of functions, and said complete
plate, said minimal control panel template defining
control panel template defining a plurality of cona minimum number of controllers for controlling a
trollers for controlling a maximum number of funcminimum number of functions, and said complete 20
tions of each class;
control panel template defining a plurality of conincluding in said application program a template detrollers for controlling a ...J..um number of funcfining a custom control panel;
tions of each class;
specifying in said no ^TE message which template
and selecting one of said templates;
is to be used in step (d)
said step (d) being performed by creating said control 25
said step (d) creates a custom control panel in said
panel in said control screen in accordance with the
control screen when said custom control panel
definition in the one template so selected.
template is specified in said ""=^TE message.
16. The method according to claim 15 comprising:
20. The method according to claim 14 comprising:
selectively enabling and disabling said controllers in
said control screen according to whether or not 30 sending a message to said application program in
response to receiving said user selection message in
said MMD has a controllable function correspondstep (e), which message requires a response from
ing to the control function of each controller.
17. The method according to claim 15 comprising:
said application program indicating whether or not
determining device capabilities and device status, and
said application program will process said user
sending a message from said control instance to 35
selection message.
said user interface to update said control panel in
*****

40

45

50

55

60

65

APLNDC00028571

04860.P1365THE UNITED STATES PATENT AND TRADEMARK
tion of
St

FeFntCE

)

W. chutensen

) Examiner:

) Art Unit:

o. 08/316,237

Dela Torre, C.

2415

Filing Date: September 30, 1994

)

For: METHOD AND APPARATUS FOR
DISPLAYING AND ACCESSING
CONTROL AND STATUS
INFORMATION IN A CO- -R

)
)
)
)

evenu

)

Commissioner of
Patents and Trademarks
Washington, D.C. 20231
PETITION FOR EXTENSION OF TIME (37 C.F.R. § 1.136(a)
Sir:
Pursuant to Rule 1.136(a) Applicant hereby petitions for an extension of time
to respond to the outstanding Office Action dated March 20, 1996. Our check in the
amount of $380.00, the extension fee required for a response filed within the second

month (37 C.F.R. § 1.17(b)), is submitted concurrently herewith.
Please charge any debits and credit any overages to our Deposit Account
Number 02-2666.

Respectfully submitted,
BL LA KB I Y, SOKOLOFF, TAYLOR & ZAFMAN

Dated:

½ , 1996
Michaei'J. Mallie
Reg. No. 36,591

12400 Wilshire Blvd.
Seventh Floor
Los Angeles, CA 90025-1026
[phone]
(37 C.F.R. § 1.8( )
I hereb certi y that this correspondence is being deposited with the United States Postal Service as first class
mail w th sufacient postage in an envelope addressed to the Commissioner of Patents and Trademarks,
Washington, D.C. 20231
Date of Deposit

Name o
Signature

erson Mailing Correspondence
Date

APLNDC00028572

Att

's

In

in

ge A

ati

Applic

PATENT

. Steven W. Christensen

8/316.237

Filed:
For:

004R60 P13As

9/30/94
.METHOD AND APPARATUS FOR DISPLAYlNG AND ACCESSING CONTROL AND
STATUS INFORMATION IN A COMPUTER SYSTEM
(title)

THE COMMISSIONER OF PATENTS AND TRADEMARKS
Washington, D.C. 20231

SIR: Transmitted herewith is an Amendment for the above application.
/ Small entity status of this application under 37 C.F.R. §§ 1.9 and 1.27 has been established by averified
statement previously submitted.
/ A verified statement to establish small entity status under 37 C.F.R. §§ 1.9 and 1.27 is endlosed.

I

No additional fee is required.

I
The fee has been calculated as shown below:
(Col. 1)

(Col. 2)

(Col. 3)

Claims remaining
after amendment

Highest no.
previously paid for

Present
extra

Indep.

Claims:

Rate

Additional
fee

minus

20

4*

x $11.00=

x $22.00=

minus

3

0

x $39.00=

x $78.00=

+ $125.00

+ $250.00

Fir t presentation c Multip e Dependent Claim(s)
*If the difference in Col. ? Is less than zero, ente "0"
in Col. 3

A check in the amount of

I

fee

Other than a Small Entity

3

Claims:

XXX

Additional

24

Total

| XXX

Small Entity
Rate

m a nno

Total
Add. Fee:

$88.00
$0.00

Total
Add. Fee:

$88.00

is attached for presentation of additional claim(s).

Applicant(s) hereby Petition(s) for an Extension of Time of
twn 191
month(s) pursuant to
37 C.F.R. § 1.136(a).
A check for * n ^^
is attached for processing fees under 37 C.F.R. § 1.17.

Please charge my Deposit Account No. 09-2666 the amount of
. A duplicate copy
of this sheet is enclosed.
The Commissioner is hereby authorized to charge payment of the following fees assodated with this
communication or credit any overpayment to Deposit Account No. no oman (a duplicate copy of
this sheet is enclosed):
Any additional filing fees required under 37 C.F.R. § 1.16 for presentation of extra claims.
VW
Any extension or petition fees under 37 C.F.R. § 1.17.

BLAKELY, S-..-.., TAYLOR & ZAFMAN

Date:
Michael
allie
Reg. No. 36,591
12400 Wilshire Blvd., 7th Floor
Los Angeles, California 90025

[phone]

I hereby certify that this correspondence is being deposited with the
United States Postal Service as first class mail with sufficient postage in
an envelope addressed to: Commissioner of Patents & Trademarks,
Was
on, D.C. 20231 on:

Angdfa M. Quinn

vare

LJV/ccm (1/31/96)

APLNDC00028573

PATENT COOPERATION TREATY
From the

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY
To:

JAMES C. SCHELLER
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN
12400 WILSHIRE BOULEVARD, 7TH FLOOR

GELES C

-

-1026

WRTITEN OPINION

O

(PCT Rule 66)

OC I 1 1 199ô
BgY, 80KOLOFF, TAYLOR & 2AFMAN
LOS ANGELES

Date of Mailing

(day/month/year)

Applicant's or agent's file reference

04860.P1365

REPLY DUE

/

International application No.

International filing date (day/monthlyear)

PCTIUS95/11025 |

29 AUGUST 1995

aancT W98
" ¯ ~

within TWO months
from the above date of mailing
Priority date (day/monthlyear)

30 --- --BER 1994

International Patent Classification (IPC) or.both national classification and IPC

IPC(6): GO6F 9/44; and US Cl.: 395/161
Applicant
APPLE COMPUTER, INC.

1. This written opinion is the Et

(first, etc.) drawn by this International Preliminary Examining Authority.

2. This opinion contains indications relating to the following items:
I

Basis of the opinion

II

Priority

III

Non-establishment of opinion with regard to novelty, inventive step or industrial applicability

IV

Lack of unity of invention

V

Reasoned statement under Rule 66.2(a)(ii) with regard to novelty, inventive step or industrial applicability;
citations and explanations supporting such statement

VI

Certain documents cited

VII

Certain defects in the international application

VIII

Certain observations on the international application

3. The applicant is hereby invited to reply to this opinion.
When?

See the time limit indicated above.

How?

By submitting a written reply, accompanied, where appropriate, by amendments, according to Rule 66.3.
For the form and the language of the amendments, see Rules 66.8 and 66.9.

Also

For an additional opportunity to submit amendments, see Rule 66.4.
For the examiner's obligation to consider amendments and/or arguments, see Rule 66.4 bis.
For an informal communication with the examiner, see Rule 66.6.
If no reply is filed, the international preliminary examination report will be established on the basis of this opinion.
4. The final date by which the international preliminary
examination report must be established according to Rule 69.2 is: 30 JANUARY 1997

Name and mailing address of the IPEA/US

Authorize officer

Commissioner of Patents and Trademarks

Box PCI'
Washington, D.C. 20231

Facsimile No. [phone]
Form PCT/IPEA/408 (cover sheet) (January 1994)*

WiARK R. POWELL
Telephone No.

[phone]

APLNDC00028574

-.. ...n. OPINION
I.

International application No.

PCT/US95/11025

Basis of the opinion

1. This opinion has been drawn on the basis of (Substitute sheets which have beenfurnished to the receiving Ojice in response to an

invitation under Article 14 are referred to in this opinion as "originaHyfded".):

the international application as originally filed.

the description, pages 1-54
pages NONE

, filed with the demand.

pages NONE

, filed with the letter of

the claims,

Nos.

1-18

, as originally ñled.

, as originally filed.

Nos. NONE

, filed with the demand.

Nos. NONE

the drawings,

, as amended under Article 19.

Nos. NONE

, filed with the letter of

sheets/Ag 1-13

, as originally filed.

shœtsing NONE

, filed with the demand.

sheets/Ag NONE

, filed with the letter of

2. The amendments have resulted in the cancellation of:
the description, pages NONE
the claims,

Nos. NON

the drawings, shots/Rg
3.

NONE

This opinion has been established as if (some of) the amendments had not been made, since they have been considered
to go beyond the disclosure as filed, as indicated in the,SupplemmtabBox Additional observations below (Rule 70.2(c)).

4. Additional observations, if necessary:
NONE

Form PCT/IPEA/408 (Box I) (January 1994)*

APLNDC00028575

International application No.

WRITTEN OPINION

PCT/US95/11025

V. Reasoned statement under Rule 66.2(a)(ii) with regard to novelty, inventive step or industrial applicability;
citations and explanations supporting such statement
1.

STATEMENT
Novelty (N)

2.

YES

1-18

NO

Claims

NONE

YES

Claims

Industrial Applicability (IA)

NONE

Claims

Inventive Step (IS)

Claims

1-18

NO

Claims
Claims

1-18
NONE

YES
NO

CITATIONS AND EXPLANATIONS
(See Supplemental Sheet.)

Form PCT/IPEA/408 (Box V) (January 1994)*

APLNDC00028576

International application No.

WRITTEN OPINION
VI.

PCT/US95/11025

Certain documents cited

1. Certain published documents (Rule 70.10)
Application No.
Patent No.

Filing Date
(dayhnonth/year)

US, A, 5,416,895

16 MAY 1995

8 APRU 1992

NONE

US, A, 5,428,730

2.

Publication Date
(day/month/year)

Priority date (valid claim)
(day/month/year)

27 JUNE 1995

15 DECEMBER 1992

NONE

Non-written disclosures (Rule 70.9)
Kind of non-written disclosuæ

Date of non-written disclosure
(day/monthlyear)

Date of written disclosure
referring to non-written disclosure
(day/month/year)

Form PCT/IPEA/408 (Box VI) (January 1994)*

APLNDC00028577

Intemational application No.

-

"N OPINION

PCT/US95/11025

Supplemental Box
(To be used when the space in any of the preceding boxes is not sufficient)
Continuation of: Boxes I - VIII

Sheet 10

TIME LIMIT:
The time limit set for response to a Written Opinion may not be extended. 37 CFR 1.484(d). Any
response received after the expiration of the time limit set in the Written Opinion will not be considered in
preparing the International Preliminary Examination Report.

V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued):

Claims 1-18 lack novelty under PCT Article 33(2) as being anticipated by the patent to Mills.
As to claim 1, see Fig. 1, which shows a processor 14, display screen 18, and cursor control
device 19. Mills also teaches 'window generation logic' to 'generate and display a first
window region' with view window 20, at Fig. 2, and at col. 3, line 68; and 'indicia generation
logic' to 'generate data for display in at least one display area in the first window' such as
windows 22,24 at Fig. 2, such that the 'indicia generation logic use message-based
communications to exchange information to coordinate activities' at col. 3, lines 22-26.
Mills also teaches a 'control strip' [claim 2] with controller 36, at Fig. 2, and at col. 4, lines
32-40; wherein at least one display area is variably sized [claim 3] using size boxes, at Fig. 2,
and at col. 4, lines 8-9; and that the size of the first window region is variable [claim 4] also
at col. 4, lines 8-9.
In addition, Mills teaches sizing the first window region so that none of the display areas are
visible [claim 5] with close box 28, at Fig. 2, and at col. 4, lines 7-8, or all [claim 6] or a
portion [claim 7] of the display areas are visible, both at col. 4, lines 8-9.

Mills also teaches that one of the display areas displays only information [claim 8] with video
window 22, at Fig. 2, or provides -s to control information [claim 9] with video window
24, at Fig. 2, or displays an additional display element [claim 10] with video window 22, at
Fig. 2.
As to claim 11, it corresponds to claim 1, and also includes the following: using the 'indicia
generation logic' to 'generate user sensitive graphics for display' at col. 4, lines 32-40;
wherein the 'window generation logic' determines when a data display area has been selected,
signals the indicia generation logic, which then initiates a response, at col. 2, lines 9-13.
Claims 12-14 correspond respectively to claims 6,2,3.
Regarding claim 15, Mills teaches the following: 'generating a first window' with view
window 22, at Fig. 2, to accommodate one display area, as with control window 24, for
'indicia' with control buttons 42-54, at Fig. 2, and col. 4, lines 37-40, by 'executing a first
programming module' at col. 3, line 66; 'displaying an indicia' at col. 4, lines 32-40;
'selecting one of the indicia' at col. 4, lines 48-55, and sending a message for generating the
display at col. 3, lines 22-26; with the programming module performing a function, at col. 4,
lines 38-40.
Mills also teaches 'status information' [claim 16] such as tentative-set state at col. 5, lines 1014, or fixed-set state at col. 5, lines 23-35, and describes the operating stat_e or mode of the
controller at col. 6, lines 21-32.
Form PCT/IPEA/408 (Supplemental Box) (January 1994)*

APLNDC00028578

w m OPINION

International application No.

PCT/US95/11025

Supplemental Box
(To be used when the space in any of the preceding boxes is not sufficient)
Continuation of: Boxes I - VIII

Sheet 11

In addition, Mills teaches 'control information' [claim 17] in control window 24, at Fig. 2, and
at col. 4, lines 32-40. As to claim 18, Mills teaches that the first programming module
requests a set of features at col. 4, lines 20-25, sends a message to the programming module
indicative of features, and the programming module returns a message; such that the
programming modules interact with each other in response to user interaction with the first
programming module, all at col. 4, lines 25-40.
Claims 1-18 lack an inventive step under PCT Article 33(3) for the reasons given forth above.
Claims 1-18 have industrial applicability as defined by PCT Article 33(4) because the subject
matter claimed can be made or used in industry.
-- NEW CITATIONS
US, A, 4,896,291 (Gest et al) 23 JANUARY 1990, see Fig. 1, which shows graphical user
interface tools, such as slide bars 12, dials 11, and menus 14.
US, A, 5,202,961 (Mills et al) 13 APRIL 1993, see Figs. 2,3, which show a slider bar 36,
with velocity and direction indicators, and a control icon 40.

Forn PCT/IPEA/408 (Supplemental Box) (January 1994)*

APLNDC00028579

65

JDS

I

Paten

' E UNITED STATES PATENT AND TRADEMARK OFFICE

In re Application of

)

Steven W. Christensen

) Examiner:

Dela Torre, C.

Serial No. 08/316,237

) Art Unit:

2415

Filing Date: September 30, 1994

)

For: METHOD Amn APP^R^TUS FOR
DISPLAYING AND ACCESSING
COIN 1xOL AND STATUS
INFORMATION IN A COMPUTER

)
)
)
)

SYSTEM

)

AMENDMRNT

Commissioner of Patents
and Trademarks
Washington, D.C. 20231

Sir:
In response to the Office Action mailed March 20, 1996, Applicant
respectfully requests the Examiner to enter the following amendments and
consider the following remarks:

IN · - SPECTFICATION

At page 2, line 1
At page 4, line 10,

ease replace "it" with --It--.
ase replace "individiual" with --individual--.

At page 21, line 23, please insert --not-- after "has".
Serial No. 08/316,237

1

04860.P1365

APLNDC00028580

At page 28, lines 11-12, please delete "If the cursor location is

determined to be within the control strip."
AT page 28, line 15, please insert -within the control strip-- after
"occurs".

IN THF. CT.ATMS

.

2

(Amended) An interactive computer-controlÏ&d d s¯pÏay system

comprisin :

3

a pro ssor;

4

a data d play screen coupled to the processor;

5

a cursor c

6

7

trol device coupled to said processor for positioning a

cursor on said data isplay - a w;

a window gen ation and contr.ol logic coupled to the processor and

8

data display screen to c ate an operating environment for a plurality of

9

individual programming

odules that provide status and control funci;ions,

10

wherein the window genera

n and control lo ic venerates and dis lavs a

11

first window region having a pl ality of display areas on said data display

12

screen, wherein each of the plural

13

the plurality of individual programuking modules;

of display areas is associated with one of

14

an_ indicia generation logic couple to the data display screen to executa

15

at least one of the plurality of programmin modules to generate inform=+1an

16

[data] for display in [at least] one of the plurah

17

window reyton wherein [a] at least one of the p rality of display areas and its

18

associated_prog---_---imodule is sensitive to us r input, and further

19

wherein the window generation anri can+ral logic an the indicia generation

20

logic use message-based communication to exchange in rmation to

Serial No. 08/316,237

2

of display areas in the first

04860.P1365

APLNDC00028581

21

coordinate activities

22

display activity.

1

5.

the indicia generation logic to enable interactive

(Amended) The display system defined in Claim 4 wherein the

2

first window region[s] is sized such that none of the phiralg of display areas

3

à [are] visible.

1

6.

(Amended) The display system defined in Claim 4 wherein the

2

first window region[s] is sized such that all of the plurality of display areas are

3

visible.

1

7.

(Amended) The display system defined in Claim 4 wherein the

2

first window region[s] is sized such that a portion of the plurality of display

3

areas n [are] visible.

1
2

1

8.

(Amended) The display system defined in Claim 1 wherein at

least one of the pluralg of the display [data] areas only displays information.

9.

(Amended) The display system defined in Claim 1 wherein at

2

least one of the display [data] areas acts to provide access to control

3

information when selected.

1

10.

(Amended) The display system defined in Claim 9 wherein said

2

at least one of the plurality of display [data] areas displays an additional

3

display element.

Serial No. 08/316,237

3

04860.P1365

L
APLNDC00028582

2

1 .
(Amended) An interactive computer-controlled display system
comprisin :

3

a proc ssor;

4

a data d play screen coupled to the processor;

5

a cursor c

6

trol device coupled to said processor for positioning a

cursor on said data isplay s--;

7

window gener tion and control logic coupled to the processor and data

8

display

reen to create an operating environment for a plurality_of

9

individual programming modules that provide status and control functions

10

wherein the window gener tion and control logic generates and displays a

11

first window region having

12

scre

13

the plurality of individual prog mming modules [, wherein the first

14

window region comprises at least

wherein each of the

lurality of display areas on said data display
ality of display areas is associated with one of

e data display area];

15

at least one indicia graphics g neration logic coupled to the processor

16

and the window generation logic, wh ein said at least one indicia graphics

17

generation logic generates user sensitive

18

one data display area by executing at leas one of the plurality of programming

19

modules;

20

aphics for display in said at least

wherein the window generation and c ntrol logic determines when

21

said at least one data display area has been sele

22

at least one indicia graphics generation logic in r ponse to user selection, and

23

further wherein said at least one indicia graphics g neration logic initiates a

24

response from said at least one of the plurality of pr gramming modules.

1
2

d by the user and signals said

(Amended) The display system defined in Claim

wherein the

first window regn is always visible to the user.
Serial No. 08/316,237

4

04860.P1365

APLNDC00028583

1 .
' 2
3

4
'

5

(Amended) A method for generating control information

comprisin the steps of:
create n operating environment for a plurality of individual

programming

odules that provide status and control fimetions·

generating

first window sized to accommodate a pluraligof [at least

6

one] display areas

7

the first window com rises] executing [a first] at least one of the plurality of

8

individual programm°

9

areas is associated with oe of the plurality of individuaLyrog

10

r indicia resulting_Jrom [, wherein the step of generating

modules, wherein each of the plurality of displaX
a

maanle=·

11

displaying an indicia ° each of said at least one display area by

12

executing one of a plurality of rogramming modules corresponding to each

13

indicia;

14

selecting one of the indicia,

herein the step of selecting comprises the

15

first progr ------- --g module determin

16

area is selected and sending a message

17

plurality of programming modules respon ible for generating the display of

18

the selected indicia;

19

20

which of said at least one display
the programming module of said

said progr-------ing module performing

function in response to the

selection.

Please add the following new claims:

I(
1
2

(New) The display system defined in Claim 1 wherein each of
the plurality of display areas is individually and variably sized.

Serial No. 08/316,237

5

04860.P1365

APLNDC00028584

1
2

(New) The display system defined in Claim 1 wherein the first
window region always appears in front of application windows.

/ 3
1

(New) The display system defined in Claim 1 wherein the first

2

window region is implemented in a private window layer that appears in

3

front of windows for all applications layers.

1
2

(New) The display system defined in Claim

the plurality of display areas is individually and variably sized.

1
2

wherein each of

(New) The display system defined in Claim

wherein the first

window region always appears in front of application windows.

1

(New) The display system defined in Claim

wherein the first

2

window region is implemented in a private window layer that appears in

3

front of windows for all applications layers.

Applicant respectfully requests reconsideration of this application as
amended. Claims 1-18 remain in the application. Claim 1, 5-12, and have
been amended. Claims 19-24 have been added. No claims have been
canceled.
The Examiner has listed a number of informalities and errors in the
application. The Applicant has corrected these informalities and errors, as
well as others, to put the application in correct form for allowance.
The Examiner has rejected Claims 1-18 under 35 U.S.C. § 102(b) as being
unpatentable over
\

Serial No. 08/316,237

111= * =1 Mills teaches the use of a slider control bar for
6

04860.P1365

L
APLNDC00028585

controlling the rate of display of sequential information. Specifically, Mills
teaches the use of such a control window to control the playback rate of video.
This control window is defined as having certain components such as
standard playback direction/velocity indicators, reverse, stop and fast forward.
Mills. uses control icon to select one of these rates of display.
Claim 1 of the present invention as amended states, in part:

a window generation and control logic coupled to the
processor and data display screen to create an operating
environment for a plurality of individual progr-------ing
modules that provide status and control functions, wherein the
window generation and control logic generates and displays a
first window region having a plurality of display areas on said
data display screen, wherein each of the plurality of display areas
is associated with one of the plurality of individual
progr-,sing modules;

Thus, the present invention provides logic that creates an operating
environment like a shell for other programming modules to provide status

and control functions. Mills does not provide such an environment. In
Mille the control window is used for controlling video generated by an
application.
Claim 1 of the present invention also includes that "window
generation and control logic and the indicia generation logic use messagebased communication to exchange information to coordinate activities of the
indicia generation logic to enable interactive display activity" Contrary to the

Ex22,.°1L€r'S RSsertion, Mills does not teach the use of message based
communication for information exchange, particularly between control logic
that creates an envi-------ent for the plurality of programming modules
Serial No. 08/316,237

7

04860.P1365

APLNDC00028586

which control the indicia generation logic. In one described embodiment, the
message based communication comprises the control strip of the present
invention passing messages to a module to either tell it what to do or to
obtain information about the module and its capabilities (e.g., See

Specification, pg. 32, lines 5-7). There is no indication in Mills that such
message based communication is used. In view of the above discussion,
Applicant respectfully submits that Mills does not anticipate the present
invention as claimed in Claim 1, nor any of its dependent claims.

Independent Claims 11 and 15 also include a similar limitation
regarding the creation of the operating environment. Therefore, based on the
same rationale given above, Applicant respectfully submits Claims 11 and 15
and their dependent claims are not anticipated by Mills.

Applicant has added Claims 19-24. Claims 19-21 are dependent on
Claim 1, and Claims 22-24 are dependent on Claim 11. Claims 19 and 22
provide that each of the plurality of display areas is individually and variably
sized. Mills does not disclose such sizing. Claims 20 and 23 provide that the
first window region always appears in front of application windows.

Similarly, Claims 21 and 24 provide that the first window region is
implemented in a private window layer that appears in front of windows for
all applications layers. Mills clearly does not show this. In fact, as soon as

another application is used in Mills, the control window will be overlapped
and at least partially non-visible. Therefore, based on this, Applicant
respectfully submits that Claims 19-24 are allowable over the art of record.
Accordingly, Applicant respectfully submits that the rejection under 35
U.S.C. § 102(b) has been overcome by the amendments and the remarks and

withdrawal of these rejections is respectfully requested. Applicant submits

Serial No. 08/316,237

8

04860.P1365

L
APLNDC00028587

that Claims 1-18 as amended and Claims 19-24 as added are now in condition
for allowance and such action is earnestly solicited.

Please charge any shortages and credit any overcharges to our Deposit
Account No. 02-2666.
Respectfully submitted,

Bl .« K NLY, SOKOLOFF, TAYLOR & ZAFMAN

Dated:

, 1996
Micha J Mallie
Attorney for Applicant
Registration No. 36,591

12400 Wilshire Boulevard
Seventh Floor
Los Angeles, CA wo-1026

[phone]

I hereby certify that this correspondence is being deposited
with the United States Postal Service as first class mail with
sufficient postage in an envelo e addressed to the
ommissioner of Patents and rademarks, Washington, D.C.

0231 on Augmt on

Angel& Quinn

Serial No. 08/316,237

9

'

6

10

A

August 20, 1996
Date

04860.P1365

APLNDC00028588

Attorney's Docket No. 04860.Pl365

PA

IN THE • " •• ~••STATES PATENT AND = a = mviAKE OFFICE
In Re Patent Application of:

)

Steven W. Christensen

Application No.:
Filed:
For:

08/316,237

September 30, 1994

Examiner:

)

Art Unit:

C. Dela Torre

2306

)

METHOD AND APP A RATUS
)
FOR DISPLAYING AND STATUS)
INFORMATION IN A COMUTER )
SYSTRM

)

Assistant Co......:-sioner for Patents
Washington, D.C. 20231

REGENED

INFORMATION DISCLOSURE STATEMENT

Sir:

6800,

()

Enclosed is a copy of Information Disclosure Citation Form PTO-1449 together
with copies of the documents cited on that form. It is respectfully requested that the cited
--nts be considered and that the enclosed copy of Information Disclosure Citation
Form PTO-1449 be initialed by the E--··-----er to indicate such consideration and a copy
thereof returned to applicant(s).
Pursuant to 37 C.F.R. § 1.97, the submission of this Information Disclosure
Statement is not to be construed as a representation that a search has been made and is not
to be construed as an admission that the information cited in this statement is material to
patentability.
Pursuant to 37 C.F.R. § 1.97, this Information Disclosure Statement is being
submitted under one of the following (as indicated by an "X" to the left of
the appropriate paragraph):

37 C.F.R. §1.97(b).
X

37 C.F.R. §1.97(c). If so, then enclosed with this InformationÛ
Disclosure Statement is one of the following:
C
X

A certification pursuant to 37 C.F.R. §1.97(e) y
A check for $

LJV/cak (10/01/96)

for the fee under 37 C.F.R. § 1.17(p).

- 1 -

APLNDC00028589

37 C.F.R. §1.97(d). If so, then enclosed with this Information
Disclosure Statement are the following:
(1)

A certification pursuant to 37 C.F.R. §1.97(e);

(2)

A petition requesting consideration of the Information Disclosure
Statement; and

(3)

A check for $
for the fee under 37 C.F.R. § l.17(i)
for submission of the Information Disclosure Statement.

If there are any additional charges, please charge Deposit Account No. 02-2666.
Respectfully submitted,
BLAKELY, SOKOLOFF, TAYLOR &

Dated:

, \

-- LLP

, 1996
s C. Scheller, Jr.
g. No. 31,195

12400 Wilshire Blvd.
Seventh Floor

Los Angeles, CA 90025-1026
[phone]
I hereby certify that this correspondence is being deposited with the United States Postal Service as first class
mail with sufficient postage in an envelope adoressed to the Assistant Commissioner for Patents, Washington, D.
C. 20231on
L
\¾. ( 9
p
(Date of Deposit)

Dehemh McDovem
(Typed or printed na e of person mailing correspondence)

(Signature of person mailing correspondence)

LJV/cak (10/01/96)

to

o,

-2-

APLNDC00028590

AVAC.B To: PAPER &
Sheet
U.S. DEPARTMENT OF COMMERCE ATTY. DOCKET NO.
PATENT AND WADEMARK OFFICE

rm PTO-1449

EV. 8-83)

1

of

1

SERIAL NO.

04860.P1365

08

APPLICANT

4 TRA06f

INFORMATION DISCLOSURE CITATION

Steven W. Christensen
FILING DATE

(Use several sheets if necessary)

GROUP

9/30/94

289& 246

U.S. PATENT DOCUMENTS
*E

I ER

-umm. NUMBER

DATE

NAME

5 4 1 6 8 9 5 5/16/95 Anderson, et al.

CLASS

395-

-

†4W

IF FILING DATAETE

4/8/92

FOREIGN PATENT DOCUMENTS
DOC"MENT GUMBER

DATE

COUNTRY

CLASS

SUBCLASS

TRANSLATION
YES

NO

I

I
THER DOCUMENTS (including Author, 71tle, Date, Pertinent Pages, Etc.)

fj3

c

"PCT Written Opinion", October 4, 1996

Ch

EXAMINER

DATE CONSIDERED
Q .ORLA

-TORí2g

i\/(2/96

*EXAMINER: Initial if citation considered, whether or not citation is in conformance with MPEP 609; Draw line
through citation if not in conformance and not considered. Include copy of this form with next communication to
applicant.

APLNDC00028591

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll!
United States Patent [191
Anderson et al.

[11]
[45]

[54] SYSI'EM AND METHODS FOR IMPROVED
SPREADSHEET INTERFACE WITH
USER-Famr sax OBJECTS

Lotus Development Corporation, Lotus User's Guide:

Lotus 1-2-3 for Windows Release 1.0, 1991, pp. 37-46,
307-317.
Proceedingr Advanced Computer Technology, Reliable
Systems and Applications, 5th Annual European Computer Conf., CompEuro '91, May 13, 1991, Italy, pp.
282-286.
Andreas Hutflesz et al., The R-File·An Effcient Access
Structurefor Proximity Queries, Proceedings Sixth International Conference on Data Engineering, Feb. 5-9,
1990, Los Angeles Airport Hilton and Towers, Los
Angeles, Calif., USA, pp. 372-379.
D. Cakmakov et al., "A Multimedia Cognitive-Based
System Model" Advanced Computer Technology, Reliable Systems and Applications, 5th Annual European
Computer Conference, Bologna, May 13-16, 1991, pp.

[73] Assignee:

Borland International, Inc., Scotts
Valley, Calif.
[21] Appl. No.: 866,658
[22] Filed:
Apr. 8, 1992

[52] U.S. CI.

. GO6F 17/30; GO6F 17/40

.............. 395/148; 395/159;

395/161; 395/158; 395/155
[58] Field of Search ............... 395/155, 148, 158, 149,
395/159, 160, 161; 340/747, 748; 364/401;
345/24, 25
[56]
References Cited
U.S. PAxen1 DOCUMENTS
5,021,973 6/199

5,033,009 7/199

282-286.

Birkholz, "Die Datenbank in der Tabellenkalkulation,"
CIIIP 2eitschnf fur Mikrocomputer-Technik, No. 3,
Mar. 1991, pp. 450-454.
"Hyperpaper," IBM Technical Disclosure Bulletin, vol.
33, No. 9, Feb. 1991, pp. 283-384.

Hernandez et al. ................ 364/518

Dubnoff .............................. 364/523

Pdmary Examiner-Mark K. Zimmerman
Assistant Examiner-J. Feild
Attorney, Agent, or Firm-John A. Smart; David N.
Slone; Vernon A. Norviel

5,055,998 10/ 99 Wright et al. ....................... 364/200
5,072,412 12/ 99 Henderson, Jr. et al. .......... 395/159
5,093,907 3/ 992 Hwong et al.
...... 395/152

5,095,429
5,121,499
5,140,678
5,182,793
5,231,577
5,237,651
5,283,864

3/ 992
8/1992
1/1993
7/1993
8/1993
2/1994

Harris et aL ........................ 364/408
Torres ........-........................ 395/159
Alexander et al. .---............... 395/13
Koss .................-......--........ 364/419
Randall
........ 395/148
Knowlton ........................... 395/158

6/1992 McCaskill et al- .................. 395/700

[57]
ABSTRACT
An electronic spreadsheet system of the present invention includes a notebook interface having a plurality of
notebook pages, each of which may contain a spread of
information cells, or other desired page type (e.g.,
Graphs page). Methods are provided for rapidly accessing and processing information on the different pages,
including displaying a plurality of page identifiers for
selecting individual pages, and further including a preferred syntax for referencing information. Additional
methods are provided for editing cells and blocks of
cells.

FOREIGN PA mm DOCUMENTS
0428261
0431638A2
0443184A2
WO92/04678
WO92/08199

5/1991
6/1991
8/1991
3/1§92
5/1992

5,416,895
May 16, 1995

Lotus User's Guide: Lotus1-2-3for Windows Release l.0;
1991; pp. 159-171.
Lucia 3-D Tutorial Guide, Chapter 16, pp. 1-18, 1987.

[75] Inventors: Charles R. Anderson, Santa Cruz;
Robert W. Warfield, Aptos; Istvan
Cseri, Scotts Valley; Murray K. Low,
Santa Cruz; Weikuo Liaw, Scotts
Valley; Alan M. Bush, Palo Alto, all
of Calif.

[51] Int. CL6 ........

- ' USOO5416895A

Patent Number:
Date of Patent:

European Pat. Off. .
European Pat. Off. .
European Pat. OfT .
WIPO .
WIPO .

OTHER - CATIONS
Microsoft Excel User's Guide, 1990, pp. 76-79·

.

17 Claims, 35 Drawing Sheets

250

APLNDC00028592

U.S. Patent

May 16, 1995

Sheet 1 of 35

5,416,895

104

KEYBOARD

105

PONTING
DEVICE

¯

ios
102

D--

DISPLAY
107
103

MASS

STORAGE
108

PRINTING
D••·,

-

MAIN
MEMORY

Ilo
CONTROLLER

-

101

CENTRAL
PROCESSOR
110

FIG. 1A
150
152

151

153

APPLICATION T
SOFTWARE
OPERATING
F
SYSTEM
A

USER

FIG. iB
APLNDC00028593

172

160

--

174

- 175

•
.

Sample Appilcation

EIIe Edit Iext grawing Ereferences

+Y

The quick brown
182
{TEXTOBÆCT)

f0x jumped over ...

r- m

165

181
(GRAPHIC OBÆCT
(D

FIG. 1C

178

APLNDC00028594

r

U.S. Patent

C‡

Sheet 3 of 35

5,416,895

C3

w·

May 16, 1995

CO

N

N

S

APLNDC00028595

220
221

224

\

l

cut

I

copy pastel C

I

i

22L

22L

.
226
227
229
Ililli

Normal
225

•

a...2 7
22

FIG. 28

A
261a

262a

263a

FIG. 2D
260b

b

FIG. 2E

'y

APLNDC00028596

'

A

250
,253

,,256(CELLC4")

255

254

NOTEBK1.WB1

liilt

118

257···

252-

.*

cilD

(0

|
E

F

I'd

259

e+

GlH

4

5
a
7

1

25

10
11
12

13
14

DJ
DY

©

15
18

271

260

272 273

FIG. 2C

278

CA
00

APLNDC00028597

U.S. Patent

May 16, 1995

Sheet 6 of 35

5,416,895

260a
I

2eaa

271a

261a

272a

FIG. 3A

273

2eoo
I
262c

271c

Graphs

261c

E

272c

FIG. 38
F

G
..

H
.

+I

273

259

VERTICAL
NAVIGATION

HORIZONTÆ NAVIGATION

FIG. 3C
APLNDC00028598

U.S. Patent

May 16, 1995

Sheet 7 of 35

5,416,895

COLB

410

I

411

13-ir

I

2 3 -

I

*

fŒff/MI
F////ff//l

-

Y/////'7/A

i

I 6

f////////d

6

V/ff(//fd

FIG. 4A

421
420ROW3

-i
4

-

A

-

e

_

r

-

V/////Æ///////

.I

6

410m

L-i .

420 ¡

\

myyNI

1 I
N

1 2 ,

YR/«///1
V/ ///

I

* -

Yl//f///d

I

6 -

FIG.48

c

(«/€///1

! 5 -

VN¥

\

.!

FIG. 4C
431 - 4\ /////////////////////////A

FIG. 4D
441

' 4

, , , , , , ////////

/////M 27////E,

.....

442

FIG. 4E
APLNDC00028599

U.S. Patent

May 16, 1995

Sheet 8 of 35

5,416,895

to

8

CO

APLNDC00028600

I

A

- Elle

B

Edit Block

\

C

|

D

E

I

F

H

1

c

Ouattro Pro for Windows - PROFITS.WB1
Ioola §raph Eroperty
ndow Belp

D.ata

IBIBIRI IHeading2 le]
Salads:04

G

Igla.

1

1 PROFITS: SALADS
2

3

4
7
8
9
10

La
Food Cost

(D
þ|g

J
am

liÏ

Total Öõi

1.00

6Ï
smummuma .

ofit

2
397.22%

Las

...

fahd

73

4
17
«D

51.00
$2.75

24

$0.50
$2.75

$1.75

22

52.25

00%

FIG. 4G

0.75
$2.75
07

36Ò ËϾ

00

APLNDC00028601