Patent Harbor, LLC v. Apple, Inc.

Filing 1

COMPLAINT against Apple, Inc. ( Filing fee $ 400 receipt number 0540-4598128.), filed by Patent Harbor, LLC. (Attachments: # 1 Exhibit A - '514 Patent, # 2 Exhibit B - Claim Construction Order, # 3 Civil Cover Sheet, # 4 Summons(es) Summons to Apple, Inc.)(Cooke, Michael)

EXHIBIT “A”

U.S. Patent

Nov. 4, 1997

sheet 1 of s

5,684,514

111 ’ï

1011")
v|DEo MONITOR

m
M
1c
m

A

m
D
1

rH._-|_/4l _11411K_1Á|H4I1
_
_
_

_
_
_

A
D»
O

__

à
œ
Rr1
wm
w
œ
r

IlIl_l
.IIL
|

:Gb
m
<

_
_

\
\C
\l\ d¿mmmmme
\ _\ 11mmm
w01œ„.0n
wmm

œ

O

CAMERA
POSITION
CONTROL

lr.-___.__________._-________

FIG. l

U.S. Patent

Nov. 4, 1997

sheet 2 of s

CONTENT MONITOR

5,684,514

m
200

Ã_____

,I

ñ

I' +--- f =\
'

A

l =I
'| I
' f I@ f I

207 /

:Il

L_______1

\

f 206
cuRsoR CNTL

r

201

INPUT
DEVlCE

Y

COMPUTER
\

'

202

vIDED DISK # 1

vIDEo DISK :I 2

f 203

/\ 204
SWITCH

I

vIDED MONITOR
I_EFT
SIDE

f 205

/ î FRONT
SIDE

BOTTOM
SIDE

FIG. 2

U.S. Patent

Nov. 4, 1997

5,684,514

sheet 3 of s

ß soo
GENERATE CONTENT IMAGE BASED ON
DESIRED VIDEO FRAMES

ASSIGN KEYS TO POSITIONS IN CURRENT
IMAGE

CONTROL CAMERA POSITION IN RESPONSE
TO KEYS AND GENERATE
VIDEO FRAMES

STORE KEYS WITH GENERATED
VIDEO FRAMES

COMPILE VIDEO FRAMES FOR ACCESSING IN
RESPONSE TO KEYS

PROVIDE USER INPUT DEVICE FOR
SELECTING VIDEO FRAME IN RESPONSE TO
POSITION ON
CONTENT IMAGE

FIG. 3

U.S. Patent

Nov. 4, 1997

sheet 4 of s

5,684,514

ß 40o
sToRlNG vlDEo
FRAMES

ß 401
AsslGN KEY To EACH
VIDEO FRAME

GENERATE CONTENT lMAGE

p 402

BASED oN KEYS

coMPlLE vlDEo FRAMES FOR
ADDREsslNG |N RESPONSE

/\ 403

To KEYS

n 404
PROVIDE USER INPUT DEVICE FOR
SELECTING VIDEO FRAME IN RESPONSE TO
POSITION ON CONTENT IMAGE

FIG. 4

U.S. Patent

Nov. 4, 1997

sheet s of s

5,684,514

SCRIPT

Transitions

Transitions

C 501-1 / \(` 501-2

Exryessions

Events

500-1

Y 501 -3

Expressions

Events

Events

500-2

Events

500-3

500-4

/Expressions
Events

Events

L 500-5

50G-6

FIG. 5

Cement Image VWth Graphic Space

Are we really

happy

with

this

lonely

FIG. 6

game

weplay ?. ..

5,684,514
1

2

APPARATUS AND METHOD FOR
ASSElWBLING CONTENT ADDRFSSABLE
VIDEO

by the associated frame. A processing unit, assembles a
content video image in response to the tags, including
positions in the content image for corresponding frames of

CROSS REFERENCE TO RELATED
APPLICATIONS

provided for associating the positions in the content video
image with addresses of storage locations storing the cor
responding frames of video data. A user input device is
provided by which the user selects a particular frame of
video data by selecting a position in the content video image,

video data. Finally, a means, such as a look-up table, is

This application is a continuation of application Ser. No.
08/l46,400, filed Nov. l, 1993, now abandoned, which is a

continuation-in-part of application Ser. No. (J8/000.927,
tiled Jan. 6, 1993, now abandoned, which was a continuation

10

of application Ser. No. 07/460,849, filed Dec. 8, 1989, now
abandoned.
The present application is related to U.S. Pat. No. 4,857,
902, entitled POSII‘ION-DEPENDENT INI‘ERACI‘IVII'Y
SYS'I'EM FOR IMAGE DISPLAY, invented by Naimark, et

stood by considering the following example. If a plurality of
frames of video data comprise iilrn of a geographic area,
each frame can be tagged with a position stamp, indicating
the position of the camera when it was taken, and the focal
point of the camera. This position stamp can then be
translated automatically to a virtual position on a map. The

al., issued Aug. 15, 1989; and to U.S. patent application Ser.
No. 07l356.543, entitled CONTENT ADDRESSABLE
VIDEO SYSTEM FOR IMAGE DISPLAY. filed May 5,
1989.

20

25

of video data so that it may be accessed in response te the
content of the respective frames.

BACKGROUND OF THE INVENTION

30

Interactive video systems, such as described in the above
cross referenced U.S. Pat. No. 4,857,902, in which a ûrst
monitor displays a virtual space which indicates the content
of a library of video frames, and a second monitor displays
video frames, in response to the position of a cursor on the 35

frames based on the content of the video frames. The content
is indicated by the virtual space. To access video based on
content, a user moves a cursor to a position on the virtual

space. In response to the position of the cursor on the virtual
space, a video frame is accessed which has a content

from the content video image. A processing unit then asso
ciates each frame of video data generated by the controllable
camera, with positions in the content video image.
The generation of content addressable video according to
this aspect of the invention can be understood by consider
ing the following example. Assume that one wanted to
generate a content addressable video library of an object,
such as the transmission of an automobile. A content video

indicated by the position of the cursor.
A problem has arisen in compiling video frames for the
systems. In particular, the video frames are generated in such
systems based on a plan organized by a human producer. The

cursor is then translated into the position stamp of the frame
of video data, and from the position stamp, into an address
for the frame.
According to another' aspect, the present invention com
prises an apparatus or method for generating content addres
sable video, which is the converse of assembling the content
addressable video. In particular, according to this aspea, the
content video image is ñrst generated. Positions in the
content video image are then translated by a control circuit
into camera positioning signals. A controllable camera, such
as a robot mounted camera, then generates the frames of

video in response to the position control signals derived

virtual space, provide a powerful system for accessing video

purposes of addressing them by content for use in such

map superimposed with the virtual positions of all the video
frames in the plurality then constitutes the content video
image. This content video image can be displayed using a bit
map graphic system, in which a cursor can be positioned on

any position in the content video image. The position of the

FIELD OF THE INVENTION

The present invention relates to systems for generating
and storing video frames of information, such as produced
using video/film cameras. In particular, the present invention
provides a method and an apparatus for assembling frames

such as by positioning a cursor on the selected position.
’I‘he assembly of the content video image can be undm'

45

ñlm is then shot, such as by iiying a helicopter on a

image would then be graphically created in which an icon
representing the transmission is placed in a virtual graphic
space. A grid, three dimensional, or two dimensional, is then
drafted using computer software graphics packages, to cre
ate a grid indicating the pa'spective ofthe transmission to be
filmed. Thus, each position in the grid would correspond to

correlation of the virtual space to the video frames is done

a frame of video data depicting a diferent perspective of the
transmission.
A cursor is then scanned along the grid, either manually
or using automatic computer programs, and cursor position
is then translated into a camera control signal. The camera,
mounted on a robot, then traverses real space corresponding
to the virtual space of the content video image in response
to the control signals, while filming the transmission. A
processing unit then associates a position tag with each
frame of video, and ca'relates the position tag with a

manually in a time-consuming, tedious process.

position in the content video image.

geographic grid while filming a center point of reference.
For instance, in the one system, a helicopter flew over the

Golden Gate Bridge in San Francisco, while focusing the

50

camera on the center of the bridge.

Once the ñlm has been generated, it must be manually
compiled into sequences of video frames which correspond
to the grid ilown over the city. The grid is then drawn in a
bit map graphic form for display as the virtual space. Al

It is desirable to provide a method for compiling or

According to the invention, a user input device is
provided, by which a user selects a particular frame of video
by positioning a cursor on the content video image. This

generating content addressable video automatically.
SUMMARY OF THE INVENTION

The present invention provides an apparatus and method
for assembling content addressable video which is based on
storing a plurality of frames of video data at addressable
storage locations. Each frame of video data is stored with a
tag which indicates the contents of the video image deiìued

allows for highly interactive video systems in which frames
of Video data are addressed by content in an easily under
standable and fun to use manner.
65

Other aspects and advantages of the invention can be seen
upon review of the drawings, the detailed description, and
the claims which follow.

5,684,514
4

3

Thus, the system of FIG. 1 can be generalized by pro

BRIEF DESCRIPTION 0F THE DRAwiNGs
FIG. 1 is a schematic diagram of a system for generating,
or assembling content addressable video according to the

viding cameras attached to control arms that can move

within a confined space. A virtual map or content image of
frame content is mapped in a graphic image displayed on a
content monitor. Control of the cameras, and their move
ments and point of view, are functions of positioning of

present invention.
FIG. 2 is a schematic diagram of a system for interactively
displaying content addressable video according to the
present invention.
FIG. 3 is a flow chart illustrating steps in a method for
generating content addressable video according to the
present invention.
FIG. 4 is a llow chart illustrating the steps in a method for
assembling content addressable video according to the
present invention.

camm‘a icons or cursors in the graphic content image.

FIG. 2 illustrates the preferred system for displaying the
content addressable video according to the present inven
tion. In this system, a content monitor 200 is provided which
is connected to the computer 201. 'Iwo video disk players
202‘ and 203 store a plurality of video frames. A video

switcher 204 which is controlled by computer 201 selects
output video from the video disks in the players 202 and 203.

FIG. Sis a diagram of the assembly algorithm module for

The output video supplied to video monitor 205. A cursor

an uncontrolled camera environment.

control input device 206, like a mouse or track ball in the

FIG. 6 is a diagram illustrating the development of the
database for writing the script of a song.
DETAILED DESCRIPTION
A detailed description of preferred embodiments of the
present invention is described with reference to the figures.
In FIG. 1, a system for assembling or generating content

preferred system, is coupled to the computer 201. A user
interactively positions the cursor 207 within the content
video image 208 on the content monitor 200. A perspective
of the object is displayed on the video monitor 205 which is
indicated by the position of the cursor 207 in the content
image 208. Thus, for the virtual space content image 208
illustrated in FIG. 2. a frame corresponding to the position
of cursor 207, will provide a view of the object illustrated in
the content image 208, which shows the bottom side, left
side, and front side, as schematically illustrated in PIG. 2.
Background conca'ning how to implement these intœac

addressable video is illustrated. The system includes a

computer 100 which provides processing capability used in
assembling or generating the video. A computer is coupled
to a camera position control robot 101 or other mechanism
for controlling the position of a camera. The camera position

is controlled to generate a plurality of frames of video
information of an object 102 in a space. Video frames are
then stored in video storage 103. A video monitor 104 is

tive display systems can be found in the above cross
30

provided for displaying the video frames as they are being
filmed, or for accessing the video storage 103 to review the

addressable video. According to this method, a user uses a

computer software graphics package or other tools to gen

tilm. A content monitor 105 is provided which includes a

graphic iilm content image 106. The iìhn content image
includes a depiction 107 of the object to be filmed, and a grid
108 having a plurality of positions. Positions on the grid 108
indicate the content of corresponding video frames in the

referenced U.S. Pat. No. 4,857,902.
FIG. 3 illustrates the ilow chart for the method using the
apparatus of FIG. l, or similar apparatus, to generate content

35

erate a content image based on desired video frames to be

generated (block 300).
Next, the computer 100 is used to assign keys to positions
in the content image. For the content image illustrated in
FIG. 1, these keys would correspond to the X, Y, and Z

video storage. Thus, as the camera position control robot 101

positions on the grid 108 (block 301).
Next, the keys are used by the computer 100 to generate

moves the camera 109 along a plurality of camera positions

indicated by the grid 110, the computer assigns a key to each
frame of video. This key is correlated with a position in the

camera position control signals which control the robot to

graphic film content image 10S along the grid 108.

film the object in response to the keys (block 302). Next, the

keys are stored with or otherwise associated with each
A user input device 111, such as a mouse or track ball in
the preferred system, can be used to position a cursor icon 45 generated video image (block 303). The video frames are

then compiled in a format which facilitates accessing in

112 along the grid 108 in the graphic iilm content image 106.
Thus, the position of the cursor icon 112 indicates the
content of a frame of video being displayed on the video
monitor 104. Therefore, for the object 102 being lilmed from
carriera position as indicated at 109, and as represented by
the cursor icon 112 on the content image 106, an image

would be displayed in the video monitor 104, which shows
the bottom, left side, and front of the object 102, as Sche
matically illustrated in FIG. 1.
'I‘he system can be expanded to include a plurality of

response to the keys (block 304). The step of compiling
involves assigning the video frames to addresses in the Video
storage, and providing a means for translating the keys to the
50

55

address of the corresponding video frame. These are stan
dard computer techniques which can be based on look up

tables and the like. Also, this compiling step involves
identifying the frames of video on intersecting segments of
the grid 108 and storing them on video disks, er other
storage means, in storage positions which are accessible

cameras for filming a specific object, such as an automobile

within a seek time at least as fast as the update rate of the

transmission. Also. the object 102 can be moved instead of
the camera to generate the frames of video.
The content image 106 can be generated before filming
the object 102, and thereby used to generate camera position
control signals to control the generation of video. Similarly,
the film can be shot ñrst, and keys associated with each
frame of video. The keys can then be analyzed by the
computer 100 in order to gena-ate a content image 106. This

video monitor. In the preferred system, this is at least 15
times per second.
Finally, a user input device is provided for selecting a
video frame in response to a position on the content image

(block 305). Again, this is provided by positioning a cursor
on the content image using an input device. The position of
the cursor then identities a key which can be translated to an

address of the corresponding video frame. Using the

content image 106 can be generated at the same time that the 65 address, the video frame is accessed and displayed.
Therefore, the content of the video frames is used to
camera is being used to lilm the object, or it can be generated
access them in a straightforward, highly interactive manner.
on an as needed basis from a huge library of video frames.

5,684,514
5

6

Furthermore, this content image is used to automatically

A joystick controls the direction of the camera (cursor or
camera icon) while the cursor traverses the content image in
preview mode. This provides for accurate pan, tilt keys as
well as accurate position data. Once the operator decides this
content image and camera point of view are correct, he
implements an “animate" command and each frame is

control the generation of the video frames to be accessed in
this content addressable video scheme.
FIG. 4 is a ñow chart illustrating the steps used for

assembling content addressable video from a large library of
video frames.

This method involves first storing a library of video
frames (block 400). Such video frames may, for instance,
correspond to film of a highway system.
Next, a key is assigned to video frames or segments of
frames that are stored. These keys may correspond to
geographic position stamps of the location filmed in the
corresponding video frames (block 401). These keys can be
calculated using interpolation techniques in which a
sequence of video along a given highway. having a known
start point and stop point can be assigned keys based on the
estimated position of each frame.
The next step involves generating a content image based
on the assigned keys (block 402). According to this method,
the assigned keys are used to compile a map using bit map
graphics or the like, in which each key has a corresponding
position on the map. This content image can be generated for
a subset of the entire library video frames. which consists of
a few video frames or of the entire library.

rendered, stored and finally compiled for interactive control.
Hence. a system as described in FIG. 2 with the CAD and

animation tools with the content addressable image software
10

technology.
The key stamps consist of position, pan, and tilt with their
respective video frame derived directly from their location
in the graphic virtual space.
This module provides segments or geometric shapes to be
recognized by the camera paths. These shapes may be drawn
with the normal draw routines. The program will ask for the
frame density along the paths and the camm’a POV. The line
segments will parse frame locations along the line segments
and send keys out to a frame database and then calculate the
frame by frame animation. The frames are stored to optical
discs or other memory, such as large 3-D arrays of DRAMs
or VRAMs, accessed in a massively parallel means, and the
content image is assembled. The matrix arrangements are

25

The next step involves compiling the video frames for

addressing in response to the assigned key (block 403). The
generated content image is used to compile the video frames

compiled and a cursor appears on the content image in the

gr'aphic space once the compilation and assembly is com
plete providing control to the user.
B. Controlled Camera Environment

by identifying positions on the content image of adjacent
keys, and storing the video frames at addresses indicated by

In this method of data acquisition, the visual data is

acquired in controlled ñlming environments; specifically

the keys. For a large library of frames, a subset of the library
which corresponds to at least a section of the content image
is moved into a storage buffer that has reasonable access

those where cameras are controlled in stationary or motion

controlled environments.` The assembly of these content

addressable video systems will require predefined matrices
times, such as a writable optical disc or array of RAM.
35 of film maps generated on a graphic computer platform
Finally, a user input device is provided for selecting a
which translates actual commands to a camera motion
video frame in response to a position on the content image
control platform for videomap production and automated
(block 404). Having assembled a subset of the video frames
assembly. In this environment, the space which the cameras
by generating a content image representative of the content
can cover are scaled to the virtual graphic space such that an
of the subset, a user can use a highly powerful interactive
incremental movement of a ctn-sor on a content image
scheme for positioning a cursor on the content image to 40
solicits a relative movement of the camera (or video once

select the particular video frame having the desired content
'I'he basic tools and methodology for the assembly of

shot and compiled) in its respective space.

content addressable video systems are very similar for the
various data acquisition methods. 'I‘he process involves the

marriage between a graphic space, acquired visual data and

45

l. Scale Camera Locations in real space to camera loca
tions in the graphic content image space. Also zero out
camera pan, tilt and direction POV. Position subject in

the relational content image of this visual data in the graphic
space. In the system there exist at least an element of

organization, i.e., a physical location of visual data and its

content image space.

respective position in a graphic space (content image). This
organization is critical to the representation of information

2. Locate graphic model center at x, y, z in graphic virtual
space on motion platform.

and serves as the base methodology for a visual language.

Content Addressable Video systems may be built from
closed systems, controlled camera environments, and
uncontrolled or free camera environments. They may be
built from a combination of all three.

3. Build the content image in the graphic space using the
graphic software tool commands. Or build sequence in real
55

A. Closed System Environment

from a cursor which represents the focal plane of the camera.

space using actual cameras. The actual camera moves are

recorded, stamped, and converted to the graphic space.
4. Stamp keys on frames (or other recording methods
discussed below) by recording camera position, POV, Pan,
and Tilt in real time while recording frames.
5. Record filmed frames to optical disc player or to other

A closed system implies a graphic or artificial environ
ment. An example of such a system is a CAD workstation
which has tools for designing objects as well as for the three

dimensional animation of these objects frame by frame into
a movie. Software provides the producer the capability to
build the content image into the graphic space. The tools for
drawing a content image already exist in CAD systems.
Once the content image is constructed, a light is projected

In our system, we will place the subject, say an automo
bile transmission, on the platform.
"I‘he algorithm works as follows:

archive technology.
6. Convert camera position stamp locations to content

65

image scale and location. Store key data locations of camera
in the content image space frame by frame. Location data
consist of x, y, z location, Pan, Tilt, and focal plane.
7. Store locations of frames in content image space (x, y,
z) relationship of each frame to its corresponding content

5,684,514
7

8

image in a graphic space using keys and the system’s

Design of an Aerial Videomap
The design requires a nmp system which contains actual
spatial elements of the arca which is to be mapped. Since the
current iilming system is by helicopter and much of the data

processing unit.

`

8. Sort and determine the visual matrix as well as rela

tionship of each frame to its corresponding content image in

acquisition is implemented by sight flying with some GPS

a graphic space using keys and the systern’s processing unit.

navigation aid. A remote powered vehicle drone is utilized
for “close-in” mapping.
Position Stamping

9. Compile database in its matrix structure.
10. Generate content image space in virtual space.
11. Choose starting point in matrix.
12. Compile and organize frame data and reference con
tent image space relative to virtual space.

The current method of frame location in video is a
standard called SEMPI’E Time Code. Its structure is as

follows: Hour, Minutes, Seconds, Frames
In the preferred embodiment, this code is expanded into
the following structure: Hour, Minutes, Seconds, Frames,
Earth Reference Fixed GPS Receive Position, x, y, z, posi
tion GPS (Time T), Camn'a Receiver Position x, y, z GPS,

Camera System
'I‘he way it works is as follows:
Cameras are attached on control arms and can move

anywhere in one virtual space. A content image is drawn in
response to the camera movement or by hand in the graphic
virtual space (computer screen) such that an incremental

Velocity (Time T), Acceleration Time (T), Corrected Com
pass Camm'a Mount Facing (Time T), Pan degree (Time T),

move or matrix location in the camera space is equal to the

Tilt (Time T), Rotation (Time T), Frame Number and the
content image scale to the graphic space (map).

incremental vector in the graphic space. In other Words,
from picture to picture the video matrix corresponds to its
content image in its graphic space.

The sample rate of the data will be controlled by the
camera frarne rate or speeds which determine frame accurate

stamping. As frames are shot, they will be written to the

C. Uncontrolled (Free) Camera Environment
The third data acquisition environment deals with data
acquired in uncontrolled filming environments. This acqui
sition requires accurate position infomation and Pan (P),

nearest sample. Current GPS data is available on one second

intervals but may change given satellite positioning and
reception quality. Post processing is currently necessary to
average distance between GPS position samples, distance

Tilt (T) as frame stamping as keys for content image
assembly. These video matrices are assembled into accurate
survey mapping systems as well as other GIS systems for

display and overlay capability.

covered and number of frames per second. Thus, location

stamping requires post processing in this situation. The
satellite locations are scrambled and also have to be post
30

processed.

FIG. 5 is a diagram as to the modules of the code in the

The on-board filming system is designed to send data to

assembly algorithm. This system integrates an uncontrolled
platform (free) or other film systems for videomap produc

a counter and then to the data accumulator per frame of

tion. There is no limit placed upon the size of matrices. 'I'his

position database. 'I'he position stamps are included with

may require multiple stores of optical discs. In this case, the

video or film. If film is used, each reel will have its own
35

each individual frame. Currently, production technology

database structure requires an additional data location

provides a standard way of adding SEMP'I'E time code to

stamps for archive purposes.

video. Position stamping according to the present invention

Aerial or Free Cama'a Systems
In the case of free iilming camera systems, (i.e., cameras
in helicopters, balloons, Remote Piloted Vehicles, etc.) a

Methods of Storing Stamps and Prescribed Data Keys Are:
l. Actual time stamping and a comparison algorithm of

linal post processing step must be implemented to compile
accurate relationships between the content image space and

camera location at time (T) and frame exposure time (T).
2. Writing actual time and position data on sound track of
video.

its relative virtual space. The method of scripting of content
addressable video systems provide for a combination of
free-fam data acquisition and tight grid or matrix data

acquisition.

may follow the same standard methodology.

3. Use of computer memory for storing frame exposure

time and position simultaneously.
Trimble Navigation provides the Trimble Advanced Navi

important for tlying or developing videomaps for uncon

gation Sensor System for GPS. Our system will use this
technology or some other GPS system which is designed to

trolled spaces.

triangulate satellite signals for position. A second ground

The following additional aspects to this workstation are

l. Camera speed should be directly controlled to provide
an accurate matrix when using a ù‘ee virtual iilming system
such as a helicopter, balloon, ground gyrostabilized camera
mount such as a steadycam etc. This provides compensation
for environmental effects of wind and other factors which
alfect camera position.

station receiver will also be used to improve accuracy and
the data collected and merged with the data received (sample
rate) in the aircraft and processed to refine error rates and
improve position accuracy to an area of S meters cube or
better.

Another system is currently being designed which utilizes

2. Actual location of the camera is determined by updates
of camera location, pan, tilt axis on a frame by frame basis.

a mirror system on a drone PRV and a laser. The laser is

This information is processed back to the navigational
controls of the camera platform (including helicopter) in
order to provide corrections and adjustments in direction

rately surveyed, processes the laser band rellection and
samples actual RPV location for real time positioning. This
system and position stamp per frame will be similar,
however, it will provide very accurate positioning.

location and POV.
3. The system is designed with a predetermined content
image structure prior to recording video or iilm. This content
image structtn'e produces a navigational data set for the

camera. This navigational information incorporates support
ing flying surfaces such as those used to guide the iiying
machine.

directed at the PRV and a receiver whose position is accu

A software routine will build a system of frame location
in the three dimensional virtual space based upon the

following organization.
Content Addressable Video Assembly
The following treatment of the sottware tools which may
be used to develop a prescribed sequence of routes and

5,684,514
10
camera position which when implemented will provide an

An interactive movie must contain events. These events

autopilot control. camera position (pan and tilt) and frame

must be expressed. The interactive dimension is established
through creative transitional elements as illustrated in FIG.
5.
Let’s design an interactive piano and then assemble a
song using this method.

controlled for the data acquisition hence the tools for an

assembly language.
The elements of this system are:

l. Accurate maps of prescribed videomap area.
The U.S. Geological Survey (USGS) has accurate map

I. Develop the database for the Script
A. Define the number of events 500-1 through 500-6.
These events are all chords in the music spectrum.
B. Create the music for each event and store the sound

ping maps in vector form of many areas which have been

processed by satellite. The design of a videomap begins with
this information in a workstation. It is not essential to have

this information for the design because the data acquisition

date. However, all productions begin with a script or plan

data chord (Ff-time).
C. Develop transitions between Events. Film expressions
(e.g.. 501-1 through 501-3) of all combinations of two
events. That is, all of the independent physical hand move

based upon a desired application and the sequences of visual
material which best lit the application. Obviously, there are

1I. Write the script

position stamping will provide the information necessary to
merge this infomation into these imap databases at a later

ments from chord event to chord event.

many accurate methods of obtaining pictures of the ground;

they include SPOT Satellite, Space Shuttle picture systems

A. ‘This Masquerade" c1973 Teddy Jack

and high reconnaissance aircraft. A Moviemap or Videomap
of these pictures and their translation into continuous imag

Cm

m’y via a continuum of connected images in 25 FPS or 30

Cm7

F9

Cm
“E we really happy withG7 Cm game we play?
Ab? this lgely
Cm#’7

FPS or some visual translation from frame to frame in a

raster frame merger is another option and used in a closed

system.
Vìdeomap Production

Cm#'7

l'àkingforwordsgsaßggmhingbutnotñndipg
25

Cm7

F9

As mentioned above, the beginning design requires some
element of production scripting and design. The GPS system

undeüding anywgy!

will also be designed as a pilot aid for navigation. The

’I'herem lleventsploposedinthissongïheym:

B. List Events - (Underlincs indicate location of events.)

Cm

designer begins by using the Content Addressable Video
workstation to design his production, that is, the content
image to scale in the prescribed graphic space. A location
cursor in the aircraft in conjunction with the GPS hardware

30

and a two dimensional map with the actual routes to scale

will provide the pilot an on-board navigation system. This is
integrated with the autopilot. Though a good quality struc
ture improves the overhead processing requirements and
delta graphic overlay displays it is not essential to get it
exact. However, it is important to know precisely the posi

35

tion of what was recorded. Pilots are very capable of ilying

III. Now implement the automated assembly function.

VFR and instruments accurately with navigational assis

End result for preview is illustrated in FIG. 6.
First frame of hands in the Cm playing position El.

tance.

System prepared to cue ?rst Expression El-EZ (Cm-Cmit'ï)

D. Data Acquisition and Assembly of a Content

transition is the hand movement from Cm to Cm#'7. Inter
active control with cursor (***).
The assembly techniques can be used in a variety of 45 Sound Data and Relational Data Location
contexts. For example, an automatic assembly routine can be
Sound of Cm chord based upon movement of location
used for teaching music. In the music example: One might
cursor (***).
íilm someone playing a song on a piano (hands only). The
Content Addressable Video System Process (Automated
script of the song may be broken down to sound and visual
Procedure) Step 1
entities of frames of each depression of the keys. Each chord
The first step involves storing video frames as in block
and the associated frames would be an event. The assembly
400 of FIG. 4. The store of video frames will be stored in a
of two chords together would be an expression of events, the
database as set out in Table L entitled Database Frame

Addressable Video System

multiple expression of events would result in the script
So the fundamental logic or algorithms behind the Con
tent Addresssable Video System are driven and organized in
this example by this fundamental structure.

Location below. As can be seen in the Table I, trames are
55

This example assumes a database of extensive events and
a logical form of transition. It is essential to include a

distinction between the actual tools of organization and
assembly of data and the creative elements which provide a
transition continuity to the data.
A database must have a multiplicity of creative transi

tional elements. Current post production equipment has so

60

stored in sequences which consist of an expression which
proceeds from note Cm to note Cm#7, corresponding to a
transition from event E1 to event E2. This consists of frames
1-420. Each of the l0 expressions in the database are stored.
In the next step, a key is assigned to each video frame
corresponding to block 401 of FIG. 4. This is done by
searching the database frame location database for the
events El through E11. According to this step, a table such
as Table III, entitled Record Database, set out below is

tion etîects” as transitional elements. Hitchcock was a

computed. Thus, the iìrst entry in the Table I corresponds to
the first entry in Table Ill. The iirst event El is in original
frames 0-60. The transition from El to E2 is in original
frames 60-360, and the second event stored in frames

master of the study of transitional elements.

360420. This sequence is divided so that new frame

many “hacks” to handle a lack of these transitional elements.

A leading film maker established the use of “special transi

5,684,514
11

12

numbers 0-220 are stored with a key identifying them as
transition E1 to F2.
In the next step. a content image as shown in FIG. 6 is

TABLE III-continued

generated based on the keys. 'I‘his content image is generated
by signing each key a position on the content image as
shown on Table II below. Thus, event El will be assigned
X,Y position (0,160) on the XY scale. Table II also shows
the number of frames between each event on the display.
'I'he iinal step. the video frames are compiled for address
ing in response to the key corresponding to block 403 of
FIG. 4. This step of compiling video frames results in an
assembled database as shown below in Table IV. A given
ñ'ame, e.g. frame 0 is compiled so it can be accessed in
response to position (0,160) on the frame. In this
embodiment, a sequence of frames is displayed in response
to a user selecting the position which range from frame

x
B11

y

Dx

Dy(line map)

s0
so

5 E10

10o
10o

3o
50

0
0

#Framcs

3011606100)
501160000)

= s6
= 94

TABLE III

10

Record Database

Recorded
`

New

Event

ElE2
0-220 as shown in Table I1. In a more simple case as
EZdescribed above, only a single frame may be displayed in
E3
E3response to positioning of the cursor.
E4
Once the content image of FIG. 6 is displayed and the
E4assembled database of Table IV is computed, a user input
E5
ESdevise may be used for accessing specific frames in response
E6
to a position on the content image as described with refer
E6ence to block 404 of FIG. 4.
E7
25 E7These processing steps are thus executed by the processor
E8
201 of FIG. 2 and interconnected by the databases and tables
EScomputed by the processing steps.
E9
E9TABLE I
B10
E10-

Search chtabnae for El-Ell Expressions

Transition

Frames

Event

Frame tt

0-60

60-360

3

S60-420

1500-1560

1560-1860

5

1860-1920

221-400

2lœ-2160

2160-2460

l

2460-2520

401-670

3000-3060

3060-3360

5

3360-3420

671-850

350D-3560

3560-3860

3

3860-3920

851-1070

40œ-4060

4060-4360

1

4360-4420

1071-1340

5000-5060

5060-5360

15

5360-5420

1341-1480

6KB-6060

6060-6360

l

6360-6420

1431-1750

70œ-7060

7060-7360

5

7360-7420

1751-1930

I‘l H Il-l

8060-3360

3

8360-8420

1931-2150

0-220

E11

Database Frame Location (arbitgg for this mie)
Event
Frames

Transition
Frames

Event
Frames

1. Cm to Cmtt’î

1-60

60-360

S60-420

2. Cmtfî to Cm7
3. Cm7 to P9
4. P9 to Cm
5. Cm to A b7
6. A b7 to G7
7. G7 to Cm
8. Cm to CmtH
9. Cm#'7 to Cm?
10. Cm? to F9

1500-1560
Z100-2160
3000-3060
350D-3560
400G-4060
5000-5060
6000-6060
7000-7060

1560-1860
2160-2460
3060-3360
3560-3860
4060-4360
5060-5360
6060-6360
7060-7360
0060-8360

1860-192)
2460-2520
3360-3420
3860-3920
4360-4420
5360-5420
6360-642)
7360-7420
8630-8420

Eqxessim

Assembled Database
Node
Num

45

Determine location of all underlines in x, y screen coor

dinates of content image. In this case, the words designate
the map distance between events. In this example, the

TABLE II
x

y

Dx

El
E2
B3
B4
B5
ns

0
60
80
140
0
s0

160
160
160
160
130
130

60
20
80
30
60

E1

70

130

10

Bs
E9

a0
160

130
130

1o
80

Dy(line map)
0
0
0
0
0

#Frames

0

50l160(300)
311/160000)
80160000)
301160600)
601160900)

=
=
=
=
=

94
56
150
56
112

0

101160900)

- 150

0
0

10/160(3o0)
80060600)

= 19
= 150

Nodes

Node

Num
Frame

1
2

In this exnnple, the:

ditferential of the x, y coordinates between events provides
the transition scale for the expressions. Total line length
x=160. The special coordinates of the content image are
defined below in Table II. Table IL

TABLE IV

35

1
2

2
l

0
2m

3
4
5
6
7
8
9
10
11

2
2
2
2
2
2
2
2
1

2
3
4
5
6
7
8
9
10

41X)
670
850
1070
1340
1480
1750
1930
2150

Num
Noœ Fratrie
3
4
5
6
7
8
9
l0
11

220
400
670
850
1070
1340
1480
1750
1930

X,Y

Audio

0,160
60,160

Cm
Cm#7
C1117
F9
Cm
Ab7
G7
Cm
Cif!
Cm?
Fb

80,160
140,160
0,130
60,130
70,130
80,130
160,130
30,100
80,130

In Table IV, the tìrst column corresponds to a given node,

55

e.g. node B1 in the database. The second column identities
the number of nodes that are associated with the node in
column 1. The third column identities one of the adjacent
nodes. 'I‘he fourth column identifies the number of the frame
associated with the iìrst adjacent node identified in column
3. The ñfth column identities a second adjacent node, if any.
'I'he sixth column identifies the frame number associated
with the second adjacent node. The seventh column identi
ñes the X, Y coordinates on the content image of the node
of column l. The eighth column identities the audio which
is to be produced in association with the node in column 1.

E. Matrix Assembly Algorithms
1. Beginning, End. or flaw in frame data algorithm
Key stamps of the video data are read and stored into a

key database. This is particularly useful and essential for the

5,684,514
13

14

assembly of the video ñ'ame matrix. A number of algorithms
may be processed during the read and store function from

3. An apparatus for genm‘ating content addressable video,

comprising:

tape to the key database, including the following two impor

a content image display which displays a content video
image representative of an organization of content

tant components:
l. Relational Frame Matrix Algorithm
This process studies each frame and their keys to deter

addressable video, the content video image having
positions within the content video image corresponding
to desired content of video images to be displayed;
a controller, in communication with the content image

mine their matrix locations. This process establishes com

mon frames, nodes, and the spatial relationships for the
content image and its relational virtual space.

2. Image Recognition Key

10

tions within the content video image;

During the process of recording from tape to key database
an algorithm is used to study the differential in pixel change
from frame to frame in real time. A key is stored when the

controllable video image generator, responsive to the
control signals, which produces flames of video data,
each frame determining a video image having the
content indicated by the control signals corresponding

differential is large enough from frame to frame to locate a
ñaw or edit or cut between sequences. This will aid in

to the respective position in the content video image;
video storage which stores flames of video data generated

reaflirming the relational frame matrix algorithm
SUMMARY

by the controllable video image generator in storage
locations having addresses; and

This technology automates the organization and assembly
of video frame data such that the acquisition, assembly, and
ultimate integration of images may be represented in a
relational graphic space via a content image. These systems'

data processing resources, the controllable video image
generator and the controller, including resources that
execute program steps which associate the addresses of
the stored ñames of video data with respective posi
tions in the content video image.
4. The apparatus of claim 3, wherein the controllable
video image generator comprises a robot mounted video

real time capability are a function of speed memory and
memory burst rate and the platform will eñect the perfor
mance. The display format of these large stores of archived
image data, content images tiles and relational graphic
spaces form a powerful interactive visual display language.

cama'a.

The foregoing description of preferred embodiments of
the present invention has been provided for the purposes of
illustration and description. Itis not intended to be exhaus»

5. The apparatus of claim 3, further including:
30

an input by which a usm' selects a position in the content

35

a video monitor, response to the input and the tag storage,
which displays the frames of video data in the video
storage in response to selected positions.
6. A method for assembling content addressable video,

video image; and

tive or to limit the invention to the precise forms disclosed.

Obviously, many modifications and variations will be appar
ent to practitioners skilled in this art. The embodiments were

chosen and described in order to best explain the principles

of the invention and its practical application, thereby

comprising:

enabling others skilled in the art to understand the invention

storing, in an addressable memory, a plurality of llames of

for various embodiments and with various modiiications as
are suited to the particular use contemplated. Itis intended

that the scope of the invention be defined by the following
claims and their equivalents.
What is claimed is:
1. An apparatus for assembling content addressable video,

video data in storage locations having addresses, each
frame defining a video image having a content for
40

images defined by the associated flames;

video storage which stores a plurality of frames of Video 45

data in storage locations having addresses, each frame
defining a video image having a content for display;

frames;

of corresponding llames of video data in the plurality;
and
50

the video storage, including resources that execute first
program steps which assemble a content video image in
55

plurality; and

executing program steps which associate the positions in
the content video image with addresses of storage
locations storing corresponding frames of video data.
7. The method of claim 6, further including:
selecting, with a user input device, a position in the
content video image;
reading, with data processing resources, the frames of
video data in the addressable memory in response to a

selected position.

the processing resources including resources that execute

8. The method of claim 6, further including:

second program steps which associate positions in the
content video image with addresses of storage locations
storing corresponding flames of video data.
2. The apparatus of claim 1, further including:

storing in a cache memory a subset of the plurality of

frames, the subset including frames having content
indicated by at least a portion of the content video

image.

an input by which a user selects a position in the content

video image; a video monitor, responsive to the input
and the tag storage, which displays the frames of video
data in the video storage in response to selected posi
tions.

executing program steps which assemble and display a
content video image in response to the tags, the content

video image including positions indicating the content

processing resources, connected with the tag storage and
response to the tags, the content video image including
positions for corresponding flames of video data in the

display;
storing tags in memory for llames of video data in the
plurality, the tags indicating the contents of the video

comprising:

tag storage which stores tags for associated frames of
video data in the plurality, the tags indicating the
contents of the video images defined by the associated

display, which generates control signals indicating con
tent for video images in response to respective posi

9. A method for generating content addressable video,
65

comprising:
displaying a content video image representative of an
organization of content addressable video, the content

5,684,514
16

15

executing program steps with data processing resources
video image having positions within the content video
which associates the address of each frame of video
image corresponding to desired content of video
data with a position in the content video image.
images to be displayed;
10. 'I‘he method of claim 9, wherein the step of generating
selecting with data processing resources positions within
5 frames comprises:
the content video image;
controlling a robot mounted video camera in response to
executing program steps with data processing resources
the control signals.
which generate control signals indicating a content for
11. The method of claim 9, further including:
a video image in response to the selected positions
within the content video image;
generating frames of video data in response to the control

signals, each frame defining a video image having the
content indicated by the control signals;
storing generated frames of video data in storage locations
having addresses; and

selecting with a user input device a position in the content

video image; and
accessing the frames of video data in the storage locations
in response to selected positions.
*

*

ik

Ik

ik

UNITED STATES PATENT AND TRADEMARK oEEICE

CERTIFICATE OF CORRECTION
PATENT NO.

: 5,684,514

Page 1 of 1

APPLICATION NO. : 08/243046

DATED
INVENTOR(S)

: November 4, 1997
: Hi11Branscornb

|t is certified that error appears inthe above-identified patent and that said Letters Patent is
hereby corrected as shown below:

On Title Page should read,
[63] Continuation of Ser. No. 08/ 146,400, Nov. 1, 1993, abandoned, which is a
oontinuation-in-part of Ser. No. 08/000,927, Ian. 6, 1993, abandoned, which is a
continuation of Ser. No. 07,640,489, Ian. 11, 1991, abandoned.

Signed and Sealed this

Sixth Day of November, 2007

JON W. DUDAS

Director ofthe United States Patent and Trademark Oßìce

UNITED sTATEs PATENT AND TRADEMARK oEEICE

CERTIFICATE OF CORRECTION
PATENT NO.

: 5,684,514

Page 1 of 1

APPLICATION NO. : 08/243046

DATED
INVENTOR(S)

: November 4, 1997
: Hill Branscornb

|t is certified that error appears inthe above-identified patent and that said Letters Patent is
hereby corrected as shown below:

In column 1, Lines 8 to 13:
“This application is a continuation of application Ser. No. O8/ 146,400, tiled Nov. 1,
1993, now abandoned, Which is a continuation-in-part of application Ser. No.
08/000,927, tiled Ian. 6, 1993, noW abandoned, Which Was a continuation of application
Ser. No. 07/460,849, tiled Dec. 8, 1989, noW abandoned.”
Should read as:

--This application is a continuation of application Ser. No. O8/ 146,400, tiled Nov. 1,
1993, now abandoned, Which is a continuation of application Ser. No. 08/000,927, tiled
Ian. 6, 1993, noW abandoned, Which is a continuation-in-part of application Ser. No.
07/640,489, tiled Ian. 11, 1991, noW abandoned-

Signed and Sealed this

Third Day of June, 2008

JON W. DUDAS

Director ofthe United States Patent and Trademark Oßìce

US005684514C1

(12) EX PARTE REEXAMINATION CERTIFICATE (9613th)
United States Patent
(10) Number:
US 5,684,514 C1
Branscomb
(54)

(45) Certificate Issued:
(56)

APPARATUS AND METHOD FOR
ASSEMBLING CONTENT ADDRESSABLE
VIDEO

(75) Inventor:

Apr. 29, 2013

References Cited

To vieW the complete listing of prior art documents cited
during the proceeding for Reexamination Control Number
90/012,188, please refer to the USPTO’S public Patent
Application Information Retrieval (PAIR) system under the
Display References tab.

Hill Branscomb, San Francisco, CA

(US)
(73) Assignee: Patent Harbor, LLC, Tyler, TX (U S)

Primary Examiner ? Henry N Tran

Reexamination Request:
No. 90/012,188, Jun. 13,2012

(57)

Reexamination Certificate for:

Patent No.:
Issued:

A system and method for assembling or generating content
addressable video based on storing a plurality of frames of
video data at addressable storage locations. Each frame of

5,684,514
Nov. 4, 1997

Appl. No.:
Filed:

08/243,046
May 16, 1994

video data is stored With a tag Which indicates the contents of

the video image defined by the associated frame. For assem
bly, a processing unit assembles a content video image in
response to the tags: the content video image, including posi
tions for corresponding frames of video data. Finally, a
means, such as a look up table, is provided for associating the

Certificate of Correction issued Nov. 6, 2007
Certificate of Correction issued Jun. 3, 2008

Related U.S. Application Data

(63)

ABSTRACT

Continuation of application No. 08/ 146,400, filed on
Nov. 1, 1993, noW abandoned, Which is a continuation

positions in the content video image With addresses ofstorage
location storing the corresponding frames of video data. A

of application No. 08/000,927, filed on Jan. 6, 1993,

user input device is provided by Which the user selects a

noW abandoned, Which is a continuation-in-part of

particular frame of video data, by selecting a position in the

application No. 07/640,489, filed on Jan. 11, 1991,

content video image, such as by positioning a cursor on the

noW abandoned.

selected position.
For generating content addressable video, the content video

(51)
(52)

Int. Cl.
G09G 5/08
G06F 3/038
U.S. Cl.

image is first generated. Positions in the content video image
(2006.01)
(2006.01)

are then translated by a control circuit into camera positioning
signals. A controllable camera, such as a robot mounted cam
era, then generates the frames of video in response to the

USPC .......... .. 345/162; 345/157; 345/163; 345/184

(58)

Field of Classification Search
USPC ........................................ .. 345/157, 162, 184

See application file for complete search history.

position control signals derived from the content video
image. A processing unit then associates each frame of video
data generated bythe controllable camera, With positions in
the content video image.

Content Image Wth Graphic Spaoe

Arewereally

Looking

happy

for words

to

with

this

lonely

game

weplay?...

say...searchíng but not finding

US 5,684,514 C1
1
EX PARTE
REEXAMINATION CERTIFICATE
ISSUED UNDER 35 U.S.C. 307
NO AMENDMENTS HAVE BEEN MADE TO
THE PATENT

AS A RESULT OF REEXAMINATION, IT HAS BEEN
DETERMINED THAT:

10

The patentability of claim 6 is confirmed.
Claims 1-5 and 7-11 Were not reexamined.
*

*

*

*

*