The Regents of the University of California et al v. Staples, Inc.

Filing 1

COMPLAINT FOR PATENT INFRINGEMENT against Staples, Inc. ( Filing fee $ 350 receipt number 0540-3345775.), filed by The Regents of the University of California, Eolas Technologies Incorporated. (Attachments: # 1 5,838,906 Patent, # 2 7,599,985 Patent, # 3 Civil Cover Sheet)(McKool, Mike)

Download PDF
United States Patent Doyle et al. [ill Date of Patent: 1541 DISTRIBUTED HYPERMEDIA METHOD FOR AUTOMATICALLY INVOKING EXTERNAL APPLICATION PROVIDING INTERACTION AND DISPLAY OF EMBEDDED OBJECTS WITHIN A HYPERMEDIA DOCUMENT [75] Inventors: Michael D. Doyle, Alameda; David C. Martin, San Jose; Cheong S. Ang, Pacifica, all of Calif. [22] Filed: Oct. 17, 1994 [51] Int. C1.6 .............................. C06F 9/44; C06F 15116; C06F 17130 [52] U.S. C1. ................................ 3951200.32; 3951200.28; 3951680; 3951685; 3451326; 3451346; 7071501; 7071513; 7071515; 7071516 [58] Field of Search ............................... 3951157, 200.03, 3951161, 118, 144, 145, 146, 147, 148, 683, 777, 778, 762, 326, 333, 334, 335, 676, 682, 685, 684, 200.32, 200.33, 200.47-200.49; 7071501, 513, 515, 516; 3451326, 343, 346 References Cited ~561 U.S. PATENT DOCUMENTS , , 5,321,806 5,321,808 5,347,632 Barker et al. ........................... 7071516 Doyle ...................................... 3401706 Caro ................................... 3951200.03 Noguchi et al. ........................ 7071516 Vertelney et al. ...................... 3641419 Bernstein et al. ...................... 3951157 Khoyi et al. ............................ 3951683 Rouquie .................................. 3951705 Yin ........................................ 3951700 611994 Meinerth et al. ....................... 3951162 611994 Rupp et al. ............................. 3951164 911994 Filepp et al. ....................... 3951200.09 (List continued on next page.) Nov. 17,1998 OTHER PUBLICATIONS Stephen Le Hunte, "<EEMBED>-Embedded Objects", HTML Reference Library-HTMLIB v2.1, 1995: n.pag. Online. Internet. "A Little History of the world Wide Web", n.pag. Online. Internet: available http://www.w3.orgiHistory.html. "NCSA Mosaic Version Information", n.pag. Online. Internet: available "The second phase of the revolution", WIRED, Oct. 1994, pp. 116-152. [73] Assignee: The Regents of the University of California, Oakland, Calif. [21] Appl. No.: 324,443 5,838,906 Patent Number: [45] [I91 (List continued on next page.) Primary E x a m i n e r 4 i n h C. Dung Attorney, Agent, or Firm-Townsend Crew LLP [571 and Townsend and ABSTRACT A system allowing a user of a browser program on a computer connected to an open distributed hypermedia system to access and execute an embedded program object. The program object is embedded into a hypermedia document much like data objects. The user may select the program object from the screen. Once selected the program object executes on the user's (client) computer or may execute on a remote server or additional remote computers in a distributed processing arrangement. After launching the program object, the user is able to interact with the object as the invention provides for ongoing interprocess communication between the application object (program) and the browser program. One application of the embedded program object allows a user to view large and complex multidimensional objects from within the browser's window. The user can manipulate a control panel to change the viewpoint used to view the image. The invention allows a program to execute on a remote server or other computers to calculate the viewing transformations and send frame data to the client computer thus providing the user of the client computer with interactive features and allowing the user to have access to greater computing power than may be available at the user's client computer. 10 Claims, 9 Drawing Sheets Microfiche Appendix Included (4 Microfiche, 375 Pages) 5,838,906 Page 2 U.S. PATENT DOCUMENTS 5,367,635 1111994 Bauer et al. ....................... 3951200.32 5,390,314 211995 Swanson ................................. 3951500 5,418,908 511995 Keller et al. ....................... 3951200.32 5,544,320 811996 Konrad ............................... 3951200.09 5,581,686 1211996 Koppolu et al. ........................ 3951340 5,606,493 211997 Duscher et al. ................... 3951200.32 5,652,876 711997 Ashe et al. .............................. 7071516 OTHER PUBLICATIONS Vetter, Ronald "Mosaic and the World-Wide Web," Computer Magazine, v.27, Iss.10, pp. 49-57, Oct. 1994. Wynne et al. "Lean Management, Group Support Systems, and Hypermedia: a Combination Whose Time Has Come," System Sciences, 1993 Anuall Hawaii Int'l Conf., pp. 112-121. Hansen, Wilfred "Andrew as a Multiparadigm Environment for Visual Languages," Visual Languages, 1993 IEEE Symposium, pp. 256-260. Moran, Patrick "Tele-Nicer-slicer-Dicer: A New Tool for the Visualization of Large Volumetric Data", NCSA Technical Report (TR014), Aug. 1993. Berners-Lee "Hypertext Markup Language (HTML)", HTML Internet Draft, IIIR working Group, Jun. 1993. University of Southern California's Mercury Project"USC Mercury Project:Interfacen, Project Milestones, USC Press Release4btained from Internet, deptlraidersl. Hansen, Wilfred "Enhancing documents with embedded programs: How Ness extends in the Andrew ToolKit", IEEE Computer Language, 1990 International Conference. Tani, M., et al., "Object-Oriented Video: Interaction with Real-World Objects Through Live Video", May 1992, p. 593-598. Crowley, T., et al., "MMConf: An Infrastructure for Building Shared Multimedia Applications", CSCW 90 Proceedings, Oct. 1990, p. 329-342. Davis, H., et al., "Towards An Integrated Information Environment With Open Hypermedia System", ACM ECHT Conference, Dec. 1992, pp. 181-190. Ferrara, F., "The KIM Query System", Abstract, SIGCHI Bulletin, vol. 6, No. 3, Jul. 1994, pp. 3&39. Gibbs, S., "Composite Multimedia and Active Objects", OOPSLA '91, pp. 97-112. Davis, H., et al., "Microcosm: An Open Hypermedia System", Interchi '93, Apr. 1993, p. 526. Vaziri, A,, "Scientific Visualization in High-Speed Network Environments", Computer Networks and ISDN Systems 22, 1991, pp. 111-129. Cullen, J., et al., "The Use of FTAM to access graphical pictures across wide area networks", Computer Networks and ISDN Systems, 1992, pp. 337-383. Lashkari, Y.Z., et al., "PLX: A Proposal to Implement a General Broadcasting Facility in a Distributed Environment Running X Windows", Comput. & Graphics, vol. 16, No. 2, pp. 143-149, 1992. Kirste, T., "Spacepicture-An Interactive Hypermedia Satellite Image Archival System", Comput. & Graphics, vol. 17, No. 3, pp. 251-260, 1993. Coulson, G., et al., "Extensions to ANSA for Multimedia Computing", Computers Networks and ISDN Systems 25, 1992, pp. 305-323. Huynh, Duong Le, et al., "PIX: An Object-Oriented Network Graphics Environment", Comput. & Graphics, vol. 17, No. 3, pp. 295-304, 1993. Berners-Lee, T.J., et al., The World-Wide Web, Computer Networks and ISDN Systems 25, 1992, pp. 454-459. Shackelford, D.E., et al., "The Architecture and Implementation of a Distributed Hypermedia Storage System", Hypertext '93 Proceedings, Nov. 1993, pp. 1-13. Labriola, D., "Remote Possibilities", PC Magazine, Jun. 14, 1994, pp. 223-228. Udell, J., "Visual Basic Custom Controls Meet O L E , Byte Magazine, Mar. 1994, pp. 197-200. Sarna, D.E., et al., "OLE Gains Without (Much) Pain", Datamation Magazine, Jun. 15, 1994, pp. 31 and 113. Rizzo, J., "What's OpenDoc?", MacUser magazine, Apr. 1994, pp. 119-123. Fogarty, K., et al., "Microsoft's OLE can be network Trojan Horse", Network World Magazine, Jun. 27, 1994, vol. 11, No. 26, pp. 1 and 75. "Cello WWW Browser Release 1.01a", Article obtained from the Internet, no DDE, Mar. 16, 1994, pp. 2-9. "OLE 2.0: Death to Monoliths", Byte Magazine, Mar. 1994, p. 122. 20 f THIS HYPERMEDIA DOCUMENT INCLUDES AN IMAGE ICON J / 10 16 P /G 30- f 2 2 AJ~~ A SOUND ICON a)) AND SOME HYPERTEXT. # 26 J~~ TEXT, ASSOCIATED WITH IT. WHEW AN OBJECT I S OTHER TEXT THE DOCUMENT IS SAID TO BE A HYPERMEDIA DOCUMENT. \28 14 /A GYPERTEXTHAS ANOTHER OBJECT, PRIMARILY) IS TEXT THAT HYPERMEDIA DOCUMENT MAY HAVE IMAGE ICONS LIKE THl I AND OTHER OBJECTS EMBEDDED IN IT. I F/& PRIOR ART U.S. Patent NOV.17,1998 Sheet 2 of 9 5,838,906 U.S. Patent Nov. 17,1998 Sheet 3 of 9 CLIENT COMPUTER f f BROWSER CLIENT , 1 l 212 HYPERMEDIA DOC / / NETWORK PROTOCOL LAYER I SERVER COMPUTER f ' / NETWORK PROTOCOL LAYER 1 SERVER PROCESS 1 <2l8 \ ?220 APPLICATION (SERVER) ) 204 CLIENT COMPUTER r [ZOO \ I ADDITIONAL COMPUTER j CLIENT I y NETWORK PROTOCOL LAYER c SERVER COMPUTER ADDITIONAL COMPUTER f NETWORK PROTOCOL LAYER / 204 LC v CD s wl , , 0 / 224 w F/GI 6 \O CrO wl W CrO U U 0 w rn U.S. Patent NOV.17,1998 Sheet 6 of 9 ENTER 5,838,906 7 252 I PARSE DOCUMENT ASSIGN ENUMERATED TYPE lNlT PARAMETERS FOR DRAWING AREA CREATE INTERNAL OBJECT REPRESENTATION OF HTML EMBED TAG. U.S. Patent NOV.17,1998 Sheet 7 of 9 CREATE DRAWING AREA WIDGET / CREATE PlXMAP I ES ( ENTER REMOVE WINDOW, PlXMAP AND INTERNAL OBJECT REPRESENTATION ) - LAUNCH EXTERNAL APPLICATION 5,838,906 1I NCSA Mosaic: Document V i e n File Options Novlpcrte Annotate File Options W b d e Annotoh ~ocument V ~ s u a l ~ocument~lt$ Visual Embryo ProJrct interaotive - fl ,DMUmOlt Document U R L http://www. ~ I 1 http://ww~r likar y . ~ ~ 8 t . a d ~ / p ~ b l i ~ / ~ ~ ~ 11 Low coal Workstation ..-. Low coat Workata ... U fl I Demonstrat, ,/ 354 J This projrc.. . . . 0 I snapshot] I1 II Density I I I U.S. Patent Nov. 17,1998 Sheet 9 of 9 MOSAIC PAN EL - INTER-CLIENT COMMUNICATION I VISUALIZATION IMAGE COMMANDS DATA RENDERING REQUESTS 1 IHAGE SEGMENTS 1 2 objects. In this way, the user is able to navigate easily among data objects. The data objects may be local to the user's computer system or remotely located over a network. An early hypertext system is Hypercard, by Apple Computer, s Inc. Hypercard is a standalone system where the data objects are local to the user's system. When a user selects a phrase in a hypertext document that NOTICE REGARDING COPYRIGHTED has an associated link to another document, the linked MATERIAL document is retrieved and displayed on the user's display A portion of the disclosure of this patent document 10 screen. This allows the user to obtain more information in an efficient and easy manner. This provides the user with a contains material which is subject to copyright protection. simple, intuitive and powerful way to "branch off' from a The copyright owner has no objection to the facsimile main document to learn more about topics of interest. reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office Objects may be text, images, sound files, video data, file or records, but otherwise reserves all copyright rights documents or other types of information that is presentable whatsoever. to a user of a computer system. When a document is primarily text and includes links to other data objects BACKGROUND OF THE INVENTION according to the hvvertext format, the document is said to be a hypertext document. When graphics, sound, video or other This invention relates generally to manipulating data in a computer network, and specifically to retrieving, presenting 20 media capable of being manipulated and presented in a and manipulating embedded program objects in distributed computer system is used as the object linked to, the document is said to be a hypermedia document. A hypermedia hypermedia systems. Computer networks are becoming increasingly popular as document is similar to a hypertext document, except that the a medium for locating and accessing a wide range of data user is able to click on images, sound icons, video icons, etc., media types, such as from locations all over the world. The most popular global 2s that link network is the Internet with millions of computer systems additional graphics, sound, video, text, or hypermedia or hypertext connected to it. The Internet has become popular due to FIG. 1 shows examples of hypertext and hypermedia widely adopted standard protocols that allow a vast interdocuments and links associating data objects in the ~ O C U connection of computers and localized computer networks to communicate with each other. Computer systems con- 30 m a t s to other data objects. Hypermedia document 10 nected to a network such as the Internet may be of varying includes hypertext 20, an image icon at 22, a sound icon at types, e.g., mainframes, workstations, personal computers, 24 and more hypertext 26. FIG. 1 shows hypermedia docuetc. The computers are manufactured by different companies ment 10 substantially as it would appear on a user's display screen. The user is able to select, or "click on icons and text using proprietary hardware and operating systems and thus have incompatibilities in their instruction sets, busses, 3s on a display screen by using an input device, such as a mouse, in a manner well-known in the art. software, file formats and other aspects of their architecture and operating systems. Localized computer networks conWhen the user clicks on the phrase "hypermedia," softnected to the Internet may be incompatible with other ware running on the user's computer obtains the link assocomputer systems and localized networks in terms of the 40 ciated with the phrase, symbolically shown by arrow 30, to physical layer of communication including the specific access hypermedia document 14. Hypermedia document 14 hardware used to implement the network. Also, different is retrieved and displayed on the user's display screen. Thus, networks use differing, incompatible protocols for transferthe user is presented with more information on the phrase ring information and are not able to communicate with each "hypermedia." The mechanism for specifying and locating a other without a translation mechanism such as a "gateway". 4s linked object such as hypermedia document 14 is an HTML The Internet provides a uniform and open standard for "element" that includes an object address in the format of a allowing various computers and networks to communicate Uniform Resource Locator (URL). with each other. For example, the Internet uses Transfer Similarly, additional hypertext 26 can be selected by the Control Protocol/Internet Protocol ("TCPDP) that defines a user to access hypertext document 12 via link 32 as shown uniform packet-switched communication standard which is in FIG. 1. If the user selects additional hypertext 26, then the ultimately used in every transfer of information that takes text for hypertext document 12 is displayed on the user place over the Internet. screen. Note that hypertext document 12, itself, has hypertext at 28. Thus, the user can click on the phrase ''hyperOther Internet standards are the HyperText Transmission media" while viewing document 12 to access hypermedia Protocol ("HTTp) that allows hypertext documents to be exchanged freely among any computers connected to the ss document 14 in a manner similar to that discussed above. Documents, and other data objects, can be referenced by Internet and HyperText Markup Language ("HTML") that defines the way in which hypertext documents designate many links from many different source documents. FIG. 1 links to information. See, e.g., Berners-Lee, T. J., "The shows document 14 serving as a target link for both docuworld-wide web," Computer Networks and ISDN Systems ments 10 and 12. A distributed hypertext or hypermedia 60 document typically has many links within it that specify 25 (1992). many different data objects located in computers at different Ahypertext document is a document that allows a user to geographical locations connected by a network. Hypermedia view a text document displayed on a display device condocument 10 includes image icon 22 with a link to image 16. netted to the user's computer and to access, retrieve and One method of viewing images is to include an icon, or other view other data objects that are linked to hypertext words or phrases in the hypertext document. In a hypertext document, 6s indicator, within the text. the user may "click on," or select, certain words or phrases Typically, the indicator is a very small image and may be in the text that specify a link to other documents, or data a scaled down version of the full image. The indicator may DISTRIBUTED HYPERMEDIA METHOD FOR AUTOMATICALLY INVOKING EXTERNAL APPLICATION PROVIDING INTERACTION AND DISPLAY OF EMBEDDED OBJECTS WITHIN A HYPERMEDIA DOCUMENT - A 5,838,906 3 4 be shown embedded within the text when the text is disdisplay (LCD), etc.), local storage (hard disk drive, etc.), and other components. Typically, small computer 104 is conplayed on the display screen. The user may select the nected to a larger computer, such as server A at 106. The indicator to obtain the full image. When the user clicks on larger computer may have additional users and computer image icon 22 browser software executing on the user's to computer system retrieves the corresponding full image, s Systems ~onnected it, such as computer 108 operated by llO. group form a localized e.g., a bit map, and displays it by using external software network. Alocalized network does not necessarily adopt the called a hi^ results in the full image, represented uniform protocols of the larger interconnecting network by image 16, being displayed on the screen. (i.e., Internet 100) and is more geographically constrained An of a browser program is the Center than the larger network. The localized network may connect for Supercomputing Application's (NCSA) Mosaic software 10 to the larger network through a or impledeveloped by the University of Illinois at Urbanal mented on, for example, a server, &lother is On the Internet 100 connects other localized networks, such as Internet at Many viewers exist server at 120, which interconnects users 122, 124 and 126 that handle various file formats such as ".TIF," ".GIF," and their respective computer systems to Internet formats. When a browser program invokes a viewer Internet 100 is made up of many interconnected computer program, the viewer is launched as a separate process. The systems and communication links, Communication links view the image in a separate (in a may be by hardwire, fiber optic cable, satellite or other radio environment) Or On a separate screen. This wave propagation, etc, Data may move from server A to means that the browser program is no longer active while the server through any number of intermediate servers and viewer is active. By using indicators to act as place holders 20 communication links or other computers and data processing for images that are retrieved and when a equipment not shown in FIG, 2 but symbolically represented user selects the indicator, data traffic over the network is by Internet reduced. Also, since the retrieval and display of large images A user at a workstation or personal computer need not may require several seconds or more of transfer time the user connect to the Internet via a larger computer, such as server does not have to wait to have images transferred that are of 25 A or server B. This is shown, for example, by small no interest to the user. computer 130 connected directly to Internet 100 as by a Returning type data is a telephone modem or other link. Also, a server need not have object shown as sound icon 24 within the hypermedia users connected to it locally, as is shown by server C at 132 document. when the user selects sound icon 24, the user's FIG. 2. Many configurations of large and small computers computer accesses sound data shown symbolically by data O are possible, ' 40. The accessed data plays a Or Typically, a computer on the Internet is characterized as other audio device. either a "client" or "server" depending on the role that the As discussed above, hypermedia d ~ ~ u m e nallow a user ts computer is playing with respect to requesting information to access different data objects. The objects may be text, '5 or providing information. Client computers are computers images, sound files, video, additional documents, etc. As that typically request information from a server computer which provides the information, F~~this reason, servers are used in this specification, a data object is information capable of being retrieved and presented to a user of a usually larger and faster machines that have access to many computer system. Some data objects include executable data files, programs, etc., in a large storage associated with code combined with data. An example of such a combination 40 the server. However, the role of a server may also be adopted by a smaller machine depending on the transaction, ~ h is, ~ is a "self-extracting" data object that includes code to ''unpack or decompress data that has been compressed to user 110 may request information via their computer 108 make it smaller before transferring. When a browser from server A. At a later time, server A may make a request retrieves an object such as a self-extracting data object the for information from computer 108, the first case, where bmwser may allow the user to ''~aunch"the self-extracting 45 computer 108 issues a request for information from server A, data object to automatically execute the unpacking instruc108 is a making a request of information tions to expand the data object to its original size. Such a from server A, server may have the information in a A combination of executable code and data is limited in that ,torage device that is local to server server A may have A or the user can do no more than invoke the code to perform a to make requests of other computer systems to obtain the singular function such as performing the self-extraction after information, user110 may also request information via their which time the object is a standard data object. computer 108 from a server, such as server B located at a Other existing approaches to embedding interactive proremote geographical location on the Internet. However, user 110 may also request information from a computer, such as gram objects in documents include the Object Linking and Embedding (OLE) facility in Microsoft Windows, by small computer 124, thus placing small computer 124 in the Microsoft Corp., and OpenDoc, by Apple Computer, Inc. At 5 s role of a "server." For purposes of this specification, client least one shortcoming of these approaches is that neither is and server computers are categorized in terms of their capable of allowing a user to access embedded interactive predominant role as either an information requestor or program objects in distributed hypermedia documents over provider. Clients are generally information requestors, while networks. servers are generally information providers. Referring again to FIG. 1, data objects such as distributed FIG. 2 is an example of a computer network. In FIG. 2, 60 hypermedia documents 10, 12 and 14, image 16 and sound computer systems are connected to Internet 100, although in data file 40, may be located at any of the computers shown practice Internet 100 may be replaced by any suitable in FIG. 2. Since these data objects may be linked to a computer network. In FIG. 2, a user 102 operates a small document located on another computer the Internet allows computer 104, such as a personal computer or a work station. The user's computer is equipped with various 65 for remote object linking. components, such as user input devices (mouse, trackball, For example, hypertext document 10 of FIG. 1 may be keyboard, etc.), a display device (monitor, liquid crystal located at user 110's client computer 108. When user 110 t 5,838,906 5 6 makes a request by, for example, clicking on hypertext 20 nance Imaging (MRI) and Computed Tomography (CT), are (i.e., the phrase "hypermedia"), user 110's small client widely used in the fields of medicine, quality assurance and computer 108 processes links within hypertext document 10 meteorology to present physicians, technicians and meteoto retrieve document 14. In this example, we assume that rologists with large amounts of data in an efficient way, document 14 is stored at a remote location on server B. s Because visualization of the data is the best way for a user Thus, in this example, computer 108 issues a command that to grasp the data's meaning, a variety of visualization includes the address of document 14. This command is techniques and real time computer graphics methods have routed through server A and Internet loo and is been developed, However, these systems are bandwidthreceived server B. Server processes the command and intensive and compute-intensive and often require rnultiprolocates document l4 On its local storage' Server l4 then 10 cessor arrays and other specialized graphics hardware to transfers a copy of the document back to client 108 via carry them out in real time. Also, large amounts of secondary Internet and server A, After client computer for data are required. The expense these requirereceives document 14, it is displayed so that user 110 may ments has limited the ability of researchers to readily view it. exchange findings since these larger computers required to Similarly, image object 16 and sound data file 40 may reside at any of the computers shown in FIG, 2, Assuming IS store, present and manipulate images are not available to image object 16 resides on server c when user 110 clicks on many of the researchers that need to have access to the data. On the other hand, small client computers in the form of image icon 22, client computer 108 generates a command to personal computers or workstations such as client computer retrieve image object 16 to server C. Server C receives the 108 of FIG. 2 are generally available to a much larger command and transfers a copy of image object 16 to client computer 108. Alternatively, an object, such as sound data 20 number of researchers. Further, it is common for these file 40, may reside on server A so that it is not necessary to smaller computers to be connected to the Internet. Thus, it is desirable to have a system that allows the accessing, traverse long distances via the Internet in order to retrieve the data object. display and manipulation of large amounts of data, espeThe Internet is said to provide an ''open distributed cially image data, over the Internet to a small, and relatively hypermedia system." It is an "open" system since Internet 2s cheap, client computer, 100 implements a standard protocol that each of the conDue to the relatively low bandwidth of the Internet (as necting computer systems, 106,130,120,132 and 134 must compared to today's large data objects) and the relatively implement (TCP/IP), It is a "hypermedian system because it amount of processing power at 'lient is able to handle hypermedia documents as described above Computers, many tasks performed via standards such as the HTTP and HTML hypertext at 'lient computers On the transmission and mark up standards, respectively. Further, it 30 cannot be performed by Internet. Also, while the present open distributed hypermeis a "distributed" system because data objects that are dia system on the Internet allows users to locate and retrieve imbedded within a document may be located on many of the objects it allows ~ users very little, if any, interaction with computer systems connected to the I ~~n example of ~ data~ ~ ~ ~ , an open distributed hypermedia system is the so-called 35 these data objects. Users are limited to traditional hypertext and hypermedia forms of selecting linked data objects for ''world-wide implemented on the I~~~~~~~ and disand ~ cussed in papers such as the B ~ ~ reference~ given ~ ~ ~retrieval L launching viewers or other forms of external ~ software to have the data objects presented in a comprehenabove. sible way. The open distributed hypermedia system provided by the Thus, it is desirable to have a system that allows a user at Internet allows users to easily access and retrieve different 40 a small client computer connected to the Internet to locate, data objects located in remote geographic locations on the retrieve and manipulate data objects when the data objects Internet. However, this open distributed hypermedia system are bandwidth-intensive and compute-intensive. Further, it as it currently exists has s~ortcomings that today's large in is desirable to allow a user to manipulate data objects in an data objects are limited largely by bandwidth constraints in the various communication links in the ~ ~and local- 45~ Fteractive way to provide the user with a better understandt ~ ~ ~ t 1% of information presented and to allow the user to ized networks, and by the limited processing power, or accomplish a wider variety of tasks. computing constraints, of small computer systems normally provided to most users. Large data objects are difficult to SUMMARY OF THE INVENTION update at frame rates fast enough (e.g., 30 frames per second) to achieve smooth animation. Moreover, the pro- 50 The present invention provides a method for running cessing power needed to perform the calculations to animate embedded program objects in a computer network environsuch images in real time does not exist on most ment. The method includes the steps of providing at least workstations, not to mention personal computers. Today's one client workstation and one network server coupled to the browsers and viewers are not capable of performing the network environment where the network environment is a computation necessary to generate and render new views of 55 distributed hypermedia environment; displaying, on the clithese large data objects in real time. ent workstation, a portion of a hypermedia document received over the network from the server, where the hyperFor example, the Internet's open distributed hypermedia media document includes an embedded controllable applisystem allows users to view still images. These images are cation; and interactively controlling the embedded controlsimple non-interactive two-dimensional images, similar to photographs. Much digital data available today exists in the 60 lable application from the client workstation via communication sent over the distributed hypermedia enviform of high-resolution multi-dimensional image data (e.g., three dimensional images) which is viewed on a computer ronment. while allowing the user to perform real time viewing transThe present invention allows a user at a client computer formations on the data in order for the user to better connected to a network to locate, retrieve and manipulate understand the data. 65 objects in an interactive way. The invention not only allows An example of such type of data is in medical imaging the user to use a hypermedia format to locate and retrieve program objects, but also allows the user to interact with an where advanced scanning devices, such as Magnetic Reso- 5,838,906 7 8 application program located at a remote computer. InterproFIG. 10 is a diagram of the various processes and data paths in the present invention. cess communication between the hypermedia browser and the embedded application program is ongoing after the DETAILED DESCRIPTION OF A PREFERRED program object has been launched. The user is able to use a EMBODIMENT vast amount of computing power beyond that which is s 375 pages of Source code on 4 microfiche Appendices A contained in the user's client computer. and B are provided to this specification. The source code 1n one application, high resolution three dimensional should be consulted to provide details of a specific embodiimages are processed in a distributed manner by several ment of the invention in conjunction with the discussion of computers located remotely from the user's client computer. the ,,,tines in this specification, ~h~ source code in A ~ This amounts to providing parallel distributed processing for 10 dix A includes NCSA ~~~~i~ version 2.4 source code along tasks such as volume rendering or three dimensional image with modifications to the source code to implement the transformation and display. Also, the user is able to rotate, present invention, Appendix B includes source code implescale and otherwise reposition the viewpoint with respect to ,,,ting an application program interface, me source code these images without exiting the hypermedia browser softis ,,itten in the computer language to run on an ware. The control and interaction of viewing the image may 15 x-window be provided within the same window that the browser is FIG, 3 is an illustration of a computer system suitable for using assuming the environment is a "windowing" environuse with the present invention, FIG, 3 depicts but one merit. The viewing transformation and volume rendering example of many possible computer types or configurations calculations may be performed by remote distributed comcapable of being used with the present invention, FIG, 3 puter systems. 20 shows computer system 150 including display device 153, Once an image representing a new viewpoint is computed display screen 155, cabinet 157, keyboard 159 and mouse the frame image is transmitted over the network to the user's 161. Mouse 161 and keyboard 159 are ''user input devices." client computer where it is displayed at a designated position Other examples of user input devices are a touch screen, within a hypermedia document. By transmitting only light pen, track ball, data glove, etc. enough information to update the image, the need for a high 25 Mouse 161 may have one or more buttons such as buttons bandwidth data connection is reduced. Compression can be 163 shown in FIG, 3, cabinet 157 houses familiar computer used to further reduce the bandwidth requirements for data components such as disk drives, a processor, storage means, transmission. etc. As used in this specification "storage means" includes Other applications of the invention are possible. For 30 any storage device used in connection with a computer example, the user can operate a spreadsheet program that is system such as disk drives, magnetic tape, solid state being executed by one or more other computer systems memory, bubble memory, etc. Cabinet 157 may include connected via the network to the user's client computer. additional hardware such as inputloutput (110) interface Once the spreadsheet program has calculated results, the cards for connecting computer system 150 to external results may be sent over the network to the user's client 35 devices such as an optical character reader, external storage computer for display to the user. In this way, computer devices, other computers or additional devices. systems located remotely on the network can be used to FIG. 4 is an illustration of basic subsystems in computer provide the computing power that may be required for system 150 of FIG. 3. In FIG. 4, subsystems are represented certain tasks and to reduce the data bandwidth by only by blocks such as central processor 180, system memory 181 transmitting results of the computations. 40 consisting of random access memory (RAM) andlor readonly memory (ROM), display adapter 182, monitor 183 Still other applications of the present invention are (equivalent to display device 153 of FIG. 3), etc. The possible, as disclosed in the specification, below. subsystems are interconnected via a system bus 184. AddiBRIEF DESCRIPTION OF THE DRAWINGS tional subsvstems such as a vrinter, keyboard, fixed disk and others are shown. ~ e r i ~ h e r aand inp;t/output (110) devices is FIG. 1 illustrates examples of hypertext and hypermedia 45 can be connected to the computer system by, for example documents and links; serial port 185. For example, serial port 185 can be used to FIG. 2 is an example of a computer network; connect the computer system to a modem for connection to FIG. 3 is an illustration of a computer system suitable for a network or serial port 185 can be used to interface with a use with the present invention; mouse input device. The interconnection via system bus 184 allows central processor 180 to communicate with each FIG. 4 is an illustration of basic subsystems in the subsystem and to control the execution of instructions from computer system of FIG. 3; 181 Or fixed disk lS6, the exchange and FIG. 5 is an illustration of an embodiment of the invention information between subsystems. Other arrangements of using a client computer, server computer and a network; 55 subsystems and interconnections are possible. FIG. 6 shows another embodiment of the present invenFIG. 5 is an illustration of an embodiment of the invention tion using additional computers on the network; using a client computer, server computer and a network. FIG. 7Ais a flowchart of some of the functionality within In FIG. 5, client computer 200 communicates with server the HTMLparse.~ file; computer 204 via network 206. Both client computer 200 FIG. 7B is a flowchart of some of the functionality within 60 and server computer 204 use a network protocol layer to the HTMLf0rmat.c file; communicate with network 206. In a preferred embodiment, FIG. 8Ais a flowchart of some of the functionality within network 206 is the Internet and the network protocol layers the HTMLwidget.~file; are TCP/IP. Other networks and network protocols may be FIG. 8B is a flowchart of some of the functionality within used. For ease of illustration, additional hardware and softthe HTML.c file; 65 ware layers are not shown in FIG. 5. FIG. 9 is a screen display generated in accordance with Client computer 200 includes processes, such as browser the present invention; and client 208 and application client 210. In a preferred ~ ~ 5,838,906 9 10 embodiment, application client 210 is resident within client to display the multidimensional embryo data on the display screen to a user of the client computer 200. The user is then computer 200 prior to browser client 208's parsing of a able to interactively operate controls to recompute different hypermedia document as discussed below. In a preferred views for the image data. In a preferred embodiment, a embodiment application client 210 resides on the hard disk or RAM of client computer 200 and is loaded (if necessary) s control window is displayed within, or adjacent to, a window generated by browser client 208 that contains a display and executed when browser client 208 detects a link to of hypermedia document 212. An example of such display application client 210. The preferred embodiment uses the is discussed below in connection with FIG. 9. Thus, the user XEvent interprocess communication protocol to exchange is able to interactively manipulate a multidimensional image information between browser client 208 and application client 210 as described in more detail, below, Another lo object by means of the present invention. In order to make possibility is to install application client 210 as a "terminate 210 with created browser client 208, both the browser client and the appliand stay (TSR) program in an operating system cation client must be in communication with each other, as environment, such as X-Window, Thereby making access to shown by the arrow connecting the two within client comapplication client 210 much faster. puter 200. The manner of communication is through an Browser 'lient 208 is a process that a of 'lient application program interface (API), discussed below, computer 200 invokes in order to access various data Browser client 208 is a process, such as NCSA Mosaic, objects, such as hypermedia documents, on network 206. Cello, etc. Application client 210 is embodied in software Hypermedia document 212 shown within client computer presently under development called "VIS" and "Panel" 200 is an example of a hypermedia document, or object, that created by the Center for Knowledge Management at the a user has requested access to. In this example, hypermedia 20 University of California, San Francisco, as part of the Doyle document 212 has been retrieved from a server connected to Group's distributed hypermedia object embedding approach network 206 and has been loaded into, e.g., client computer described in "Integrated Control of Distributed Volume 200's RAM or other storage device. Visualization Through the World-Wide-Web," by C. Ang, D. Once 212 has been loaded Martin, M. Doyle; to be published in the Proceedings of 200, browser 208 parses " Visualization 1994, IEEE Press, Washington, D.C., October document 212. In parsing hypermedia document 212, 1994, browser client 208 detects links to data objects as discussed Versions and descriptions of software embodying the above in the Background of the IIIvention section. In FIG. 5 , present invention are generally available as hyperlinked data 212 an embedded program 30 Objects from the Visible Embryo Project's World Wide Web link at 214. Embedded program link 214 identifies applicadocument at the URL address ~~HTTP~llvisembryo, tion client 212 as an application to invoke. In this present ucsf,edui", example, the application, namely, application client 210, Another embodiment of the present invention uses an resides on the same computer as the browser client 208 that application server process executing on server computer 204 the user is executing to view the hypermedia document. 35 to assist in processing that may need to be performed by an Embedded program link 214 may include additional external program. For example, in FIG. 5, application server information, such as parameters, that tell application client 220 resides on server computer 204. Application server 220 210 how to proceed. For example, embedded program link works in communication with application client 210 residing 214 may include a specification as to a data object that on client computer 200. In a preferred embodiment, appliapplication client 210 is to retrieve and process. 40 cation server 220 is called VRServer, also a part of Doyle When browser 'lient 208 encounters embedded program Group's approach. Since server computer 204 is typically a link 214, it invokes application client 210 (optionally, with larger computer having more data processing capabilities parameters or other information) and application client 210 and larger storage capacity, application server 220 can executes perform processing in operate more and much faster, than application with the present invention. 45 client 210 in executing complicated and numerous instrucAn example of the type of processing that application tions, client 210 may perform is multidimensional image visual1, the present example where a multidimensional image ization. Note that application client 210 is in communication object representing medical data for an embryo is being with network 206 via the network protocol layer of client viewed, application server 220 could perform much of the computer 200. This means that application client 210 can 50 viewing transformation and volume rendering calculations make requests Over network 206 for data such as to allow a user to interactively view the embryo data at their multidimensional image objects. For example, application client computer display screen. In a preferred embodiment, client 210 may request an object, such as object 1 at 216, application client 210 receives signals from a user input located in server computer 204. Application client 210 may device at the user's client computer 200. An example of such make the request any means. Assuming 55 input would be to rotate the embryo image from a current 206 is the ~nternet, such a request would typically be made position to a new position from the user's point of view. This by using HTTP in response to a HTML-style link definition information is received by application client 210 and profor embedded program link 214. cessed to generate a command sent over network 206 to Assuming application client 210 has made a request for application server 220. Once application server 220 receives the data object at 216, server Process 218 ultimately receives 60 the information in the form of, e.g., a coordinate transforthe request. Server process 218 then retrieves data object mation for a new viewing position, application server 220 216 and transfers it over network 206 back to application performs the mathematical calculations to compute a new client 210. TO continue with the example of a multidimenview for the embryo image, Once the new view has been sional visualization application, data object 216 may be a computed, the image data for the new view is sent over three dimensional view of medical data for, e.g., an embryo. 65 network 206 to application client 210 so that application After application client 210 receives the multidimensional client 210 can update the viewing window currently displaying the embryo image. In a preferred embodiment, data object 216, application client 210 executes instructions 5,838,906 11 12 application server 220 computes a frame buffer of raster together with an associated image, within a hypermedia display data, e.g., pixel values, and transfers this frame document, allowing objects within the MetaMAP-processed buffer to application client 210. Techniques, such as data image to have URL addresses associated with them. When compression and delta encoding, can be used to compress a User clicks with a mouse upon an object within the the data before transmitting over network 206 to reduce the 5 M ~ ~ ~ M image, P - ~ ~ ~ M ~ P~ ~ ~ A the M ~ ~ client applicaA bandwidth requirement. tion relays the relevant URL back to the hypermedia browser It will be readily seen that application server 220 can application, which then retrieves the HTML file or hyperadvantageously use server computer 204's computing media object which correspon~sto that URL, resources to perform the viewing transformation much more The various processes in the system of the present invenquickly than could application client 210 executing on client 10 tion communicate through a custom API called Mosaic1 computer 200. Further, by only transmitting the updated External Application Program Interface MEAPI. The frame buffer containing a new view for the embryo image, MEAPI set of predefined messages includes those shown in the amount of data sent over network 206 is reduced. By Table I. using appropriate compression techniques, such as, e.g., MPEG (Motion Picture Experts Group) or JPEG (Joint TABLE I Photographic Experts Group), efficient use of network 206 is preserved. Message Function Message Name FIG. 6 shows yet another embodiment of the present Messages from server to client: invention. FIG. 6 is similar to FIG. 5, except that additional computers 222 and 224 are illustrated. Each additional 20 1. Server Update Done XtNrefreshNotify computer includes a process labeled "Application 2. Server Ready XtNpanelStartNotify (Distributed)." The distributed application performs a por3. Server Exiting XtNpanelExitNotify Messages from client to server: tion of the task that an application, such as application server 220 or application client 210, perform. In the present. 4. Area Shown XtNmapNotify 5. Area Hidden XtNunmapNotify example, tasks such as volume rendering may be broken up 25 6. Area Destroyed XtNexitNotify and easily performed among two or more computers. These computers can be remote from each other on network 206. Thus, several computers, such as server computer 204 and The messages in Table I are defined in the file protocol-1ib.h additional computers 222 and 224 can all work together to in ~ ~B, me functions of the MEUI are provided in ~ ~ ~ d i perform the task of computing a new viewpoint and frame 30 protocol-1ib.c of Appendix B. Thus, by using MEAPI a buffer for the embryo for the new orientation of the embryo server process communicates to a client application program image in the Present example. The coordination of the to let the client application know when the server has finished updating information, such as an image frame distributed processing can be performed at client computer 200 by application client 210, at server computer 204 by buffer, or pixmap (Message 1); when the server is ready to application server 220, or by any of the distributed appli- 35 start processing messages (Message 2) and when the server cations executing on additional computers, such as 222 and is exiting or stopping related to the server 224. In a preferred embodiment, distributed processing is application program, ~oordinatedby a Program called " V I S represented by For client to server communications, MEAPI provides for application client 210 in FIG. 6. the client informing the server when the image display Other applications of the invention are possible. For 40 window area is visible, when the area is hidden and when the example, the user can operate a spreadsheet program that is area is destroyed. Such information allows the server to being executed by one or more other computer systems decide whether to allocate computing resources for, e.g., connected via the network to the user's client computer. rendering and viewing transformation tasks where the server Once the spreadsheet program has calculated results, those is running an application program to generate new views of results may be sent over the network to the user's client 45 a multi dimensional object. Source code for MEAPI fun&computer for display within the hypermedia document on registermental functions such as handle-client-msg, the user's client computer. In this way, computer systems and send-clientclient, register~clientmsg~callback located remotely on the network can be used to provide the msg may be found in protocol-1ib.c as part of the source computing power that may be required for certain tasks and code in Appendix B. to reduce the data bandwidth required by only transmitting so Next, a discussion of the software processes that perform results of the computations. parsing of a hypermedia document and launching of an Another type of possible application of this invention application program is provided in connection with Table I1 would involve embedding a program which runs only on the and FIGS. 7A, 7B, 8A and 8B. client machine, but which provides the user with more Table 11, below, shows an example of an HTML tag functionality than exists in the hypermedia browser alone. ss format used by the present invention to embed a link to an An example of this is an embedded client application which application program within a hypermedia document. is capable of viewing and interacting with images which have been processed with Dr. Doyle's MetaMAP invention TABLE I1 (U.S. Pat. No. 4,847,604). This MetaMAP process uses <EMBED object-oriented color map processing to allow individual 60 TYPE = "type" color index ranges within paletted images to have object HREF = "href' WIDTH = width identities, and is useful for the creation of, for example, HEIGHT = height interactive picture atlases. It is a more efficient means for > defining irregular "hotspots" on images than the ISMAP function of the World Wide Web, which uses polygonal 65 outlines to define objects in images. A MetaMAP-capable As shown in Table 11, the EMBED tag includes TYPE, client-based image browser application can be embedded, HREF, WIDTH and HEIGHT elements. The TYPE element ~ ~ ~ 5,838,906 13 14 is a Multipurpose Internet Mail Extensions (MIME) type. scope of the invention. Additional considerations in implementing the method described by the flowchart in software Examples of values for the TYPE element are "application1 may dictate changes in the selection and order of steps. x-vis" or "video/mpeg". The type "application /x-visn indiSome considerations are event handling by interrupt driven, cates that an application named "x-vis" is to be used to handle the object at the URL specified by the HREF. Other s polled, or other schemes. A multi~rocessingor multitasking environment could allow steps to be executed "concurtypes are possible such as ~ a p p ~ i c a t i o n ~ x - i n v e n t o r ~ , rently." For ease of discussion the implementation of each ~~app~ication~postscript~the case where TYPE is etc, flowchart may be referred to as if implemented in a single ~ ~ a p p ~ ~ c a t ~ o nthis- means that the object at the URL lx v~s~ address is a three dimensional image object since the proThe modifications to NCSA Mosaic version 2.4 software gram "x-vis" is a data visualization tool designed to operate 10 files HTMLparse.~,HTMLformat.~,HTMLwidget.~ and on three dimensional image objects. However, any manner HTML.c will next be discussed, in turn. of program may be 'pecified the TYPE Returning to FIG, 7, it is assumed that a hypermedia that types such as a document has been obtained at a user's client computer and program, database program, processor, that a browser program executing on the client computer etc. may be used with the present invention. Accordingly, the IS displays the document and calls a first routine in the HTMobject reference by the HREF element would be, L p a r s e . ~file called "HTMLparse". This first routine, respectively, ;a spreadsheet object, database object, word HTMLparse, is entered at step 252 where a pointer to the processor document object, etc. start of the document portion is passed. Steps 254, 256 and On the other hand, TYPE values such as "videolmpeg", 258 represent a loop where the document is parsed or "imagelgif', "videolx-sgi-movie", etc. describe the type of 20 scanned for HTML tags or other symbols. While the file data that HREF specifies. This is useful where an external HTMLparse.~ includes routines to handle all possible tags application program, such as a video player, needs to know and symbols that may be encountered, FIG. 7A, for what format the data is in, or where the browser client needs simplicity, only illustrates the handling of EMBED tags. to determine which application to launch based on the data Assuming there is more text to parse, execution proceeds obtain the next format. Thus, the TYPE value can specify either an appli- 25 to step 256 where routines in HTMLparse.~ cation program or a data type. Other TYPE values are item (e.g., word, tag or symbol) from the document. At step possible. HREF specifies a URL address as discussed above 258 a check is made as to whether the current tag is the EMBED tag. If not, execution returns to step 254 where the for a data object. Where TYPE is "applicationlx-vis" the URL address specifies a multi-dimensional image object. next tag in the document is obtained. If, at step 258, it is Where TYPE is "videolmpeg" the URL address specifies a 30 determined that the tag is the EMBED tag, execution provideo object. ceeds to step 260 where an enumerated type is assigned for WIDTH and HEIGHT elements specify the width and the tag. Each occurrence of a valid EMBED tag specifies an embedded object. HTMLParse calls a routine "get-mark height dimensions, respectively, of a Distributed Hypermedia Object Embedding (DHOE) window to display an exterin HTMLparse.~ put sections of HTML document text to nal application object such as the three dimensional image 35 into a "markup" text data structure. Routine get-mark, in object or video object discussed above. turn, calls ParseMarkType to assign an enumerated type. The FIG. 7A is a flowchart describing some of the functionenumerated type is an identifier with a unique integer ality within the HTMLparse.~ of routines. The routines file associated with it that is used in later processing described perform the task of parsing a hypermedia below. in HTMLparse.~ document and detecting the EMBED tag. In a preferred 40 Once all of the hypermedia text in the text portion to be rouembodiment, the enhancements to include the EMBED tag displayed has been parsed, execution of HTMLparse.~ are made to an HTML library included in public domain tines terminates at step 262. NCSA Mosaic version 2.4. Note that much of the source FIG. 7B is a flowchart of routines in file HTMLf0rmat.c code in is pre-existing NCSA Mosaic code. Only those to process the enumerated type created for the EMBED tag portions of the source code that relate to the new function- 45 by routines in HTMLparse.~. the X-Window implemenIn ality discussed in this specification should be considered as tation of a preferred embodiment, the enumerated type is part of the invention. The new functionality is identifiable as processed as if it is a regular Motif/XT widget. For details being set off from the main body of source code by condion X-Window development see, e.g., "Xlib Programming tional compilation macros such as "#ifdef . . . #endif' as will Manual," "X Toolkit Intrinsics Programming Manual" and so "Motif Programming Manual" published by O'Reilly & be readily apparent to one of skill in the art. In general, the flowcharts in this specification illustrate Associates, Inc. HTMLformat is entered at step 270 where one or more software routines executing in a computer a pointer to the enumerated type to process is passed. system such as computer system 1of FIG. 1.The routines At step 272 the parameters of the structure are initialized may be implemented by any means as is known in the art. in preparation for inserting a DrawingArea widget on an For example, any number of computer programming ss HTML page. This includes providing values for the width languages, such as "C", Pascal, FORTRAN, assembly and height of a window on the display to contain an image, language, etc., may be used. Further, various programming position of the window, style, URL of the image object, etc. approaches such as procedural, object oriented or artificial Various codes are also added by routines in HTMLf0rmat.c intelligence techniques may be employed. (such as TriggerMarkChanges) to insert an internal repreThe steps of the flowcharts may be implemented by one 60 sentation of the HTML statement into an object list mainor more software routines, processes, subroutines, modules, tained internally by the browser. In the X-Window applicaetc. It will be apparent that each flowchart is illustrative of tion corresponding to the source code of Appendix A, the merely the broad logical flow of the method of the present browser is NCSA Mosaic version 2.4. invention and that steps may be added to, or taken away FIG. 8A is a flowchart for routine HTMLwidget. HTMfrom, the flowcharts without departing from the scope of the 65 Lwidget creates display data structures and launches an invention. Further, the order of execution of steps in the external application program to handle the data object flowcharts may be changed without departing from the specified by the URL in the EMBED tag. 5,838,906 16 15 HTMLwidget is entered at step 280 after HTMLformat has created the internal object representation of the EMBED tag. HTMLwidget is passed the internal object and performs its processing on the object. At step 282 the DrawingArea widget is created according to the type of the internal s representation, from HTMLformat, specified in the internal object. Similarly, at step 284 a pixmap area for backing storage is defined. At step 286 a check is made as to whether the type attribute of the object, i.e., the value for the TYPE element 10 of the EMBED tag, is an If step 290 is executed to launch a predetermined application. In a preferred embodiment an application is launched according to a user-defined list of application typelapplication pairs. The list is defined as a user-configurable XResource as described IS in "Xlib Programming Manual." An alternative embodiment could use the MIME database as the source of the list of application typelapplication pairs. The routine "vis-start external-application" in file HTMLf0rmat.c is invoked to match the application type and to identify the application to 20 launch. The external application is started as a child process of the current running process (Mosaic), and informed about the window ID of the ~ ~created in H T M L ~ ~ ~ ~ The external application is also passed information about the 25 ID of the pixmap, the data URL and dimensions, codes for communication such as popping-upliconifying, start notification, quit notification and refresh notification with external applications and DrawingArea refreshing are also added. Examples of such codes are (1) "setuplstart" in 30 vis-register-client and in HTMLwidgets.c; (2) messages a~~licatiOns" in vis-handle panelPmsg in HTMLwidgets.c; (3) "send messages to external applications" in V~S-send-msg in HTMLwidgets.~; "terminate external applications" in 35 (4) V~S-exit in HTMLwidgets.~ which calls V~S-send-msg to send a quit message; and (5) "respond to refresh msgs" in vis-redraw in HTMLwidgets.~. If, at step 286, the type is determined not to be an application object (e.g., a three dimensional image object in 40 the case of application "x-vis") a check is made at step 288 to determine if the type is a video object, If so, step 292 is executed to launch a video player application. Parameters are passed to the video player application to allow the player to display the video object within the ~ ~ within~ 45 the display of the portion of hypermedia document on the client's computer, Note that many other application objects types are possible as described above. FIG. 8B is a flowchart for routine HTML. Routine HTML takes care of "shutting down" the objects, data areas, etc. so that were set up launch the and display the data object. HTML is entered at step 300 and is called when the display or other processing of the EMBED tag has been completed. At step 302 the display window is removed and the memory areas for the pixmap and internal 55 object structure is made free for other uses. Completion of processing can be by user command or by computer control. The present invention allows a user to have interactive control over application objects such as three dimensional image objects and video objects. In a preferred embodiment, 60 controls are provided on the external applications' user interface. In the case of a VISIpanel application, a process, "panel" creates a graphical user interface (GUI) thru which the user interacts with the data. The application program, VIS, can be executing locally with the user's computer or 65 remotely on a server, or on one or more different computers, on the network. The application program updates pixmap " 3 data and transfers the pixmap data (frame image data) to a buffer to which the browser has access. The browser only needs to respond to the refresh request to copy the contents from the updated pixmap to the Drawin&ea. The Panel Process sends messages as ''Msg" sending performed by routines such asvis-send-msg andvis-handle ~anel-msg to send events (mousemove, keypress, etc.1 to the external application. FIG. 9 is a screen display of the invention showing an interactive application object (in this case a three dimensional image object) in a window within a browser window. In FIG, 9, the browser is NCSA Mosaic version 2.4. The processes VIS, Panel and VRServer work as discussed ,hove, FIG, 9 shows screen display 356 ~~~~i~ window 350 containing image window 352 and a portion of a panel window 354, ~~t~ that image window 352 is within ~~~~i~ window 350 while panel window 354 is external to ~~~~i~ window 350. Another possibility is to have panel window 354 within Mosaic window 350. By using the controls in panel window 354 the user is able to manipulate the image within image window 352 in real time do perform such operations as scaling, rotation, translation, color map selection, etc. In FIG. 9, two Mosaic windows are being used to show two different views of an embryo image. One of the views ~ irotated by six degrees from the other view so that r ~ ~ is ~ , ~ ~ ~ effect can be achieved when viewing the a images. Communication between Panel and VIS is via "Tooltalk" described in, e.g., "Tooltalk 1.1.1 Reference from Sunsoft. FIG. 10 is an illustration of the processes VIS, Panel and VRServer discussed above. As shown in FIG. 10, the browser process, Mosaic, communicates with the Panel process via inter-client communication mechanisms such as provided in the X-Window environment, The Panel process comm,nicates with the VIS process through a communications protocol (ToolTalk, in the preferred embodiment) to exchange visualization command messages and image data, The image data is computed by one or more copies of a process called VRServer that may be executing on remote computers on the network. VRServer processes respond to requests such as rendering requests to generate image segments. The image segments are sent to VIS and combined into a pixmap, or frame image, by VIS. The frame image is then transferred to the Mosaic screen via communications between VIS, Panel and Mosaic. Afurther description of the data transfer may be found in the Paper ''Integrated Control of Distributed iVolume Visualization Through the World~ ~ & ~ ~ Wide-Web," referenced above. In the the has been described with reference to a specific exemplary embodiment thereof. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the appended claims, For example, various programming languages and techniques can be used to implement the disclosed invention. Also, the specific logic presented to accomplish tasks within the present invention may be modified without departing from the scope of the invention. Many such changes or modifications will be readily apparent to one of ordinary skill in the art. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense, the invention being limited only by the provided claims. What is claimed is: 1. A method for running an application program in a computer network environment, comprising: providing at least one client workstation and one network server coupled to said network environment, wherein said network environment is a distributed hypermedia environment; 5,838,906 17 18 executing, at said client workstation, a browser a first distributed hypermedia document to identify application, that parses a first distributed hypermedia text formats included in said distributed hypermedia document to identify text formats included in said document and to respond to predetermined text fordistributed hypermedia document and for responding to mats to initiate processes specified by said text predetermined text formats to initiate processing speci- 5 formats; fied by said text formats; utilizing said browser to computer readable program code for causing said client display, on said client workstation, at least a portion of workstation to utilize said browser to display, on said neta first hypermedia document received over said client workstation, at least a portion of a first hyperwork from said server, wherein the portion of said first media document received over said network from hypermedia document is displayed within a first lo said server, wherein the portion of said first hyperbrowser-controlled window on said client workstation, wherein said first distributed hypermedia document media document is displayed within a first browserincludes an embed text format, located at a first location controlled window on said client workstation, in said first distributed hypermedia document, that wherein said first distributed hypermedia document specifies the location of at least a portion of an object includes an embed text format, located at a first 15 external to the first distributed hypermedia document, location in said first distributed hypermedia wherein said object has type information associated document, that specifies the location of at least a with it utilized by said browser to identify and locate an portion of an object external to the first distributed executable application external to the first distributed hypermedia document, wherein said object has type hypermedia document, and wherein said embed text 20 information associated with it utilized by said format is parsed by said browser to automatically browser to identify and locate an executable appliinvoke said executable application to execute on said cation external to the first distributed hypermedia client workstation in order to display said object and document, and wherein said embed text format is enable interactive processing of said object within a parsed by said browser to automatically invoke said display area created at said first location within the 25 executable application to execute on said client portion of said first distributed hypermedia document workstation in order to display said object and enable being displayed in said first browser-controlled wininteractive processing of said object within a display dow. area created at said first location within the portion of 2. The method of claim 1,wherein said executable applisaid first distributed hypermedia document being cation is a controllable application and further comprising 30 displayed in said first browser-controlled window. the step of: 7. The computer program product of claim 6, wherein said interactively controlling said controllable application on executable application is a controllable application and said client workstation via inter-process communicafurther comprising: tions between said browser and said controllable applicomputer readable program code for causing said client cation. 35 workstation to interactively control said controllable 3. The method of claim 2, wherein the communications to application on said client workstation via inter-process interactively control said controllable application continue communications between said browser and said conto be exchanged between the controllable application and trollable application. the browser even after the controllable application program 8. The computer program product of claim 7, wherein the has been launched. 4, The method of claim 3, wherein additional instructions 40 communications to interactively control said controllable be exchanged between the for controlling said controllable application reside on said network server, wherein said step of interactively controlling lable and the even after the program has been launched. said controllable application includes the following sub9. The computer program product of claim 8, wherein steps: 45 additional instructions for controlling said controllable issuing, from the client workstation, one or more comapplication reside on said network server, wherein said step mands to the network server; of interactively controlling said controllable application executing, on the network server, one or more instructions includes: in response to said commands; computer readable program code for causing said client sending information from said network server to said so workstation to issue, from the client workstation, one or client workstation in response to said executed instrucmore commands to the network server; tions; and processing said information at the client computer readable program code for causing said network workstation to interactively control said controllable server to execute one or more instructions in response application. to said commands; 5. The method of claim 4, wherein said additional instruc- 5s computer readable program code for causing said network tions for controlling said controllable application reside on sever to send information to said client workstation in said client workstation. response to said executed instructions; and 6. Acomputer program product for use in a system having at least one client workstation and one network server computer readable program code for causing said client coupled to said network environment, wherein said network 60 workstation to process said information at the client environment is a distributed hypermedia environment, the workstation to interactively control said controllable computer program product comprising: application. a computer usable medium having computer readable 10. The computer program product of claim 9, wherein program code physically embodied therein, said comsaid additional instructions for controlling said controllable puter program product further comprising: 65 application reside on said client workstation. computer readable program code for causing said client workstation to execute a browser application to parse

Disclaimer: Justia Dockets & Filings provides public litigation records from the federal appellate and district courts. These filings and docket sheets should not be considered findings of fact or liability, nor do they necessarily reflect the view of Justia.

Why Is My Information Online?