Freshub, Inc. et al v. Amazon.Com Inc. et al
Filing
1
COMPLAINT ( Filing fee $ 400 receipt number 0542-12267080), filed by Freshub, Inc., Freshub, Ltd.. (Attachments: # 1 Exhibit 1, # 2 Exhibit 2, # 3 Exhibit 3, # 4 Exhibit 4, # 5 Exhibit 5, # 6 Exhibit 6, # 7 Exhibit 7, # 8 Exhibit 8, # 9 Exhibit 9, # 10 Exhibit 10, # 11 Exhibit 11, # 12 Exhibit 12, # 13 Exhibit 13, # 14 Exhibit 14, # 15 Exhibit 15, # 16 Exhibit 16, # 17 Exhibit 17, # 18 Exhibit 18, # 19 Exhibit 19, # 20 Exhibit 20, # 21 Exhibit 21, # 22 Exhibit 22, # 23 Exhibit 23, # 24 Exhibit 24, # 25 Exhibit 25, # 26 Exhibit 26, # 27 Exhibit 27, # 28 Exhibit 28, # 29 Exhibit 29, # 30 Exhibit 30, # 31 Exhibit 31, # 32 Exhibit 32, # 33 Exhibit 33, # 34 Exhibit 34, # 35 Exhibit 35, # 36 Exhibit 36, # 37 Exhibit 37, # 38 Exhibit 38, # 39 Exhibit 39, # 40 Civil Cover Sheet)(Palmer, John)
EXHIBIT 14
US010239094B2
(12) United States Patent
(10) Patent No.: US 10 ,239,094 B2
Zsigmond et al.
(45 ) Date of Patent:
(54 ) SYSTEMS AND METHODS FOR SCANNING
( 56 )
INFORMATION FROM STORAGE AREA
References Cited
U .S . PATENT DOCUMENTS
CONTENTS
4 ,636 , 950 A
5 ,478 , 989 A
(71) Applicant: IKAN HOLDINGS LLC ,New York ,
NY (US)
( 72 ) Inventors : Fabio Zsigmond , Riverside , CT (US );
Sion Elie Douer , New York , NY (US );
Geraldo Yoshizawa, Sao Paulo (BR );
Frederico Wagner, New York , NY
Mar. 26 , 2019
1/ 1987 Caswell et al.
12/ 1995 Shepley
(Continued )
FOREIGN PATENT DOCUMENTS
100 17 503 A1
DE
(US)
10 /2001
OTHER PUBLICATIONS
( 73 ) Assignee : IKAN HOLDINGS LLC , New York ,
NY (US )
( * ) Notice : Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U .S .C . 154 (b ) by 0 days.
Supplementary European Search Report; Application No. 04713957.
1; dated Sep . 3 , 2008 .
(Continued )
(21) Appl. No.: 15/905,591
Primary Examiner — Andrew Joseph Rudy
(74 ) Attorney, Agent, or Firm — Knobbe , Martens, Olson
( 22 ) Filed : Feb . 26 , 2018
Prior Publication Data
(65 )
US 2018 /0178254 A1 Jun . 28 , 2018
(57)
& Bear LLP
The present invention is related to methods and systems for
collected item information for stored items. In one embodi
Related U .S. Application Data
(63) Continuation of application No . 15 /604 ,422, filed on
May 24, 2017 , now Pat. No . 9, 908, 153 , which is a
( Continued )
ment, a networked food storage system comprises a first
sensor configured to read information from item tags
coupled to items, wherein the items are stored or intended to
be stored in a storage unit. A data store is configured to store
food preferences for at least a first user. Instructions, stored
(51) Int. Ci.
G060 30/ 06
B07C 5 / 34
ABSTRACT
G06Q 40 /04
(52 ) U .S . CI.
in computer readable memory , are configured to : cause a
first user interface to be displayed to the first user via which
( 2012.01)
( 2006 .01)
the first user can request a meal suggestion ; retrieve pref
( 2012 . 01)
CPC ........... B07C 5 /34 (2013 .01 ); G06Q 30 /0601
(2013 .01 ); G06Q 40 /04 (2013.01)
( 58 ) Field of Classification Search
CPC ........ GO6Q 40/10 ; G06Q 40 /02 ; GO6Q 40 /00 ;
erence information for the first user from computer readable
memory ; retrieve information read from at least a first item
tag ; and provide a meal suggestion based at least in part on
preference information for the first user and item tag infor
mation .
G06Q 30 /04 ; G06Q 20 / 10 ; G06Q 30 /02 ;
(Continued )
24 Claims, 8 Drawing Sheets
DIGITIZE AND RECORD
SPOKEN ORDER
804
TRANSMIT ORDER TO
REMOTE SERVER
- 306
PERFORM YOICE
RECOGNITION
808
DISPLAY TEXT VERSION
OF VOICE ORDER TO
USER FOR VERIFICATION
810
TRANSMIT ORDER TO
ONE OR MORE
PROVIDERS
812
RECEIVE QUOTE AND
PROVIDE QUOTES TO
USER
2814
RECEIVE ORDER
AUTHORIZATION TO
USER
816
PLACE ORDER
US 10 ,Page ,2094 B2
239
Related U . S . Application Data
6 ,663 ,004 B2
continuation of application No . 11 /301, 291, filed on
Dec . 12 , 2005 , now Pat. No. 9 ,821 ,344.
(60 ) Provisional application No.60 /635 ,122, filed on Dec .
10 , 2004.
(58 ) Field of Classification Search
CPC .... H04W 4 /02; H04W 8 / 18 ; H04N 2007 / 145;
H04M 2207 / 18 ; H04M 1 /72522; H04M
1/ 72527; H04M 1 /7253 ; H04M 2250 /52 ;
H04M 3 /5333 ; H04M 1 /6505 ; HO4L
29 /08108
USPC . .... 705/39, 28 – 30 ; 455/414 .1 , 556 .1 , 150 .1 ,
455 /405 , 406 ; 709 /204, 217 – 219 ;
704 /231, 258 , 266 , 268 , 270 .1 ; 379 /88 . 1 ,
379/88 .16 , 114 .01, 219
See application file for complete search history .
(56 )
References Cited
U .S . PATENT DOCUMENTS
5 ,489 ,773
5 ,532,928
5 ,641,039
5 ,664 , 110
5 ,699 ,525
5 ,712 ,989
5 ,841, 115
A
A
A
A
A
5 ,914 ,472
5 ,933,829
5 ,940 ,074
5 ,957,695
5 , 960 ,402
5 , 965, 858
5 , 969,317
5 ,978 ,773
5 ,979 ,240
5 ,979, 757
5 ,983, 199
5 ,987 ,440
5 ,995, 105
6 ,024 ,281
6 ,031,621
6 ,034 ,660
6 ,047, 843
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
6 / 1999
8 / 1999
8 / 1999
9 / 1999
9/ 1999
10 / 1999
10 / 1999
11/ 1999
11/ 1999
11/ 1999
11/ 1999
11/ 1999
11/ 1999
2 /2000
2 /2000
3 /2000
4 / 2000
5 ,901, 203 A *
6 ,078 ,793
6 ,085 , 112
6 ,089 ,453
6 , 131,744
6 , 205 ,204
6 ,314 ,457
6 , 367,518
6 , 378 ,721
6 , 386 , 386
6 , 425 ,487
6 ,434 ,530
6 ,435 ,407
6 ,442, 952
6 ,446 , 979
6 ,457 ,177
6 ,473 ,740
6 ,529 ,881
A *
A *
A
A
B1 *
B1
2 / 1996
7 / 1996
6 / 1997
9 / 1997
6 / 2000
7 /2000
7 /2000
10 / 2000
3 / 2001
11/2001
B2 4 / 2002
B1 4 / 2002
B1 5 / 2002
B1 7 /2002
B1 8 / 2002
B1 8 / 2002
B2 9 / 2002
B1 9 /2002
B1
9 /2002
B2 10 / 2002
B2 * 3/ 2003
6 ,530 ,518 B1
379 / 218 .01
7 ,013 ,292
7 ,043 ,457
7 ,080 ,777
7 ,086 ,592
7 , 109 ,445
7 , 110 ,829
B1
B1
B2
B2
B2
B2
Hsu et al.
Hansen
Wagher et al.
Wagner et al.
Patterson et al.
Cunningham et al.
379 /88.01
3 /2006
5 / 2006
7 /2006
8 /2006
9 /2006
9 / 2006
7 , 114 ,656 Bi
10 /2006 Garver
7 ,133 ,739 B2
7 , 165 ,721 B2
11/ 2006 Williamson et al.
1/ 2007 Wagner et al.
7 ,206 ,387 B2 *
4 /2007 Jan .......................... H04M 3 /50
379 /219
7 ,224 ,779 B2 * 5 /2007 Finnigan ............... H04M 3 /533
379/211.02
7 ,229 ,015 B2 6 / 2007 Persky
7 ,233, 321 B1 * 6 / 2007 Larson ................ G06F 3 / 03545
345 / 156
7 ,328 , 159
7 ,328,842
7 ,344 ,063
7 ,373 ,317
7 , 386 , 100
B2
B22
B2
B1
B2 *
12 / 2007 Wagner et al.
2 /2008
/2008
3 /2008
5 /2008
6 / 2008
G06F 3 /014
340/4 . 1
Chang
Wagner et al .
Wagner et al.
Kopelman et al .
Michaelis ......... H04M 3 /42391
379/52
455 / 150 . 1
7 ,580 , 850
7 ,660,772
7 ,681,792
7 ,734, 729
7 ,949,529
B2
B2
B2
B2
B2 *
8 / 2009
2 /2010
3/ 2010
6 / 2010
5 / 2011
Lurie
Verkama
Wagner et al.
Du et al.
Weider .............. GO6F 17 /30864
704 /235
8, 131, 093 B2 * 3/ 2012 Kim ...........
H04N 19 / 119
382 /232
8 ,175 ,886 B2 * 5 /2012 Odinak ..... ........ G06Q 30 /0266
Duncan
Williams
George
Emmott et al.
Sloane et al.
Fiordelisi
Roh et al .
Schena et al.
Reams
Cockrill et al.
Morganstein ........ G06Q 20 / 206
704 / 251
379 / 88 . 22
H04M 1/575
455/414 . 1
8 ,447,602 B2 * 5 /2013 Bartosik ............... GO6F 17/273
704/ 10
8 ,705 ,705 B2 * 4/ 2014 Thenthiruperai . H04M 1 /72552
9 , 908, 153 B2 *
3/ 2018 Zsigmond
2002/0008145 A11 /2002 Walsh et al.
379 /88.04
............. B07C 5 / 34
2002/ 0016731 A1
3 /2002 Kupersmit et al.
2002 /0069137 A1
2002/0072923 A1
2002/ 0073029 Al
6 / 2002 Hiroshige et al.
6 /2002 Guidry
6 /2002 Cheaib et al.
2002 /0139848 AL
10 /2002 Catan
2002 /0143550 Al
10 /2002 Nakatsuyama
2002 /0059175 A1
2002/0120502 A1
2002/0161658
2002/0161704
2003 /0034391
2003 /0072488
2003/0164754
2003/0171944
5 /2002 Nakano
8 / 2002 Sakaguchi
A1 10 / 2002 Sussman
AL 10 /2002 Powar
A1 2 /2003 Wagner et al.
A1 4 / 2003 Barsness et al.
Al 9 /2003 Roseen
Al 9 / 2003 Fine et al.
2003 /0195818 Al
10 / 2003 Howell et al.
10 /2004 Wagner et al.
2004/0100380 A1
H04M 3 /493
455 / 150 . 1
8 ,320 ,958 B1 * 11/2012 Fujisaki
2004 /0199545 A1
3 /2003 Kirchilsky et al.
6 , 561,085 B15 / 2003 Durbin et al.
6 ,662,163 B1 * 12 /2003 Albayrak
Philyaw
Philyaw
Ogasawara
Meyer et al.
Morganstein ........... G06F 21/32
7 , 392 , 193 B2 * 6 /2008 Mault ............ GOOF 19 /00
704 /275
7 ,472 ,075 B2 * 12 /2008 Odinak .............. GO6Q 30 /0266
Satyamurti ............ HO1Q 11/12
375 / 301
Kleinschmidt . .... GO2B 27 /017
455 / 556 . 1
Kayser et al.
Pratt
Morganstein .......... GIOL 17/ 22
379 /67. 1
Schena et al.
6 ,640 ,214 B1 10 / 2003 Nambudiri et al.
6 ,643,624 B2 11/ 2003 Philipps et al.
B1
1/ 2005
B18 /2005
B2 12 /2005
B21 / 2006
B1 * 2 / 2006
7 ,303, 124 B2
5/ 1999 Morganstein ........... GIOL 17 / 22
Foladare et al.
Durst
Britt et al.
Redford et al.
Embutsu et al.
Suzuki et al .
Espy et al.
Hudetz et al.
Rix et al.
Tracy et al.
Kaneko
O 'Neil et al .
Reber et al.
Shepley
Binder
Kessenich et al.
Mecke
382 /293
6 ,845, 388
6 ,928 ,413
6 ,975 ,856
6 , 993, 507
7 ,006 ,605
7 ,246 ,745 B2 7/ 2007 Hudnut et al.
7 ,281,655 B2 10 / 2007 Wagner et al.
7 ,295 ,181 B2 * 11/2007 Alsio
Kumar
Stanczyk et al.
Dumont
Green et al.
12 / 1997 Embutsu et al .
1/ 1998 Johnson et al.
11/ 1998 Shepley
A
A
12 /2003 Wagner et al.
6 ,763 , 148 B1 * 7/ 2004 Sternberg .......... GO6F 17 /30247
5 /2004 Lindsay et al.
2004 /0260513 Al 12 /2004 Fitzpatrick et al.
2 /2005 Lo
2005 /0064900 A1 3/2005 Goris et al.
2005 /0041840 A1
US 10 ,Page ,3094 B2
239
References Cited
( 56 )
U . S . PATENT DOCUMENTS
2005 /0090233 AL
2005 /0154560 A1
2005 /0189412 A1
2006 /02 18057 A
2008 /0104596 A1
2008 /0133264 A1
2008 /0306872 AL
2009/0152348 A1
2014 /0214628
2015 /0379607
2016 /0267565
2017 /0293965
2018 /0005631
AL
Al
AL
Al
AL
4 / 2005 Chambers et al.
7 /2005
9 /2005
9/ 2006
5 /2008
6 / 2008
12 / 2008
6 / 2009
Fitzpatrick et al.
Hudnut et al.
Fitzpatrick et al.
Buonanno et al .
Wagner et al.
Felsher
Ostrowski et al.
7 / 2014 Argue et al.
12 / 2015 Pedley et al.
9 / 2016 Katcher
10 / 2017 Sasaki et al.
1/2018 Lee et al.
OTHER PUBLICATIONS
Dietician -Healthy Eating Services, “ Transforming How Dietitians
Offer Advice,” AirClic, 2001-2003 .
International Search Report, International Appln . No. PCT-IB04
01076 , May 25 , 2005 .
PCT International Search Report dated Feb . 24 , 2003.
Shop Smart, Eat Right Sample Grocery List, “Guide Yourself to
Healthier Eating," Beeline Shopper, 2001 - 2002 .
Shop Smart , Eat Right Sample Recipes, “ Great Meals Made
Easy,” Beeline Shopper, 2001 - 2002.
Shop Smart, Eat Right, “ Healthier, More NutritiousMeals,” Beeline
Shopper, 2001-2003 .
* cited by examiner
U . S . Patent
Mar. 26 , 2019
Sheet 1 of 8
US 10 ,239,094 B2
C102
NETWORK
INTERFACE
INTERFACE
INTERFACE
106
RECIPIES
CONTENTS
USER PREFS .
FIG . 1
Napane mamu oma
U . S . Patent
Mar. 26 , 2019
Sheet 2 of 8
US 10 ,239,094 B2
2208
- 210
SCANNER
SCREEN
LOCAL
SCANNER
- 202
REFRIGERATOR
COMPUTER
REFRIGERATOR
220
MICROPHONE /
CAMERA
REMOTE SYSTEM
Shelf 1
( scanner )
Shelf 1
(scanner )
Shelf 1
(scanner)
CABINET
FIG . 2
U . S . Patent
Mar. 26 , 2019
US 10 ,239,094 B2
Sheet 3 of 8
PRESENT MEAL
PARTICPANT
SELECTION MENUE
7 304
RETRIEVE CONTENT
INFORMATION FOR
STORAGE UNITS
7 306
RETRIEVE MEAL
PARTICPANT
PREFERENCES WITH
AVAILABLE ITEMS
308
VIEW SIDE - BY - SIDE
DISPLAY OF SEGMENT
CONTENTS
CONTENTS
7310
DISPLAY POTENTIAL
MEAL SELECTIONS
319
RECEIVE USER
SELECTION
314
DISPLAY DETAILED
MENU AND RECIPE
FOR SELECTED MEAL
FIG . 3
U . S . Patent
Mar. 26, 2019
Sheet 4 of 8
US 10 , 239, 094 B2
<402
RETRIEVE EXPIRATION
FOR AVAILABLE ITEMS
- 404
RETRIEVE PREFERENCE
INFORMATION FOR ITEM
ORDERING
7406
COMPARE
PREFERENCES WTH
EXPIRATION DATES
< 408
PRESENT ORDER LIST
410
RECEIVE USER
ORDER
SELECTION AND PLACE
FIG . 4
U . S . Patent
Mar. 26 , 2019
MEAL SELECTION
Sheet 5 of 8
US 10 ,239,094 B2
COURSES
SOUP
O Breakfast
Brunch
SALAD
Lunch
Dinner
MAIN COURSE
DESERT
o Late night snack
MEAL
PARTICIPANTS
? ???
DAD
BOBBY
JANE
O NANNY
GRANDMA
PROVIDE
RECOMMENDATIONS
GRANDPA
O GUEST
O NUMBER OF GUEST
FIG . 5
U . S . Patent
Mar. 26, 2019
Sheet 6 of 8
US 10 , 239, 094 B2
SALADS
TOMATO SALAD
GREEN SALAD
MAIN COURSE
O ROASTED CHICKEN
GRILLED SALMON
DESSERT
SORBET
CHOCOLATE CAKE
PROVIDE ADDITIONAL
PROVIDE RECIPES
SUGGESTIONS
FIG . 6
U . S . Patent
Mar. 26 , 2019
US 10 ,239,094 B2
Sheet 7 of 8
QUANTITY
MILK
ORANGE JUICE
O QUANTITY
YOGURT
O QUANTITY
O SODA
O QUANTITY
SUBMIT ORDER
FIG . 7
U . S . Patent
Mar. 26 , 2019
Sheet 8 of 8
US 10 ,239,094 B2
7802
DIGITIZE AND RECORD
SPOKEN ORDER
7804
TRANSMIT ORDER TO
REMOTE SERVER
7806
PERFORM VOICE
RECOGNITION
- 808
DISPLAY TEXT VERSION
OF VOICE ORDER TO
USER FOR VERIFICATION
- 810
TRANSMIT ORDER TO
ONE OR MORE
PROVIDERS
- 812
RECEIVE QUOTE AND
PROVIDE QUOTES TO
USER
- 814
RECEIVE ORDER
816
PLACE ORDER
AUTHORIZATION TO
USER
FIG . 8
US 10 ,239 ,094 B2
SYSTEMS AND METHODS FOR SCANNING
INFORMATION FROM STORAGE AREA
CONTENTS
INCORPORATION BY REFERENCE TO ANY
PRIORITY APPLICATIONS
suggestion ; retrieve preference information for the first user
from computer readable memory ; retrieve information read
from at least a first item tag; and provide a meal suggestion
based at least in part on preference information for the first
5 user and item tag information .
Still another embodiment provides a method of providing
meal suggestions , comprising : electronically receiving data
Any and all applications for which a foreign or domestic
priority claim is identified in the Application Data Sheet as
filed with the present application are hereby incorporated by 10
reference under 37 CFR 1 .57 .
BACKGROUND OF THE INVENTION
Field of the Invention
read , via a first sensor, from a first item , the data including
information related to the first item ' s ingredients ; identifying
a first user requesting a meal suggestion ; reading from
computer readable memory food preferences for at least the
first user ; and providing a meal suggestion based at least in
part on preference information for the first user and item tag
information .
Yet another example embodiment provides a networked
15 food storage system , comprising: a first sensor configured to
read expiration date information from item tags , wherein the
items are stored or intended to be stored in a storage unit; a
data store configured to store expiration date information ;
and instructions , stored in computer readable memory , con
The present invention is related to sensor systems and
processes, and in particular for sensing items stored in
storage units and performing related automated processes. 20 figured to : read expiration date information from the data
store ; determine if a first item has an expiration date within
Description of the Related Art
&a 10
first time window ; if the first item has an expiration date
no
Perishable items are often stored in refrigerated storage list.
units , such as refrigerators . Such perishable items often have
on 25
expiration dates or “ best used by dates” printed thereon . 25 Still another example embodiment provides an electronic
However, because perishable items are often densely packed system , comprising :memory configured to store a digitized
voice recording from a user ; and instructions, stored in
computer readable memory , configured to : read the digitized
voice recording ; identify a product identifier that corre
into a refrigerated storage unit, and because the expiration
dates are sometimes faintly or poorly printed , users often are
not able to conveniently monitor the expiration dates .
sponds to a first
in the
Hence, items are often retained in the refrigerated storage 30 recording ; present aitem referred to the user digitized voice
shopping list to
that includes at
unit past their expiration date , taking up valuable storage
least the first item .
space . In addition , because the user is often unaware ofwhen
an item has reached its expiration date , the user often does
Another example embodiment provides an electronic sys
tem , comprising : memory configured to store a digital
not replace the expired item in a timely fashion .
SUMMARY OF THE INVENTION
product image from a user ; and instructions, stored in
35 computer readable memory , configured to : perform object
The present invention is related to sensor systems and
processes, and in particular for sensing items stored in
storage units and performing related automated processes . 40
The following embodiments are intended to be illustrative
examples , and not to limit the scope of the invention .
One example embodiment provides a networked refrig
eration system comprising a first sensor configured to read
recognition with respect to the product image; select a
product identifier that corresponds to the product in the
image ; present a shopping list to the user that includes at
least the product.
BRIEF DESCRIPTION OF THE DRAWINGS
Embodiments of the present invention are described
herein with reference to the drawings summarized below .
herein the 451 drawings and
These
description are
information from item tags coupled to items, wherein the 45 to illustrate example the associated of the invention ,provided
and not
items are stored or intended to be stored in a refrigeration to limit the scope of embodiments .
the invention
unit ; a data store configured to store food preferences for a
FIG . 1 illustrates an example embodiment of an item
plurality of household members ; instructions, stored in
tracking and recommendation processing system .
interface to be displayed to the first user, the first user 30 sto FIG . system . an example embodiment of a networked
50 storage 2 illustrates
interface listing household members ; receive a user input
FIG . 3 illustrates an example meal suggestion process.
indicating which household members will participate in a
FIG . 4 illustrates an example order generation process.
first meal; retrieve preference information for at least a
FIG . 5 illustrates a first example meal suggestion request
portion of the household members thatwill participate in the user interface .
first meal; retrieve information read from at least a first item 55 FIG . 6 illustrates a second example meal suggestion
tag ; select at least a first recipe stored in computer readable
request user interface.
memory based at least in part on preference information for
FIG . 7 illustrates an example user interface for ordering
at least one meal participant and item tag information , and items.
provide the recipe to a first user.
FIG . 8 illustrates an example method for processing a
Another example embodiment provides a networked food 60 voice order.
computer readable memory , configured to : cause a first user
storage system , comprising: a first sensor configured to read
information from item tags coupled to items, wherein the
items are stored or intended to be stored in a storage unit ; a
data store configured to store food preferences for at least a
first user ; instructions, stored in computer readable memory, 65
configured to : cause a first user interface to be displayed to
the first user via which the first user can request a meal
DETAILED DESCRIPTION OF PREFERRED
EMBODIMENTS
Example systems and methods for scanning and obtaining
product information and processing such scanned informa
tion will be described herein .
US 10 ,239, 094 B2
Throughout the following description , the term “ Web
site ” is used to refer to a user-accessible network site that
implements the basic World Wide Web standards for the
coding and transmission of hypertextual documents . These
user prefers foods having a certain characteristic or that are
in a certain category, such as vegetarian , low fat , low
sodium , and /or kosher food products.
The scanner system can include or be coupled to one or
standards currently include HTML (the Hypertext Markup 5 more user input and /or output devices, such as a touch
Language ), HTTP (the Hypertext Transfer Protocol), Java ,
screen device , a non -touch screen display, an on / off switch ,
and XML. It should be understood that the term “ site ” is not
intended to imply a single geographic location , as a Web or
other network site can , for example , comprise multiple
a mechanical keyboard , voice recognition system , a voice
output system , a camera , a character recognition device , a
printer, and/ or other types of user interfaces . The system
geographically distributed computer systems that are appro - 10 optionally can include one or more function - specific hard
priately linked together. Furthermore , while the following
description relates to an embodiment utilizing the Internet
and related protocols, other networks , such as networked
keys and/ or soft keys displayed on a touch screen . The key
functions can be software programmable .
An example tag can include some or all of the following
interactive televisions, and other protocols may be used as
information : product code, expiration date, nutrition infor
well. In addition , unless otherwise indicated , the functions 15 mation , dietary classification information (e . g ., low sodium ,
described herein are preferably performed by executable
code and instructions running on one or more generalpurpose computers . For example , program code stored in
non - volatile and /or volatile memory can include one or more
low cholesterol, low carbohydrate , non - fat, peanut-free,
gluten - free , sugar - free , non - dairy , vegetarian ), ingredient
content, supplier , product, storage temperature range ,
dimensions ( height, length , width , weight ), cooking tem
instructions, which can optionally be straight-line code 20 perature and time, a unique item /tag identifier , etc . Option
and/ or organized as modules or objects configured to receive
ally , the tag can further include an identifier associated with
and process inputs, provide outputs , and to selectively store
an intended user.
data . However, the present invention can also be imple -
mented using special purpose computers , state machines,
If the tag is an RFID tag , the tag can include a RFID chip ,
with associated memory , an antenna , and optionally a bat
and / or hardwired electronic circuits . While certain example 25 tery . Optionally , an RFID tag can be a chipless tag , that does
processes are described herein , not all the process states
need to be performed , and the order of the process can be
varied . While several example embodiments are described
with reference to RFID tags and RFID scanners , other tags
not use integrated circuit technology to store information ,
but that does use materials (e . g . fiber ) that reflects a portion
of the scanner 's signal back to the scanner, which can then
be used as an identifier. By way of further example , if the tag
( e . g ., barcodes ) and scanners ( e . g ., a laser barcode scanner ), 30 is an RFID tag, the tag can be powered via a battery ,
electromagnetically by the RFID scanner , or otherwise .
can be used
In an example embodiment, a networked system includes
one or more scanning devices that can scan product infor -
mation . For example, a scanning system can optionally
As will be described below , the scanning system scans
item tags and optionally stores some or all of the scanned
information in a computer readable data store, such as a
include one or more of an optical barcode scanner , an RFID 35 database . Optionally , a user can then view some or all of the
(radio frequency identifier ) scanner, a character recognition
stored information via a display coupled to the scanning
scanner, a camera , and/or other scanner types . The scanning
system .
system can be configured to scan / read opticalmarkings such
By way of further example , the system can read or scan
as barcodes (printed or lasered onto the item packaging or product information located on or in item tags for items
the consumable item itself), RFID tags ( e .g ., a radio fre - 40 stored , or that are in the process of being stored , inside a
quency transponder ), solid state memory tags , and /or mag - refrigerator and /or stored in different household locations
netic tags. The foregoing data devices are generally refer - including shelves, cabinets, dry storage rooms, pantries ,
enced herein tags. By way of further example , the scanning
system can take electronic , digital pictures of product pack
automobile trunks, etc . The scanning can optionally be
performed utilizing a network of scanners and / or repeaters
aging or markings , or scan /read other information storage 45 in the house .
Thus, for example, one or more scanners, such as an RFID
In certain example embodiments , one or more scanning scanner can be placed in , or adjacent to a home refrigerator,
devices .
devices and / or related processing systems can be fixedly
as well as on or in proximity with food storage shelves
coupled or removably coupled to a utility device , such as a
and /or cabinets . The scanner location can be based on the
refrigerator , a conventional oven , a microwave oven , or 50 scanner range , wherein the scanner is placed within a
other device . Optionally, one or more scanning devices useable range of the items to be scanned , and the location
and / or related processing systems can be mounted to a wall ,
inside or outside of a cabinet, or on a stand .
The scanner system can include or be coupled to a local
can be further selected so that a barrier does not interfere
with the scanner' s ability to read item tags .
The scanner can be powered via a battery , AC power or
content database and / or can be connected to a remote 55 otherwise . The scanner can optionally be mounted in or to a
database via a network , such as the Internet. In addition , the
scanner system can be coupled to a suggestion application
refrigerator wall. The RFID scanner optionally includes an
antenna or coil, a transceiver, and a decoder. The transceiver
and database that stores food and / or meal preferences for
produces signals that are emitted by the antenna as radio
one ormore users . For example , a user can specify types of
signals . The radio signals are optionally used to activate the
foods, dishes , or other products that the user prefers or will 60 RF tag and to read and optionally write data to the tag . The
not eat, wherein the user specifications are stored in the
RFID scanner is optionally shielded by a substantially RF
database , optionally in association with an identifier ( e. g ., a
user ID and /or password ).
transparent cover to protect the sensor from dirt and /or
moisture .
The scanner can periodically ( e. g ., every 30 seconds ,
By way of further illustration , the user can specify that the
closing a refrigerator door), and /or other trigger (e.g., a
By way of illustration , the user can specify that the user
does not want to eat foods containing pork , poultry, beef, 65 every 15 minutes , every hour , or other regular or irregular
fish , gluten , wheat, eggs, nuts , strawberries, and /or seafood. time period ), in response to a user action (e . g ., opening or
US 10 ,239,094 B2
weight change detected via a weight sensor ) read items tags
able in the house, including the contents inside the refrig
for items stored within the refrigerator. Optionally , between
erator and on other various locations in the house .
scans , the scanner can enter into a power down /low power
consumption state to conserve power. Optionally, the user
The system can compare the available products or con
tents ( e .g ., those stored in the refrigerator and/ or in other
can manually cause an item tag to be scanned . For example, 5 storage locations ) with stored recipes, a serving size speci
if the tag is an optical barcode, the user can manually press fied by the user, and/or user preferences, such as: special
a scan button and /or pass the barcode in front of a barcode diets , including fat free diet, low carb , diet, vegetarian diet ,
Kosher food only , or other special diet.
scanner.
In addition to scanners that read tag information , one or
The system can suggest one or more recipes that more
more sensors can be provided that sense the presence of an
item without necessarily reading an item tag . For example ,
closely match the user ' s preferences, the available ingredi
ents , and the desired serving size , and the recipe can then be
one ormore pressure sensors can be coupled to one or more
prepared by or for the user. Because the system may not be
shelves to detect if the shelve has an item stored thereon .
aware of all the ingredients in the user ' s possession , the user
Other sensors , such as capacitor sensors , millimeter wave 15 can manually specify that the user does or does not have
certain ingredients available , and the foregoing information
sensors , or infrared sensors can be used as well.
A local and /or remote computer processing system can
can affect the recipes recommended to the user .
access the stored information , and based on some or all of
By way of further example , the system can present on a
display a list of recipe titles ( e . g., teriyaki chicken , fried
examples of which are described below . Optionally, a user 20 chicken , butter chicken , etc.), optionally with a brief listing
can specify how at least a portion of the information is to be of the main ingredients . The user can then select a listed
recipe, the complete recipe (including a complete ingredient
used .
Based on the scanned tag information , the system can listing, ingredient quantities, cooking temperature , etc.) is
the stored information , perform one or more of processes,
determine when products that are inside the refrigerator, on
then displayed , and the user can optionally print out the
dry products shelves , and/ or other selected locations , will 25 recipe . Because system may not be aware of all the ingre
expire and will remind or suggest to the user (through a user
interface , such as a touch screen display , a voice print, a
hardcopy printout, etc .) that the products be consumed on or
before the expiration date. The suggestion or reminder can
first be provided by the system a number of days before the 30
dients in the user 's possession , the user can manually specify
that the user does or does not have certain ingredients
available . By way of example , some or all of recipe infor
mation ( e . g ., cooking temperature and time) can be trans
mitted to a cooking device , such as an oven , which can use
expiration date , wherein the number of days can be a default the information in the cooking process .
value stored in computer readable memory and / or can be
By way of further example , as similarly discussed above ,
based on a value set by the user and stored in computer
the system optionally reminds the user to replace a product
readable memory. Optionally, in addition , the user can before the expiration date gathered through products tags .
specify that expiration notices are to be provided for certain 35 The system can optionally be configured by the user to
specified items and / or the user can specify that expiration
automatically trigger orders within a predetermined amount
notices are not to be provided for certain specified items
of time prior to the expiration date . Optionally, the order
Optionally, when a suggestion or reminder is provided , timing can be based in part on the time it takes to deliver the
the user can instruct the system not to provide further order once the order is placed , where , for example , the order
reminders regarding the expiring product or can instruct the 40 will be placed to ensure the order will arrive on , or shortly
system to provide another reminder at or after a specified
time or period (e . g ., a snooze instruction ).
The system will optionally suggest the reordering of
before the expiration date . In addition , the user can option
products based on one or more of expiration dates , con
charge is greater than a certain cost, the order should not yet
item inventory , and user preferences, such as : special diets,
been removed from a storage area (e . g ., a refrigerator ) by
as fat free diet, low carb . diet , vegetarian diet , Kosher food
comparing previous scanned information with current scan
information . If the system does not locate the item within a
ally specify , via a user interface , that for future orders if an
order value is below a certain amount and/ or a delivery
sumption patterns, delivery schedules , current household 45 be placed . Optionally , the system determines if an item has
only , or other special or selected diet.
As similarly described above , the suggestion or reminder
storage area within a predetermined amount of time (e . g., an
can first be provided a number of days before the expiration 50 hour, 4 hours, 8 hours , 12 hours, or 24 hours ), the system
date, wherein the number of days can be a default value
optionally infers that the item has been fully consumed , and
stored in computer readable memory and / or can be based on
adds the item to an order. Optionally , the predetermined
a value set by the user and stored in computer readable
amount of time is settable by a user.
reminders regarding the product or can instruct the system to
provide another reminder at a specified time or period .
Other consumer food -related information can also be
accessed and utilized by the system from computer memory.
memory . By way of example , the configuration information
includes the placement, depth , width and height dimensions
of the refrigerator and/ or freezer storage areas . The configu
ration information can also store information regarding
memory. Optionally, when a suggestion is provided , the user
The physical configuration of the refrigerator storage
can optionally instruct the system not to provide further 55 areas can be stored in computer local or remote physical
For example , recipes can be stored on a local and / or a remote 60 storage areas intended to store a particular type of food
information system . The recipes can include ingredients ,
quantities , and instruction of preparation . Optionally , the
system can store and /or calculate from stored information ,
recipe ingredient quantity information for different serving
sizes
Optionally, the recipes can be selected or downloaded
from another computer system based on the products avail-
product. For example , information (e .g., dimensions, special
temperature settings, etc .) regarding vegetable drawers, fruit
drawers , dairy product storage areas , and the like can be
stored . Shelve and drawer weight limitations (e . g ., recom
65 mended maximum weight) are optionally stored as well.
Certain configuration information can optionally be down
loaded over a network from a remote computer system (e. g.,
US 10 ,239 ,094 B2
from a Web site associated with the refrigerator or the
Optionally , a user interface is provided via which the user
tional storage.
The system can compare some or all of the foregoing
information with product and packaging information (such
profiles and preferences of the meal participants, as well as
the item tag , enables the system , and determine one or more
laundry company or a newspaper /magazine service pro
system provider ) based on , for example , the refrigerator
can indicate which type of meal ( e . g ., breakfast, brunch ,
lunch , dinner ), meal courses ( e. g ., salad , soup, appetizer ,
model or serial number.
In addition , the system can record which storage locations main course , dessert), and a food -type category (e . g ., fish ,
are occupied , and which locations are available for addi- 5 meat, etc .) the user wants . The system will then access the
the item inventory , and provide a corresponding meal rec
ommendation .
Optionally , a
trigger or
as volume and dimensions including depth , width and 10 through a systemuser caninterface to provide a command
om
user
communicate to the
height), that can be read from the item tag or retrieved from
central system or directly to a service provider, such as a
a local or remote database using an item identifier read from
vider, to
order
or to trigger a
ways to organize items on shelves and in drawers. Option servicingdirectly as the their servicepicking up or request for
, such
service of
delivering
ally , the system can provide a user with suggestions of 15 goods, and /or to suspend a service , such as to suspend
or
Stionsand
recommendations on how to organize the refrigerator
newspaper delivery for a specified period of time.
freezer by suggesting placement of products with respect to
Optionally, the system can further include a voice record
storage shelves and drawers .
By way of illustration , the system can compare product
ing device and/or the voice recording device can be inde
pendent of the system . The voice recording device can
dimensions and /or weight with the refrigerator or shelf 20 include digital memory ( e.g ., a disk drive, FLASH memory,
configuration and /or available locations, and using the com
parison results , suggest where a product or products should
etc .) or analog memory (e . g ., a cassette tape). A user can
record a product description on the device by activating a
be placed via a user interface. This information can be
record control, which can be in the form of a button , a
presented in a touch screen display or other user interface .
switch , a voice recognition command or other record trig
By way of example , the system can display a three dimen - 25 gering method . In an example embodiment, the device
sional representation of items already on storage shelves,
and then highlight items that are first to be removed as part
records a product description verbally provided by the user
and creates a digital file . The description can be provided as
of the reorganization process . Once the user has removed the
part of a verbally provided product order. Optionally , a
indicated items, the system displays where each of the
different file is created for each recorded production descrip
removed items are to be placed on the storage shelves . 30 tion . The device transfers the file to a remote database over
Optionally , the system provides a printout of stored items
a network such as the Internet . The transfer can be per
and the storage locations so that the user can easier locate
items and determine if the user has a certain items, and
withouthaving to physically search for items. Optionally , an
command , or on a scheduled basis .
formed substantially immediately, in response to a user
A remote computer processing system receives , stores,
" expired ” control is provided , which , when activated , causes 35 and accesses such files, which can be received from a
the system to display and / or printout a list of expired items
plurality of voice recording devices . The remote computer
and their locations.
Optionally , the system provides a user interface with a
search field , wherein the user can enter in the name of an
system then utilizes voice recognition software to translate
the voice recording files into text files . Optionally , the voice
recognition software increases recognition accuracy by,
item or an item type . The system then searches its database 40 when there are more than one potential word matches ,
to locate matches or near matches , and display search results placing more weight on words related to the types items
to the user . The search results can include a list of the
being purchased (produce, cereal, milk , etc . ) and the users
matches or near matches, the quantity of the matching and
past purchase history. The remote computer system can
near matching items, and their storage locations .
optionally match the spoken order with a SKU ( Stock
recommendations to the user. For example , the system can
a SKU database . The SKU database optionally stores SKUS
As similarly described above , optionally , the system , via 45 Keeping Unit, e .g ., an identifier, such as a unique numeric
a suggestion user interface , provides item suggestions and identifier associated with a specific product) retrieved from
suggest items to be consumed ( e . g ., via a meal recommen -
in association with a text description of the corresponding
dation including one or more of an appetizer , a main course , item . For example, if the user verbally ordered a cereal by
dessert, and/ or a drink ) based on item availability in the 50 name, the remote computer system translates the name into
user ' s house and /or on information input by one or more
text or other computer readable form , and matches the text
users (e .g., members of the household , one or more physi
with text stored in association with a SKU (or other iden
information can include food preferences, special diets ,
recognition and/or SKU matching and identification can be
ally, when a user wants the system to provide a meal
suggestion icon or button. The system then retrieves the
names of household members from its database , and pres-
sion of the order. For example , the remote computer system
web site, telephone, fax , short messaging system , or via
other communication techniques. Thus, for example , the
suggestions user interface , optionally in addition to an
verbally with one or more product suppliers and/ or with the
cians of a household member, etc . ). For example , the user tifier ) to locate the correct SKU . Optionally , the voice
modes , etc ., input by the user via a keyboard , a touch screen , 55 performed instead or additionally by the user' s local system .
The remote computer processing system optionally shares
voice recognition , or other input method .
Optionally, the system stores different profiles and pref - a text version (e . g ., the textual representation of the order
erences for different users in the same household . Option and /or the corresponding SKU ) and / or recorded verbal ver
recommendation , the user activates a physical or a soft meal 60 can share the text and / or verbal version of the order via a
ents the names of the household members on the meals
system can textually share an order that had been placed
" entire household ” control. In order to indicate who will be 65 user. For example , the remote system can add the item (s )
participating in the meal, the user can select one or more of
the household member names, or the “ entire household ” .
identified in the verbal order to the user 's shopping cart or
other shopping list, optionally in association with a digital
US 10 ,239,094 B2
10
file that includes the user 's spoken order. Users can then
preference information , user account information, etc .)
verify the text conversion of the spoken order. For example ,
across a network , such as the Internet, a local network ,
the user can click on an item in an order list for which
verification is requested , hear the user 's previously recorded
and /or other network . The interface 108 can optionally
include one or more of a digital subscriber line (DSL )
order, and determine if the written description in the shop- 5 interface , a Ti line interface, a satellite link , a cable hookup ,
ping list matches the user's spoken order. Optionally, the a dial-up modem , etc . The system 100 can further include
system uses individual consumer information , such as the wired and /or wireless digital and /or analog input/output
user' s location and the user's account information and
interfaces to receive and /or transmit information to
preference profile to better match the user' s spoken order more tag scanning sensors and / or presence sensors . one or
with a product. The system uses the product identification 10 The system 100 can include and/ or be coupled to one or
( e.g., SKU ) generated from the verbal order to request more user interface devices (e .g., a touch screen display , a
quotes from providers, provide quotes to the user, and to
printer, a keyboard , dedicated mechanical keys, voice
offer additional or alternative products to the user. The user recorder, etc .). For example , the system 100 can be coupled
can then place an order for the identified product.
to one or more user interface devices, sensors , scanners,
Optionally , the system does not provide voice recognition . 15 and /or other systems via a USB or FireWire bus, via a wired
Instead , an icon corresponding to the voice recording
local network , such an Ethernet network , and / or a wireless
pliant network , or a Bluetooth network .
appears in the user 's online shopping list (e.g., an electronic
network , such as an iEEE 802. 11b or IEEE 802. 11g com
online catalog .
quent visiting meal participants , user preferences ( e. g., food
preferences, item ordering preferences, payment prefer
shopping cart). Optionally , the icon has an associated
recording date displayed therewith . The user can click on the
The example database 106 includes recipes , content infor
icon , the system will play the recording to the user, and the 20 mation for one or more item storage units (eg .
user can manually key in the product name and /or search an
refrigerator ( s ), pantry , etc . ), household members and/ or fre
Optionally, in addition or instead , a picture of an item the
user wants to order can be taken using a camera coupled to
ences , etc . of household members and/ or frequent visitors ),
the system , a cell phone camera , a standalone camera , or 25 a system serial number, and the like. The database 106 ,
other camera . The picture can be transmitted by the system
optionally also includes dimensional and configuration
or otherwise to the remote system . The remote system can
information for one or more storage units (e.g ., available
fruit drawers, dairy product
drawer
locate a corresponding SKU from a SKU database, which 30 weight limitations, etc.). Forstorage areas, shelve and 106 can
example , the database
optionally stores image signature information and /or an store a mapping of product codes or SKUs to product names,
then perform image recognition to come up with a “ signa
ture” identifying the item . The remote system can then
image , in association with a corresponding SKU . The
remote system uses the product identification ( e. g ., SKU )
generated from the image to request quotes from providers,
provide quotes to the user, and to offer additional or alter - 35
native products to the user. The user can then place an order
for the identified product. While the foregoing verbal and
temperature settings , the existence of vegetable drawers ,
sizes, and packaging materials, as well as content informa
tion regarding items placed into storage units . Optionally,
the database 106 stores shopping lists , passwords and /or
unique identifiers for accessing remote databases and ser
vices.
The following is an example database schema that
photographic order processes can be used with a variety of includes fields for payment preferences, storage unit iden
item types, it can be particularly useful with respect to items tifiers for one or more storage units ( e .g ., one or more
that often do not have bar codes , such as fresh fruits ( e .g ., a 40 refrigerators ), storage unit shelf dimensions. food prefer
single apple ) or vegetables.
ences for one or more users (e . g ., food preferences for
Optionally, the system does not provide image recogni
tion . Instead , a thumbnail (or larger image size ) version of
breakfast, brunch , lunch , dinner, food avoidances, medical
or other dietary restrictions , etc .), and order generation /
trigger preferences .
( e.g., an electronic shopping cart ), displayed on the user's
local display . Optionally, the displayed picture has an asso
ciated image capture date displayed therewith . The user can
User Database Schema
the picture of the item , corresponding to the image captured
by the camera , appears in the user 's online shopping list 45
click on the picture , and the system will optionally display
a larger version of the picture . Thus, the picture acts as a 50 FIELD
reminder to the user. The user can then manually key in the
product name and /or search an online catalog and /or the
user ' s purchase history for the product to thereby add the
product to the order list.
Example embodiments will now be described with refer- 55
ence to the figures . Referring to FIG . 1 , an item tracking and
recommendation processing system 100 includes a central
processing unit 102 , computer readable memory 104 ( e. g .,
removable and/ or fixed : volatile memory , such as RAM ,
Alternative Form
of Payment
access and /or transmit data ( e.g., dimensionaldata regarding
a refrigerator or other item storage system , recipes, user
card , electronic fund transfer, check , place
on account, etc . ), and corresponding
Refrigeration Unit 1
Identifier
Refrigeration Unit 1
Capacity
Shelf 1
non - volatile memory, such as FLASH EEPROM , a hard disk 60 Shelf 2
drive , an optical drive , etc .), and a database 106 , which can Shelf n
be stored , in whole or in part in the computer readable User 1 Food
memory 104. A network interface 108 (e .g., a wireless or Preferences
wired network interface ) is provided to enable the item
tracking and recommendation processing system 100 to 65
DATA DESCRIPTION
Alternative form of payment (credit
payment information (credit card number,
and credit card expiration date , bank
account number, checking account number, etc .)
Unique identifier, such as a serial number,
associated with a first of the user 's disposal units
The capacity in units of measurement (gallons ,
liters, etc .) of Disposal Unit 1
Shelf 1 dimensions
Shelf 2 dimensions
Shelf n dimensions
Food preferences -Breakfast
Food avoidance - Breakfast
Food preferences -Brunch
Food avoidance - Brunch
Food preferences -Lunch
Food avoidance- Lunch
US 10 ,239 ,094 B2
12
11
FIG . 2 illustrates an example embodiment of a networked
storage system . Acomputer system 202 (which can be in the
- continued
FIELD
User n Food
Preferences
DATA DESCRIPTION
Food preferences -Dinner
Food avoidance-Dinner
Dietary restrictions
Food preferences-Breakfast
Food avoidance -Breakfast
Food preferences -Brunch
Food avoidance -Brunch
Food preferences -Lunch
form of system 100 illustrated in FIG . 1), is coupled to a
local scanner 204 , a screen 206 (e .g ., a touch screen that can
5 receive user inputs via finger and /or pen ), and optionally a
microphone and/ or camera 203 . The microphone is option
ally coupled to a digitizer which converts spoken language
into a digital representation . The camera can be a digital still
digitally and/or in analog form . The computer system 202,
and/or video camera that captures images and stores them
Food avoidance-Lunch
scanner, 204 , and/ or screen 206 optionally are physical, and
Food preferences- Dinner
optionally removably, mounted to a refrigerator 208 . The
computer system 202 , scanner, 204 , and /or screen 206 can
optionally be mounted on a wall, a stand , or other supporting
structure . The computer system 202 is optionally coupled to
remote scanners via networks 218 , 220 (e. g., WiFi net
works). The remote scanners are mounted to and /or config
Food avoidance -Dinner
Order Generation
Dietary restrictions
Generate/submit order preferences
Preference
The following is an example contents database schema
that stores item information for the contents of one or more
ured to scan one or more other storage units , such as
storage units . The information , or selected portions thereof,
refrigerator 210 and cabinet 212 . Optionally , a storage unit,
may have been scanned by one or more scanners from an 20 such as cabinet 212 , includes multiple shelves with corre
item tag, manually entered by the user, or retrieved from a
sponding scanners . Thus , optionally , one computer system
remote database using the item SKU or other identifier . The
can collect and store information scanned from items stored
example schema includes fields for a product code, ingre
in multiple storage units . Optionally , each storage unit
dients, calories, a diet category , a product name, dimensions, have a computer system that incorporates some or all ofcan
the
weight, an expiration date , a manufacturer , item location ,
elements of system 100 .
item storage temperature, and cooking guidelines .
Contents Database Schema
FIG . 3 illustrates an example meal suggestion process. At
state 302 , a user interface is displayed by an item tracking
FIELD
DATA DESCRIPTION
Product code
Ingredients
The item SKU
The item ingredients
Calories
The number of calories in the item or the number of calories per item unit
low sodium , low cholesterol, low carbohydrate , non - fat, peanut- free , gluten
Diet Category
free , sugar-free , non -dairy , vegetarian , etc .
Product name
The text name of the item
Dimensions
One or more of length , width , height, diameter, volume
Weight
Original pre -use weight
Expiration date
Item expiration or “best if used by” date
Manufacturer
The name of the manufacturer or other manufacturer identifier
Location Information Storage unit and /or shelf identifier of where item is stored
Storage Temperature The temperature or temperature range at which the item is to be stored
Cooking Information The temperature or temperature range at which the item is to be cooked ,
cooking time, cooking method (boiling, roasting, broiling , frying , microwave ,
etc . )
Recipe Database Schema
The following is an example recipe database schema. The
example schema includes fields for ingredients ( e . g ., ingre -
45
and recommendation processing system (ITRP ), via which a
user can indicate who is going to participate in a meal and
the meal type (breakfast, lunch , dinner ). Optionally, the
system can infer the meal type based on the time of day by
dient names , and amount per serving or per a specified 50 reading a real time clock ( e . g ., if the request is at 8 :00 , the
number of servings), cooking instructions, calories per serv -
system infers the meal is breakfast). At state 304, the ITRP
ing, and other nutritional information , such as the amount of
retrieves information regarding which items are currently
one or more of salt , carbohydrates , protein , fiber, fat, vita -
available ( e . g ., in one or more household storage units ). At
mins, etc . per serving and/or the % percentage of a recom
state 306 , the system retrieves food preference information
mended daily amount of the foregoing a serving will pro - 55 for selected meal participants . At state 308 , the system
matches user preferences and item availability for the meal
vide .
type. At state 310 the system displays information regarding
one or more meals that meet the user preferences and that
FIELD
DATA DESCRIPTION
can be prepared using the available food items. At state 312 ,
Ingredient 1
Ingredient 2
Ingredient n
Cooking Instructions
Calories/ serving
Salt, carbohydrates , protein ,
fiber, fat, vitamins/ serving
Name and amount per serving
Name and amount per serving
Name and amount per serving
Cooking method (s), time, temperature , etc .
60 the system receives a user selection of the displayed meals .
At state 314 , the system displays a detailed menu corre
sponding to the user selection .
FIG . 4 illustrates an example order generation process . At
state 402 , the system retrieves expiration date information
65 for items within one or more storage units ( e .g ., a refrig
erator and /or cabinet ). At state 404 , the system retrieves a
user preference regarding when an order should be gener
US 10 ,239, 094 B2
13
ated relative to at least item expiration dates (e .g ., a time
prior to an expiration date selected by the user wherein the
system is to ask the user if an item should be placed , also
14
user via a system display and/or via spoken instruction to
verbally record a shopping list or a portion thereof. The
recording can be intended as a reminder. The user may be
referred to as a time window ). At state 406 , the system
given a limited amount of time (e . g ., 5 seconds , 10 seconds ,
compares the user preference with the expiration date infor - 5 30 seconds, 1 minute ) to speak the order. The system can
mation and determines which items have expiration dates
that fall within the window . By way of example and not
inform the user of the limited time, and provide a clock or
displays the time remaining to record the order. Optionally ,
other indicator (e . g ., a countdown timer ) that continuously
limitation , the time window can be one day or less, greater
than one day , and/ or selected based upon a projected deliv
can be provided
ery date or time. At state 408 , the system presents to the user 10 the user via which the userwith an “ extend recording time"
control,
can cause system to provide
via a display device a list of items identified at state 406 and additional recording time. At the end thethe recording time,
of
asks the user to select which items are to be ordered . At state the user is optionally provided the opportunity to review the
410, the system receives the user selections and orders the recording ( e . g ., have the recording played back ), and to
selected items from a provider selected by the user.
FIG . 5 illustrates an example meal suggestion requestuser 15 delete and re -record the shopping list if the previous record
interface . The interface includes a Meal Selection section , ing was unsatisfactory .
For example , the user can speak the order to the micro
wherein the user can indicate for which meal the user wants
suggestions. In the illustrated example , the user can select
from : breakfast, brunch , lunch , dinner , and a late night
phone 203, illustrated in FIG . 2 . The system then digitizes
and records the spoken order in a file . At state 804 , the
snack . A courses section allows the user to indicate what 20 system transmits the digitized verbal order to a remote
courses suggestions are to be provided for . The course
system , such as remote system 214 . At state 806 , the remote
list is based on user set-up information , wherein the user
ticipating in the meal. In this example , a guest field is
version of the order to the user so that the user can verify if
selection optionally changes depending on the meal selec - system performs voice recognition on the order in order to
tion . For example , if the user selects breakfast, optionally, interpret the spoken order and converts the spoken order into
the course selection will not include soup or salad options. text. By way of example, the remote system can use gram
A "meal participants ” user interface enables the user to 25 mar constrained recognition and/ or natural language recog
indicate who is going to participate in the meal. The example nition . The voice recognition system optionally uses train
specifies household members and others that will be par -
ing. At state 808 , the remote system transmits the text
text version is an accurate interpretation of the spoken order.
to the system 202 or another user computer for display to the
user. Optionally , if the user determines that the order was not
correctly translated , the user can provide a corrected order
provided wherein the user can specify the number of guests 30 For example, the remote system can transmit the textversion
attending for which food preference information is not
available in the system database . The system can use the
guest number information when providing ingredient quan
( e . g ., via a keyboard , or by speaking the order again ) to the
recommendations” control is provided , which , when acti- 35 remote system .
vated will cause the system to provide one or more meal
At state 810 , the remote system transmits the translated
tities as part of recipes . Referring back to FIG . 5 . a “ provide
recommendations.
FIG . 6 illustrates another example meal suggestion
version of the order (e.g., the text version ) to one or more
providers ( e . g ., supermarkets , wholesale establishments ,
etc . ) in order to receive quotes . The remote system can
course suggestions in response to the user selections made 40 optionally match the translated version of the spoken order
via the interface illustrated in FIG . 5 , user food preferences, with a SKU retrieved from a SKU database , which stores
SKUs in association with a text description of the corre
and item availability . In this example, suggestions are pro
request user interface . The interface provides one or more
vided for a salad , a main course, and a dessert. Once the user sponding item , and transmit the SKU to the providers . At
state 812 , the remote system receives quotes from the
" provide recipe ” control, and the system will then provide 45 potential providers , and transmits the quotes to the user . At
the corresponding recipes, with the recipe portions scaled to state 814 , the user selects a provider and authorizes place
the number of meal participants. The user can optionally ment of the order. At state 816 , the remote system places the
activate a “ provide additional suggestions ” control, and the order with the selected provider.
system will provide additional suggestions for courses
It should be understood that certain variations and modi
selects the desired suggestions, the user can activate a
50 fications of this invention would suggest themselves to one
wherein the user has not yet made a selection .
FIG . 7 illustrates an example user interface for ordering of ordinary skill in the art. The scope of the present invention
items. The example list includes items whose expiration
is not to be limited by the illustrations or the foregoing
dates are within a predetermined window and / or items were
descriptions thereof.
present in a storage unit (e .g., a refrigerator), but which the
system has determined is no longer in the storage unit ( e . g ., 55 What is claimed is:
has been removed from the storage unit and not replaced
1. A computer -implemented method, the method compris
within a predetermined amount of time). The user can select ing:
which items are to be ordered and the quantity of the items.
receiving over a network at a first computer system , using
Optionally , the system defaults by placing check marks for
a network interface , a digitized spoken user order from
each item and the user unchecks the item if the user does not 60
a second computer system ,
want to order the item . Optionally , the system provides a
the second computer system comprising:
default quantity , which can be based on a previous user
FIG . 8 illustrates an example method for processing a
specified value and /or on consumption patterns.
voice order.At state 802, the user verbally provides an order. 65
By way of illustration , the user may press a “ record shop
ping list” control. In response, the system can prompt the
a microphone , a wireless network interface , and a
digitizer coupled to the microphone, wherein the
digitizer is configured to convert spoken words into
a digital representation , and the second computer
system is configured to transmit the digital represen
tation over the network to the first computer system ;
US 10 ,239,094 B2
15
second computer system , at least a portion of the
16
translating, using a translation module executed by the
17 . The computer -implemented method as defined in
claim 1, the method further comprising transmitting at least
a portion of the translated spoken order to an item provider.
18 . The computer - implemented meth
digitized spoken order to text;
matching the text, translated from the digitized spoken
order, to text descriptions of items, wherein the text 5 claim 1 , wherein the second computer system is configured
with a camera , the method further comprising:
descriptions of items are stored in a data store ;
at the
computer
based on at least an identified match , identifying a
freceiving camerafirst the secondsystem an image captured
by the
of
computer system ;
corresponding item ;
performing image recognition on an item in the received
adding the identified corresponding item to a list asso 10
image and identifying the item in the received image ;
ciated with the user ; and
and
enabling the list, including the identified item , to be
enabling the user to order the item in the received image .
displayed to the user via a user display .
19 . The computer -implemented method as defined in
2 . The computer - implemented method as defined in claim
aJ
i
claim 1, the method further comprising:
1 , wherein the second computer system further comprises a 15 receiving the digitized user order from the second com
voice output system .
puter system immediately when the user speaks the
3. The computer - implemented method as defined in claim
1, wherein enabling the list, including the identified item , to
order ;
4 . The computer -implemented method as defined in claim
1 , the method further comprising causing the list to be
recording the digitized user order from the user ;
utilizing grammar constrained recognition and / or natural
language recognition in translating at least a portion of
the digitized spoken order to text;
enabling the list to be provided to the user via a website ;
enabling the list to be provided to the user via a telephone ;
provided to the user via a telephone .
enabling the user to edit the list ; and
be provided via a user display, further comprises causing the
item identification to be provided to the second user device 20
via a website .
5 . The computer- implemented method as defined in claim 25
1, wherein enabling the list, including the identified item , to
be provided via a user display, further comprises causing the
list to be provided to the user via a short messaging system .
6 . The computer -implemented method as defined in claim
enabling the user to place an order for items on the list.
20. A voice processing system comprising:
a first computer;
a network interface;
non -transitory memory that stores instructions that when
1 , wherein identifying an item corresponding to the text 30
further comprises placing more weight on words in the
executed by the first computer cause the first system to
perform operations comprising :
digitized order related to the user 's past purchase history
7 . The computer- implemented method as defined in claim
receive over a network , using a network interface , a
when identifying the item corresponding to the text.
digitized spoken user order from a remote system ,
1 , wherein identifying an item corresponding to the text 35
further comprises utilizing a location of the user in identi
fying the item corresponding to the text.
8 . The computer -implemented method as defined in claim
the remote system comprising:
a microphone, a wireless network interface , and a
digitizer coupled to the microphone, wherein the
digitizer is configured to convert spoken words
into a digital representation , and the second com
1, wherein identifying an item corresponding to the text
further comprises utilizing preferences of the user in iden - 40
tifying the item corresponding to the text.
9 . The computer -implemented method as defined in claim
1, the method further comprising enabling the user to add a
reminder with respect to at least one item .
10 . The computer -implemented method as defined in 45
claim 1 , the method further comprising identifying a SKU
puter system is configured to transmit the digital
representation over the network to the first com
puter system ;
translate at least a portion of the digitized spoken order
to text;
match the text, translated from the digitized spoken
order , to text descriptions of items, wherein the text
descriptions of items are stored in a data store ;
based on at least an identified match , identify a corre
sponding item ;
that corresponds to the identified item .
11 . The computer- implemented method as defined in
claim 1 , wherein the digitized order comprises an item
include the identified corresponding item in a list
associated with the user; and
enable the list, including the identified item , to be
12 . The computer- implemented method as defined in
claim 1 , wherein the digitized order from the user is
displayed to the user via a user display.
recorded at least partly in response to a verbal user com 21. The voice processing system as defined in claim 20 ,
mand .
wherein identification of an item corresponding to the text
13 . The computer - implemented method as defined in 55 further comprises placing more weight on words in the
claim 1 , wherein the digitized user order is received imme digitized order related to the user's past purchase history
diately from the second computer system after the user when identifying the item corresponding to the text.
22 . The voice processing system as defined in claim 20 ,
speaks the order.
name.
50
14 . The computer -implemented method as defined in
wherein identification of an item corresponding to the text
claim 1 , wherein the second computer system comprises a 60 further comprises utilizing a location of the user in identi
refrigerator.
15 . The computer -implemented method as defined in
claim 1, wherein the second computer system is configured
to be wall mounted .
fying the item corresponding to the text.
23 . The voice processing system as defined in claim 20,
wherein identification of an item corresponding to the text
further comprises utilizing preferences of the user in iden
16 . The computer- implemented method as defined in 65 tifying the item corresponding to the text.
claim 1, the method further comprising associating a unique
24 . The voice processing system as defined in claim 20 ,
identifier with the second computer system .
the operations further comprising :
US 10 ,239 , 094 B2
record the digitized user order from the user;
utilize grammar constrained recognition and /or natural
language recognition in translating at least a portion of
the digitized spoken order to text;
enable the list to be provided to the user via a website ; 5
enable the list to be provided to the user via a telephone ;
enable the user to edit the list ; and
enable the user to place an order for items on the list .
Disclaimer: Justia Dockets & Filings provides public litigation records from the federal appellate and district courts. These filings and docket sheets should not be considered findings of fact or liability, nor do they necessarily reflect the view of Justia.
Why Is My Information Online?