Freshub, Inc. et al v. Amazon.Com Inc. et al

Filing 1

COMPLAINT ( Filing fee $ 400 receipt number 0542-12267080), filed by Freshub, Inc., Freshub, Ltd.. (Attachments: # 1 Exhibit 1, # 2 Exhibit 2, # 3 Exhibit 3, # 4 Exhibit 4, # 5 Exhibit 5, # 6 Exhibit 6, # 7 Exhibit 7, # 8 Exhibit 8, # 9 Exhibit 9, # 10 Exhibit 10, # 11 Exhibit 11, # 12 Exhibit 12, # 13 Exhibit 13, # 14 Exhibit 14, # 15 Exhibit 15, # 16 Exhibit 16, # 17 Exhibit 17, # 18 Exhibit 18, # 19 Exhibit 19, # 20 Exhibit 20, # 21 Exhibit 21, # 22 Exhibit 22, # 23 Exhibit 23, # 24 Exhibit 24, # 25 Exhibit 25, # 26 Exhibit 26, # 27 Exhibit 27, # 28 Exhibit 28, # 29 Exhibit 29, # 30 Exhibit 30, # 31 Exhibit 31, # 32 Exhibit 32, # 33 Exhibit 33, # 34 Exhibit 34, # 35 Exhibit 35, # 36 Exhibit 36, # 37 Exhibit 37, # 38 Exhibit 38, # 39 Exhibit 39, # 40 Civil Cover Sheet)(Palmer, John)

EXHIBIT 7

US009 165320B1

(12) United States Patent

(10) Patent No.:

Belvin

US 9,165,320 B1

(45) Date of Patent:

(54) AUTOMATIC ITEM SELECTION AND

7. E: ck

ORDERING BASED ON RECIPE

8,117,199 B2 *

wal -

8,146,799 B2

388

Oct. 20, 2015
se r 705/28

OWa.

2/2012 Ghani et al. .................. 707/734
4, 2012 K.

8,296,194 B2

8.412,579 B2

4/2013 Gonzalez

(73) Assignee: Amazon Technologies, Inc., Reno, NV

8.412,590 B2

4/2013 Elliott

(US)

(*) Notice:

10/2012

fetal.

(75) Inventor: Brandon W. Belvin, Kirkland, WA (US)

E.

8,706.572 B1 * 4/2014 Varadarajan ................. 705/27.2

2002fOO 10637 A1
2002fOO26363 A1

1/2002 Lieu et al.
2, 2002 D
, Jr.

Subject to any disclaimer, the term of this

2002.0029149 A1

3, 2002

patent is extended or adjusted under 35
U.S.C. 154(b) by 622 days.

2002/00953.45 A1
2003,0004831 A1
2003/0018536 A1*

7/2002 Panelli et al. ................... 705/26
1/2003 Owens
1/2003 Eggebraaten et al. .......... 705/26

2003/01 10088 A1*

(21) Appl. No.: 13/158,318

E.y, Jr

6/2003 Starmer et al. .................. 705/26

9, 2003 Fine et al.

2004.0056101 A1

Jun. 10, 2011

7/2003 Hing

2003/0171944 A1

(22) Filed:

2003/0130908 A1

3/2004 Barkan et al.

(51) Int. Cl.

(Continued)

G06O 30/00

(2012.01)

G06O 30/06

(2012.01)

(52) U.S. Cl
CPC

OTHER PUBLICATIONS

Rosen, Cheryl "Here's My Order-And Don't Forget the Milk Bar
Code Scanning Technology Eases Grocery Shopping. Speeds Com

G06O 30/0633 (2013.O1): G06O 30/0631

- ------ -

O

(

.01):

O

edy-Club

(58) Field of Classification Searc h
S
(2013.01)
ield of Classifi
CPC ....................... G06Q 30/0633; G06Q 30/0631
USPC ................ 705/26.1, 26.2, 26.25, 26.3, 26.35,

705/26.4, 26.41. 26.44, 26.5, 26.61. 26.64,
705/26.7, 26.8, 26.81, 26.82, 26.9, 27.1,

(57)

5,960,411 A * 9/1999 Hartman et al. ........... TO5/2681

6,118,820 A * 9/2000 Reitmeier et al. ....... 375/240.16
6,370,513 B1 * 4/2002 Kolawa et al. ............... 705,733
6,434,530 B1 ck s2002 Sloane et al.
f
58.

ABSTRACT

user can quickly and conveniently purchase the items. In
other instances, a purchase transaction for the selected items

Ben . . . . . . . . . . . . . . . . . . . . . . . . . . 2. 7.

can be automatically initiated following the selection of the

7/2006 Bezos et al. ................. 705/26.7
9/2006 Silva et al. ................... TO5/26.8
4/2009 Brown et al.

1tems.

19 Claims, 5 Drawing Sheets

*
Receive recipe data

? 202

E.

Arieldata Laos
referenced in catalog
Receive reops selection

/ 206

s

Select o,Gims from Ly-208

<> 210
Siirit Ordia

-y

2000.

Systems and methods enable the selection of one or more
recipes by a user. Items available for purchase from an elec
tronic marketplace can be selected based on the recipe selec
tion. In some instances, the selected items can be automati
cally placed or added to a virtual shopping cart such that a

U.S. PATENT DOCUMENTS

R: ck

13,

Stockton LLP

References Cited

$33.

Nov.

Primary Examiner Naeem Haq

705/27.2

7,082,407 B1 *
7,103,566 B2 *
7,523.302 B1

InformationWeek,

(74) Attorney, Agent, or Firm — Kilpatrick Townsend &

See application file for complete search history.
(56)

Reservations.”.

Retrieved from Dialog File: 275, Acci: 02449434.*
Continued
(Continued)

Proviceigering r212
--

Receive order confirmation u?"
- Submit order for processing / 216
Receive recipe feedback from Ly-213

US 9,165,320 B1
Page 2

(56)

References Cited
U.S. PATENT DOCUMENTS

2005/0038719
2006, OO64347
2006/0178947
2006/0224401
2007/0O84919
2008, OO86318
2008/O1998.35
2008.O3OO993
2009, OO37288
2009, O144081

A1
A1
A1
A1
A1
A1
A1
A1
A1
A1

2, 2005
3, 2006
8, 2006
10, 2006
4, 2007
4, 2008
8, 2008
12, 2008
2, 2009
6, 2009

Young et al.
Robinson et al.
Zsigmond et al.
Hunt
Petrovich

Gilley et al.
Yeager
Rozenblatt
Christensen
Harlan

2009, 0234712
2010, O292998
2011/0055044
2012fO253828
2013/0052616

A1
A1
A1
A1
A1

9, 2009
11, 2010
3, 2011
10, 2012
2/2013

Kolawa et al.
Bodlaender et al.
Wied1
Bellacicco
Silverstein et al. ........... 434f127

OTHER PUBLICATIONS

U.S. Appl. No. 13/158,323, filed Jun. 10, 2011.
U.S. Appl. No. 14/034,840, filed Sep. 24, 2013.
U.S. Appl. No. 13/347,444, filed Jan. 10, 2012, Titled: “Portioned
Ingredient Recipe Completion'.

* cited by examiner

U.S. Patent

Oct. 20, 2015

Sheet 1 of 5

US 9,165,320 B1

S. OO
Application
104

Server

106

116

120

U.S. Patent

Oct. 20, 2015

US 9,165,320 B1

Sheet 2 of 5

200

U-vReceive recipe data
ASSOCiate itemS referenced in

recipe data with items
referenced in Catalog data

Receive recipe Selection

Select One Or more itemS from

204

206

208

Catalog data

210

Subrit Order?

Provide information regarding

212

Selected itemS

ReCeive Order COnfirmation

Submit Order for processing

Receive recipe feedback from
LSer

FIG. 2

214

216

218

U.S. Patent

Oct. 20, 2015

Sheet 3 of 5

US 9,165,320 B1

300

N
Browser

EILE EDIT VIEW EAVORITES TOOLS HELP
Address

Apple Cake Recipe
Ingredients:

Nutritional Value

Calories

7 Apples

500 Calories per slice

2 Cups of Flour

Total Fat

2 Sticks of Butter

20 g

A Teaspoon of Baking Soda

Total Carbohydrate

1 Cup of Sugar

60 g

Directions:

Mix Ingredients into a large bowl
Bake at 400 degrees

FILE EDT VIEW FAVORS

Address

1. Salad
mol
3. Guacamole

* * * * * 406
O
w

J.-S.

4. AVOcado and Pear

$1.00 and $5.00
$5.00 and $10.00
$10.00 and $20.00
Greater than $20.00
Calories
s300 Calories

O 300 Calories
O 500+and 500 Calories
Foods to Avoid

Smoothie

FIG. 4

U.S. Patent

Oct. 20, 2015

Sheet 4 of 5

US 9,165,320 B1

500
EILE EDIT VIEW EAVORITES

OOLS HELP

Address Div

GO

Shopping Cart
ShipOind Information

Avocado - Recipe : Guacamole

John Doe

Quantity: 1

City, State 12345

Price: S100

1234 Example Lane

Order Summary
Items: S23.00

Shipping: S 5.00

Pear - Recipe: Salad
Quantity: 1

Total: $28.00

Price: $2.00

Place your order

Tossing Spoons - Recipe: Salad
Quantity: 1
Price: $20.00

600

FIG. 5

602

EILE EDIT VIEW FAVORITES TOOLS HELP

Address D.

VEGO

Shopping Cart

Shippind Information

1. Recipe: Avocado and Pear Salad
Ingredients in Cart: Pear, Avocado, Lettuce, Dressing
Update Ingredients

604

Price: $7.00

John Doe

1234 Example Lane
City, State 12345

Order Summary
Items: $30.00

Shipping: $5.00
2. Recipe: Steak and Musnrooms
ecipe: Steak and Mush

Ingredients in Cart: Rib eye Steak, Mushrooms

Update Indredients \

Price: $13.00

Total: $35.00

Place your order

606

3. Recipe: Cheesecake

Ingredients in Cart: Sugar, Crust, Cream Cheese
Update Ingredients

Price: S10.00

FIG. 6

U.S. Patent

Oct. 20, 2015

Sheet 5 of 5

700
Browser

FILE EDIT VIEW FAVORITES TOOLS HELP
Address

Please Rate the Recipe You Recently Tried
AVOcado and Pear Salad
Stars:

irrir six A
Comments:

FIG. 7

US 9,165,320 B1

US 9, 165,320 B1
1.
AUTOMATIC ITEM SELECTION AND
ORDERING BASED ON RECIPE
BACKGROUND

In recent years, purchase transactions conducted over the
Internet have grown considerably. Today, millions of people
access the Internet to purchase a wide variety of goods and
services. For example, many people use the Internet to pur
chase books, computers, vehicles, clothes, jewelry, furniture,
and/or the like. One area of web commerce exhibiting rapid
expansion is in the area of food purchase and delivery. Today,
many online retailers enable users to conveniently purchase
their groceries online.
In an effort to improve user shopping experiences, many
online retailers currently provide purchase recommendations
to their users. For example, a retailer might recommend that a
user purchase a particular desktop computer model that is
available from the retailer. While providing recommenda
tions can help users in their purchasing decisions, current
recommendation schemes are limited particularly in the
area of food purchase and delivery.

10

15

items such that the items are delivered to him or her. In other

instances, embodiments can, without Substantial user inter

vention, create a purchase order following the selection of the
one or more items. In this way, a user does not need to access
a virtual shopping cart to make a purchase. Rather, the user
simply needs to make a recipe selection in order to have items
for the recipe ordered and/or delivered to him or her.

BRIEF DESCRIPTION OF THE DRAWINGS
25

Various embodiments in accordance with the present dis
closure will be described with reference to the drawings, in
which:
FIG.1 illustrates an environment in which various embodi

ments can be implemented;
FIG. 2 illustrates a process for selecting items for purchase
based on one or more selected recipes in accordance with one

30

embodiment;

FIG. 3 illustrates an example of an interface that can be
generated for display to allow a user to select a recipe in

35

accordance with one embodiment;

FIG. 4 illustrates an example of an interface that can be
generated for display to allow a user to select a recipe in
accordance with another embodiment;

FIG. 5 illustrates an example of an interface that can be
generated for display to allow a user to view the contents of a
virtual shopping cart assembled based on one or more
selected recipes in accordance with one embodiment;
FIG. 6 illustrates an example of an interface that can be
generated for display to allow a user to view the contents of a
virtual shopping cart assembled based on one or more
selected recipes in accordance with another embodiment; and
FIG. 7 illustrates an example of an interface that can be
generated for display to allow a user to input rating informa
tion and comments for a recipe in accordance with one

40

45

In some embodiments, the selection of the one or more

items from the electronic marketplace can be based on any
Suitable user preference criteria, such as price ranges, food
allergies, religious requirements, favored food genres, prepa
ration skill levels, preparation times, associated diet plans,
caloric values, etc. For example, a user might have previously
indicated that he or she wishes to stay within a S10.00 budget
for preparing a meal. As such, embodiments can select items
available from the electronic marketplace that meet the price
requirements of the user. Illustratively, a recipe input by a user
might require the use of Vanilla extract. The user might have
previously indicated that the recipe cost less than $25.00 to
complete. As such, embodiments might selecta generic bottle
of vanilla extract over a more expensive branded bottle of
vanilla extract in order to keep the cost of the recipe within the
$25.00 budget.
Through automatically selecting items for recipes and
either placing those items into a virtual shopping cart or
initiating a purchase order on behalf of a user, embodiments
can improve the speed and convenience of shopping for gro
ceries online. Illustratively, at present, users can spend a large
amount of time manually searching for the required ingredi
ents of a recipe. Furthermore, some recipes can be cryptic or
hard to understand. As such, users can often find it difficult to

50

embodiment.
DETAILED DESCRIPTION

Systems and methods in accordance with various embodi
ments of the present disclosure enable the selection of one or
more recipes by a user. In some embodiments, items available
for purchase from an electronic marketplace can be selected
based on the recipe selection. In certain embodiments, the
selected items can be automatically placed or added to a
virtual shopping cart Such that a user can quickly and conve
niently purchase the items. In other embodiments, a purchase
order for the selected items can be automatically created
following the selection of the items. In such embodiments, a
specific user confirmation to create the order might not be
required. After the items are purchased, the items can be
delivered or shipped to the user.

2
Illustratively, embodiments can enable a user to input one
or more recipes. For example, a user can select one or more
recipes from a recipe catalog provided by an electronic mar
ketplace. Upon receiving the recipe selections, embodiments
can select one or more items available for purchase from the
electronic marketplace. Each selected item can be an item that
is required to complete the one or more recipes. For example,
a recipe may require the use of an apple, as Such embodiments
can select an apple item available for purchase from the
electronic marketplace. As another example, a recipe may
require the use of a stand mixer, as a result embodiments can
select a particular brand of stand mixer available for purchase
from the electronic marketplace. In some instances, after
selecting the one or more items, embodiments can present, to
the user, information for the one or more items as part of a
virtual shopping cart or shopping bag. From the virtual shop
ping cart, the user can initiate a purchase transaction for the

55

60

select the appropriate items to purchase. Embodiments, by
automatically identifying appropriate items for use with a
recipe can significantly improve the convenience of shopping
for groceries. In embodiments that automatically initiate pur
chase orders for a user without a specific user purchase con
firmation, the speed and convenience of shopping for grocer
ies can be further improved.
FIG. 1 illustrates an example of an environment 100 for
implementing aspects in accordance with various embodi
ments. As will be appreciated, although a Web-based envi
ronment is used for purposes of explanation, different envi
ronments may be used, as appropriate, to implement various
embodiments. The environment 100 shown includes both a

65

testing or development portion (or side) and a production
portion. The production portion includes an electronic client
device 102, which can include any appropriate device oper
able to send and receive requests, messages, or information
over an appropriate network 104 and convey information
back to a user of the device. Examples of such client devices

US 9, 165,320 B1
4
embodiments, an application server can function as a devel
opment server, and separate production and testing Storage
may not be used.
The data store 110 can include several separate data tables,
databases, or other data storage mechanisms and media for
storing data relating to a particular aspect (e.g., catalog data
for items available for purchase, recipe data including one or
more recipes, etc.). For example, the data store illustrated
includes mechanisms for storing production data 112 and

3
include personal computers, tablet devices, multimedia digi
tal players, cell phones, handheld messaging devices, laptop
computers, set-top boxes, personal data assistants, electronic
book readers, and/or the like. In some embodiments, client

device 102 can store data files (e.g., cookies) that can be
transmitted over network 104 to, for example, Web server 106
in order to convey identification and/or other information.
Network 104 can include any appropriate network, including
an intranet, the Internet, a cellular network, a local area net

work, or any other such network or combination thereof.
Components used for Such a system can depend at least in part
upon the type of network and/or environment selected. Pro
tocols and components for communicating via Such a net
work are well known and will not be discussed herein in

detail. Communication over the network can be enabled by

10

15

wired or wireless connections, and combinations thereof. In

this example, the network includes the Internet, as the envi
ronment includes a Web server 106 for receiving requests and
serving content in response thereto, although for other net
works an alternative device serving a similar purpose could be
used as would be apparent to one of ordinary skill in the art.
The illustrative environment includes at least one applica

additional mechanisms in the data store 110. The data store

tion server 108 and a data store 110. It should be understood

that there can be several application servers, layers, or other
elements, processes, or components, which may be chained
or otherwise configured, which can interact to perform tasks
Such as obtaining data from an appropriate data store. As used
herein the term “data store” refers to any device or combina
tion of devices capable of storing, accessing, and retrieving
data, which may include any combination and number of data
servers, databases, data storage devices, and data storage
media, in any standard, distributed, or clustered environment.
The application server can include any appropriate hardware
and Software for integrating with the data store as needed to
execute aspects of one or more applications for the client
device, handling a majority of the data access and business
logic for an application. The application server provides
access control services in cooperation with the data store, and
is able to generate content such as text, graphics, audio, and/
or video to be transferred to the user, which may be served to
the user by the Web server in the form of HTML, XML, or
another appropriate structured language in this example. The
handling of all requests and responses, as well as the delivery
of content between the client device 102 and the application
server 108, can be handled by the Web server. It should be
understood that the Web and application servers are not
required and are merely example components, as structured
code discussed herein can be executed on any appropriate
device or host machine as discussed elsewhere herein. Fur

ther, the environment can be architected in Such a way that a
test automation framework can be provided as a service to
which a user or application can Subscribe. A test automation
framework can be provided as an implementation of any of
the various testing patterns discussed herein, although Vari
ous other implementations can be used as well, as discussed
or Suggested herein.
The environment also includes a development and/or test
ing side, which includes a user device 118 allowing a user
Such as a developer, data administrator, or tester to access the
system. The user device 118 can be any appropriate device or
machine, such as is described above with respect to the client
device 102. The environment also includes a development
server 120, which functions similar to the application server
108 but typically runs code during development and testing
before the code is deployed and executed on the production
side and is accessible to outside users, for example. In some

user information 116, which can be used to serve content for

the production side. The data store also is shown to include a
mechanism for storing testing data 114, which can be used
with the user information for the testing side. It should be
understood that there can be many other aspects that may
need to be stored in the data store, Such as for page image
information and access right information, which can be stored
in any of the above listed mechanisms as appropriate or in

25

110 is operable, through logic associated therewith, to receive
instructions from the application server 108 or development
server 120, and obtain, update, or otherwise process data in
response thereto. In one example, a user might Submit a
search request for a certain type of item (e.g., a food item). In
this case, the data store might access the user information to
Verify the identity of the user, and can access the catalog detail
information to obtain information about items of that type.
The information then can be returned to the user, such as in a

30

results listing on a Web page that the user is able to view via
a browser on the user device 102. Information for a particular
item of interest can be viewed in a dedicated page or window
of the browser.

35

40

Each server typically will include an operating system that
provides executable program instructions for the general
administration and operation of that server, and typically will
include a computer-readable medium storing instructions
that, when executed by a processor of the server, allow the
server to perform its intended functions. Suitable implemen
tations for the operating system and general functionality of
the servers are known or commercially available, and are
readily implemented by persons having ordinary skill in the
art, particularly in light of the disclosure herein.
The environment in one embodiment is a distributed com

45

50

55

puting environment utilizing several computer systems and
components that are interconnected via communication links,
using one or more computer networks or direct connections.
However, it will be appreciated by those of ordinary skill in
the art that such a system could operate equally well in a
system having fewer or a greater number of components than
are illustrated in FIG.1. Thus, the depiction of the system 100
in FIG. 1 should be taken as being illustrative in nature, and
not limiting to the scope of the disclosure.
An environment such as that described with respect to FIG.
1 can be used to provide items for consumption by potential
customers. As used herein, the term “item' can refer to any
thing, such as a product (e.g., food or kitchen equipment), an
electronic embodiment of or reference to a product, a service,
or a system, that can be ordered, purchased, rented, used,
transferred, or otherwise consumed and/or accessed via a

60

65

network request or electronic Submission as part of an elec
tronic transaction. A request can include any appropriate
request sent over an appropriate system or network, Such as a
request submitted to a Web page over the Internet or a mes
sage sent via a messaging system to a content provider, for
example.
In an example in accordance with one embodiment, a user
of an electronic marketplace is able to send requests from a
client device 102 over a network 104 that are received by an

US 9, 165,320 B1
5
application server 108 to allow the user to input recipes,
browse for recipes, browse items offered by the electronic
marketplace, etc. Using any of a variety of methods known or
used in the art, the user is able to input recipes; search for and
browse recipes; and search for and browse items offered by
the marketplace. The user can be presented with any appro
priate information available for each recipe and/or item. For
example, the user can be provided with a description and price
for a food item. In the embodiment, the user can select items

that he or she would like to consume by using a virtual
shopping cart or other repository for items. When the user is
finished selecting items for a given transaction, the user can
be presented with an interface allowing the user to purchase
the items. It should be appreciated that various approaches
exist within the scope of the various embodiments.
In some embodiments, a server of a content provider, Such
as a provider of an electronic marketplace can store, on data
store 110, catalog data. The catalog data can include infor
mation regarding one or more items that are available for
purchase from the electronic marketplace. In some embodi
ments, the catalog data can include information for items
available for purchase directly from the provider of the elec
tronic marketplace and/or items available for purchase from
third party vendors associated with the electronic market
place. In some embodiments, the information for items in the
catalog data can include, for example, item prices, item fea
tures, item images, item specifications, item descriptions,
Video clips, audio clips, user comments, reviews from both
paper and online-based periodicals and/or the like. Illustra
tively, a package of lettuce available from the electronic mar
ketplace can be associated with a price and user reviews for
the product.
In some embodiments, a server of a content provider can
additionally store, on data store 110, recipe data. The recipe
data can be received from any suitable source, including one
or more third party entities. In some embodiments, the recipe
data can include one or more recipes. Each recipe can be
associated with certain recipe attributes, metadata, tags, and/
or the like. For example, each recipe can be associated with a
set of attributes indicating the recipe’s caloric value, associ
ated diet plans, associated food genres, religious approvals,
preparation skill levels, preparation times, serving numbers,
meal type (e.g., breakfast, lunch, dinner, anytime, etc.) and/or
any other suitable recipe related vector. Illustratively, a recipe
can be associated with a first attribute indicating an Italian
food genre and a second attribute indicating that the recipe is
in accordance with kosher requirements.
Each recipe can further include instructions for cooking a
particular dish or meal. The instructions can include text,
images, video, audio, hyperlinks, and/or any other Suitable
multimedia elements. In some embodiments, each recipe can
reference one or more food items, kitchen equipment items,
and/or the like. Embodiments of the present invention can
analyze the instructions (e.g., parse text-based instructions)
of each recipe to identify the one or more items referenced by
the recipe. For example, a recipe can reference use of a Sugar
ingredient and a mixing bowl. Embodiments of the present
invention can perform a parsing operation on the text of the
recipe in order to identify the aforementioned Sugar ingredi
ent and mixing bowl item.

10

dient with the item.
15

As another example, embodiments can associate or map
items belonging to the same class or category. In particular,
embodiments can determine that an item referenced in a

25

30

35

recipe is associated with a particular item category. Embodi
ments can further determine items referenced in the catalog
data are also associated with the same item category. Embodi
ments can, based on the determinations, associate or map the
item referenced in the recipe to the items referenced in the
catalog data. Illustratively, an ingredient of a recipe can be
identified as being associated with a cabbage category. Anapa
productanda Savoy product referenced in the catalog data can
also be identified as being associated with the cabbage cat
egory. As such, embodiments can associate the ingredient in
the recipe with the napa and Savoy products referenced in the
catalog data. In this way, items referenced in a recipe can be
associated with items referenced in the catalog data without
requiring that the items include identical or similaridentifiers.
Associations between an item and a category can be deter
mined in any suitable manner. For example, associations can
be manually provided or input by a user or group of users. As
another example, associations can be determined based on a
Suitable algorithm. Illustratively, associations can be deter
mined based on a fuzzy logic algorithm.
Embodiments can further associate items referenced in a

40

45

50

recipe with items referenced in the catalog databased on user
input. For example, embodiments can enable a recipe editor,
system administrator, or any other suitable user to associate
an item referenced in a recipe with one or more items refer
enced in the catalog data. Illustratively, embodiments can
provide an interface that enables a recipe editor to associate
an ingredient in a particular recipe with, for example, a straw
berry item, a blueberry item, and a raspberry item referenced
in an electronic marketplace's catalog data.
In some embodiments, items from a recipe can be associ
ated with alternative items. For example, a recipe might
specify a button mushroom. Embodiments might associate
the button mushroom with a button mushroom item refer

55

enced in the catalog data of an electronic marketplace.
Embodiments might additionally associate the button mush
room with an alternative item in the catalog data, Such as a
cremini mushroom. In some embodiments, it may be prefer
able to select an alternative item instead of a standard item.
Selection of an alternative item over a standard item can occur

for any suitable reason. For example, an alternative might be
selected because a user has indicated that he or she does not
60

Embodiments can further associate one or more of the

identified food items, kitchen equipment, and/or the like with
one or more items referenced in an electronic marketplace's
catalog data. For example, a recipe can include an apple
ingredient. Embodiments can associate or map the apple
ingredient in the recipe with an apple product referenced in
the electronic marketplace's catalog data. As another

6
example, a recipe can include a mushroom ingredient.
Embodiments can associate or map the mushroom ingredient
with a button mushroom product, an oyster mushroom prod
uct, and a chanterelle mushroom product, each of which can
be referenced in the electronic marketplace's catalog data.
Embodiments can associate or map items referenced in a
recipe with items referenced in an electronic marketplace's
catalog data in any Suitable manner. For example, embodi
ments can associate items through identifying those items
with identical or similar identifiers. Illustratively, an ingredi
ent referenced in a recipe and an item referenced in the cata
log data can both include the term “apple' in their names or
descriptions. As such, embodiments can associate the ingre

65

like the original ingredient specified in a recipe. As another
example, an alternative might be selected in order to enable a
recipe to meet certain user preferences, such as a caloric
Value, a price range, etc.
Determination of a specific alternative item for a standard
item can be performed using any Suitable criteria or informa
tion. For example, alternative items can be determined based
on associated food item categories, food item taste profiles,

US 9, 165,320 B1
7
information provided by a recipe editor, crowd-sourcing
information, and/or the like. Illustratively, users of the elec
tronic marketplace (via a suitable interface) might recom
mend that a certain cheaper ingredient be used in a recipe over
a more expensive counterpart. Based on this crowd-sourced
information, embodiments might determine that the cheaper
ingredient is a Suitable alternative for the more expensive
ingredient.
According to some embodiments, recipes and information
for items available for purchase from an electronic market
place can be presented to users in various fashions at various
locations and times, such as during item discovery (e.g.,
searching, browsing, etc.) and/or at various points in an order
pipeline.
For example, a user can access a web page associated with
the electronic marketplace to browse for one or more recipes.
Upon deciding on a particular recipe, the user can select the
recipe. Embodiments can thereafter populate the user's vir
tual shopping cart with the items needed to complete the
recipe. Thereafter, the user can access the virtual shopping
cart to obtain information regarding the different items
needed to complete the recipe, and also to create a purchase
order for the items. In some instances, rather than populating
a virtual shopping cart, embodiments can automatically cre
ate a purchase order on behalf of the user. In doing so, items
needed by the user to complete the selected recipe can be
automatically delivered to the user.
As another example, a user can access a recipe web page
available from the server of a third party content provider,
Such as the website of a cooking periodical. The web page
may include a hyperlink, button, or any other Suitable inter
face element for purchasing the items needed to complete the
recipe. In some instances, by clicking on the interface ele
ment, a virtual shopping cart for the electronic marketplace
can be populated with the items needed to complete the
recipe. The user may later access the virtual shopping cart to
initiate a purchase transaction for the items. In other
instances, rather than populating a virtual shopping cart,
embodiments can automatically create a purchase order on
behalf of the user. In this example, transfer of the information
for the recipe from the third party content provider to the
electronic marketplace can occur in any suitable manner.
Illustratively, information for the items necessary for the
recipe can be transmitted from a server of the third party
content provider to a server of the electronic marketplace.
Upon receipt, embodiments can select one or more of the
items from the electronic marketplace, and either populate a
virtual shopping cart or automatically create a purchase order

5

10

15

on historical user data. In some instances, the historical user
25

30

35

40

45

on behalf of the user.

In some embodiments, an algorithm can be employed to
select one or more items from the electronic marketplace. The
one or more items can be those items needed to complete the
recipes selected by the user. The algorithm can consider any

50

suitable criteria or combinations of criteria to select the one or

more items. In some embodiments, the algorithm can con
sider a set of user preferences in order to select the one or
more items. User preference criteria can include any suitable
criteria, such as food allergy criteria, serving number criteria,
a calorie requirement, a diet plan, an exercise plan, a religious
requirement, a meal preparation skill level, a set of preferred
food items, price criteria, a set of food item exclusions, etc.
For example, items referenced in a recipe can be associated
with more than one item available for purchase from an elec
tronic marketplace. Illustratively, a vanilla extract ingredient
in a recipe might be associated with 10 different brands of
vanilla extract available from the electronic marketplace. As
part of his or her userpreferences, a user might have indicated

8
that he or she wishes to have the cheapest option for complet
ing a selected recipe. In order to reduce the cost of the recipe,
the algorithm might select the brand of vanilla extract with the
lowest price.
In some embodiments, the algorithm can Swap items in a
recipe to allow the recipe to meet the criteria provided by a
user's preferences. For example, a particular recipe might
require a relatively expensive portabella mushroom that
would cause the recipe to cost more than $10.00 to complete.
The algorithm might, in order to reduce the cost of the recipe,
swap out the portabella mushroom with a more cost effective
button mushroom alternative. As another example, a particu
lar recipe might require a particular ingredient that does not
meet the religious requirements specified by a user. The algo
rithm might Swap out the ingredient for an ingredient that
meets the religious requirements of the user. As still another
example, a particular recipe might require an ingredient to
which a user has indicated he or she is allergic. The algorithm
might Swap out the ingredient for an ingredient to which a
user has indicated he or she is not allergic.
In some embodiments, the algorithm can select items based

55

60

data can include data specific to the user. In certain imple
mentations, the algorithm can analyze the purchase history of
the user to determine items that the user might prefer to be
used in the selected recipes. As a simple example, a user
might have previously purchased frozen pizzas, dried pasta,
and parmesan cheese. Each of the aforementioned items
might be associated with a single product brand. Based on this
information, the algorithm might selector be more inclined to
select items associated with the same product brand. In some
embodiments, historical user data can additionally or alter
natively include aggregate user data collected by the elec
tronic marketplace (e.g., via a server). In some implementa
tions, the algorithm can determine those items that are most
popular and/or most highly rated by users of the electronic
marketplace. Based on this information, the algorithm might
select or be more inclined to select items that are relatively
popular or highly rated.
In some embodiments, the algorithm can select items based
on training data provided by the user. Embodiments can
obtain the training data in any suitable manner. For example,
the training data can be based on ratings previously provided
for items that the user has previously purchased. The training
data can be additionally or alternatively be based on ratings
provided by the user for a series of example items. Illustra
tively, embodiments might present the user with a series of
example items. The user can provideratings for each example
item. In some embodiments, the ratings can indicate the
user's relative like or dislike for a particular item. The algo
rithm can use the ratings provided by the training data to
determine those items that the user might tend to prefer.
Embodiments can use any combination of item attributes
(e.g., brand name, caloric value, price, etc.) and other infor
mation gleaned from the training data to determine those
items that the user might tend to enjoy or prefer. For example,
the algorithm might determine that the user tends to prefer
brand name items over generic items. As a result, the algo
rithm might be more included to select brand name items for
the user. As another example, the algorithm might determine
that the user tends to prefer low cost food items. As a result,
the algorithm might be more inclined to select cheaper food
item options for the user.
In Some embodiments, where a user has selected more than

65

one recipe, the algorithm can determine whether any dupli
cate ingredients are shared among the recipes and adjust the
quantity of any duplicate ingredients accordingly. Illustra

US 9, 165,320 B1
9
tively, a user may have selected three recipes. Each recipe can
require, for example, a slice of bread. The algorithm can
determine that a single loaf of bread would be sufficient to
complete each of the three recipes. As such, only one loaf of
bread would be selected (rather than, for example three
loaves—one for each recipe). In this way, a user can obtain a
Sufficient amount of ingredients necessary to complete his or
her selected recipes while avoiding over purchasing the quan
tity of the ingredients.
In some embodiments, as discussed, information for one or

more items available for purchase from the electronic mar
ketplace can be presented to a user via a virtual shopping cart
upon the user's selection of a recipe that requires the items.
The user can thereafter initiate a purchase transaction by
providing an order confirmation. In other embodiments, the
one or more items might not be presented to a user prior to
initiation of the purchase transaction. Rather, a purchase
transaction can be automatically initiated for the one or more
items. In such embodiments, the user can have previously
provided payment and shipment information, which can be
securely stored in, for example, data store 110. As such, the
purchase transaction and shipment of the items to the user can
proceed without substantial user intervention.
In some embodiments, a user can indicate that the delivery
of certain items is to be spread out or suitably distributed over
a time period. Illustratively, a user might select two recipes
and a number of items associated with the recipes for pur
chase. The user might further indicate that he or she would
like to make the first recipe on Monday, and the second recipe
on Tuesday. As such, only items associated with the first
recipe would be delivered on Monday. Items associated with
the second recipe would thereafter be delivered on Tuesday.
FIG. 2 illustrates a process for selecting items for purchase
based on a selected recipe in accordance with one embodi
ment. In this example 200, recipe data can be received at
block 202. The recipe data can be received from, for example,
one or more third party entities. As discussed, the recipe data
can include one or more recipes. In some embodiments, each
recipe can include instructions specifying, for example, the
use of certain items, such as ingredients, kitchen equipment,
etc. At block 204, the items specified in each recipe can be
associated with items referenced in the catalog data of an
electronic marketplace. The items referenced in the catalog
data can be those items that are available for purchase from
the electronic marketplace. At block 206, a recipe selection
can be received from a user. The recipe selection can include
information regarding one or more recipes that the user would
like to prepare. In some embodiments, the recipe selection
can additionally include user preferences. The user prefer
ences can include any suitable recipe Vectors, including price
ranges, caloric values, religious requirements, preparation

5

Once the user is satisfied with the different items in his or her
10

15

25

30

35

40

45

face element 302. Interface element 302 can be any suitable
interface element, such as a button, hyperlink, drop down
box, etc. According to Some embodiments, upon selecting
interface element 302, embodiments can automatically gen
erate a virtual shopping cart for the user. The virtual shopping
cart can include the items referenced by the cake recipe.
Illustratively, upon selecting interface element 302, a virtual
shopping cart for the user can be automatically populated
with 7 apple items, a package of flour, 2 Sticks of butter, a
package of baking Soda, and a package of Sugar. The items in
the virtual shopping cart can be those items available for
purchase from an electronic marketplace. In some embodi
ments, upon selecting interface element 302, embodiments
can automatically create a purchase order for items refer
enced by the cake recipe. For example, an order including 7
apple items, a package of flour, 2 sticks ofbutter, a package of
baking Soda, and a package of Sugar can be processed and
delivered to the user.

50

55

some embodiments, the selection of the items can be influ

60

formed as to whether an order for the one or more items

should be automatically created. The determination can be
performed based on any suitable criteria or information. For
example, the user might have indicated previously that an
order for any selected items should be automatically created.
If it is determined that an order should be created, the process
can proceed to block 216. Otherwise, the process can proceed

result of the user selecting an order Submit selection or similar
option, for example. At block 216, the order can be submitted
for processing such that the items can be delivered to the user.
At a later time, feedback for a selected recipe (e.g., rating
information) can be provided by the user at block 218.
While FIG. 2 shows a specific set of steps, there can be
several additional and/or alternative steps that can be used
with the steps shown in FIG. 2 or fewer steps in any appro
priate order, in accordance with various embodiments.
FIG.3 illustrates an example of an interface 300 that can be
generated for display to allow a user to select a recipe in
accordance with one embodiment. In this example, a user can
view a web page that includes information for a recipe. The
web page can include recipe instructions, list of ingredients,
nutritional values, and other information. Illustratively, FIG.
3 shows an apple cake recipe that includes the ingredients
(e.g., apples, flour, etc.) required to complete the apple cake.
The example additionally includes directions for making the
apple cake and nutritional information.
In some embodiments, interface 300 can include an inter

skill levels, food items to avoid, and/or the like. At block 208,

enced by the user preferences and purchase history provided
by the user. At decision 210, a determination can be per

virtual shopping cart, a confirmation can be received from the
user for the order at block 214. The confirmation can be a

one or more items can be selected. The items can be those

items that are needed complete the selected one or more
recipes. Illustratively, a user can select a recipe for a salad. As
Such, the items (e.g., food items, kitchen equipment) needed
to complete the salad can be selected from the catalog data. In

10
to block 212. At block 212, information regarding the selected
items can be provided to the user. In particular, the selected
items can be placed in a virtual shopping cart, which can
enable a user to view various information regarding the
selected items. The information can include price informa
tion, quantity information, item descriptions, specifications,
ratings, etc. In some embodiments, the user can change,
update, or add additional items to the virtual shopping cart.

65

Interface 300 can be presented in any suitable location. For
example, in some embodiments, interface 300 can be
accessed from a server of the electronic marketplace. As
another example, interface 300 can be accessed from the
server of a third party entity. Illustratively, a user can access
interface 300 via a server associated with the website of a

cooking magazine. Upon selecting button 302, information
for the selected recipe, including the items referenced by the
recipe, can be transmitted to a server of the electronic mar
ketplace.
FIG. 4 illustrates an example of an interface 400 that can be
generated for display to allow a user to select a recipe in
accordance with another embodiment. Illustratively, inter
face 400 shows a list 402 including a salad recipe; a prawns
with chutney recipe; a guacamole recipe; and an avocado and
pear Smoothie recipe. As shown in FIG. 4, a set of user
preferences 404 is displayed that enables a user to specify
criteria that can influence the recipes presented in list 402. In

US 9, 165,320 B1
11
this example, a user can be presented with pricing ranges,
calorie ranges, a field related to foods exclusions, and a field
for inputting favored food genres. As shown in FIG. 4, the
user has, for example, specified that any listed recipes cost
greater than $20.00.
As discussed, the recipes in list 402 can meet certain user
preferences input by the user. For example, the user may have
indicated that green beans and onions be avoided in any
selected recipes. As a result, each recipe in the list might not
include green beans or onions as ingredients. In some
embodiments, not all the user preferences are required to be
met in order for a recipe to be presented. In certain embodi
ments, list 402 can be updated dynamically. More specifi
cally, as a user changes or updates his or her user preferences,
list 402 can be automatically updated (e.g., new recipes pre
sented, recipes removed, recipes re-ordered, etc.) in real-time
or Substantially real-time to reflect changes in the user pref

10

15

CCS.

In some embodiments, list 402 can be ranked or ordered.

According to some embodiments, recipes ranked higher can
appear at the top of list 402 while lower ranked recipes can
appear at the bottom of the list. The rankings of the recipes
can be based on any Suitable criteria or combinations of
criteria. In certain embodiments, recipes can be ranked based
on how closely the recipes meet a user's preferences. For
example, recipes that meet a greater number of the user's
preferences can be ranked higher. In some embodiments,
recipes can be ranked based on user historical data and/or user
provided ratings. In the example of FIG. 4, the recipes in list
402 are ranked based on the star rating for each recipe. A
recipe’s individual starrating can be the average rating for the
recipe provided by an electronic marketplace's users. In this
example, users can provide a recipe with a rating between one
and five stars. Recipes with more stars can be rated and ranked
higher than recipes with fewer stars. Illustratively, the salad
recipe has a 5 star rating 404 and as Such is rated higher than
the guacamole recipe, which has a 4 star rating 406. As a
result, the salad recipe is positioned higher on list 402 than the
guacamole recipe.
In some embodiments, interface 400 can be used in addi

25

30

35

40

tion or in alternative to interface 300 shown in FIG. 3. Illus

tratively, embodiments can enable a user to select a recipe
while viewing detailed information for the recipe (e.g., view
a web page regarding a recipe that includes instructions and
an ingredients list, etc.). Embodiments can further enable a
user to select one or more recipes from a list of recipes. In the

45

view, as shown in interface 400, detailed information, such as

an ingredients list, might not be presented to the user.
FIG. 5 illustrates an example of an interface that can be
generated for display to allow a user to view the contents of a
virtual shopping cart assembled based on one or more
selected recipes in accordance with one embodiment. In the
example, the virtual shopping cart includes a list 502 of one or
more items selected for purchase. As shown in FIG. 5, the list
includes an avocado item, a pear item, and a tossing spoons

50

55

item. Each of the aforementioned items can have been auto

matically added to the virtual shopping cart based on recipes
previously selected by the user. The avocado, pear, and toss
ing spoons items can represent those items that are available
for purchase from the electronic marketplace. In some
embodiments, each item in the list can include various pur
chase information, such as a price for the item, a selected
quantity to be purchased, etc. For example, the pear item is
shown as being $2.00 in price. In certain embodiments, each
item in the list can include the recipe with which it is associ
ated. For example, the pear item includes a description 504
that indicates that the pear item is associated with (e.g., to be

60

65

12
used for) a salad recipe previously selected by the user. In this
way, a user can be apprised of the particular recipe with which
each item in his or her shopping cart is associated.
Upon reviewing the information displayed by interface
500, a user can cause a purchase transaction to be initiated by
clicking on a suitable interface element. For example, inter
face 500 includes a “Place your order” button. Following the
purchase transaction (which may also include the input of
payment information, shipping information, etc.) the items
listed in the virtual shopping cart can be distributed or deliv
ered to the user. In certain embodiments, any deliveries of
items to the user can further include physical copies of each
selected recipe with which the items are to be used. In some
embodiments, the physical copies can include bar codes or
Some other recipe identification mechanism. In the event the
user wishes to make a recipe again, the user can use a Suitable
optical device (e.g., a mobile device camera, webcam con
nected to a computer, a bar code scanner connected to a
computer, etc.) to conveniently provide a recipe selection to a
server of the electronic marketplace.
FIG. 6 illustrates an example of an interface 600 that can be
generated for display to allow a user to view the contents of a
virtual shopping cart assembled based on one or more
selected recipes in accordance with another embodiment.
Interface 600 can be similar to interface 500 shown in FIG.S.

However, rather than presenting a list based on the items
selected for purchase, interface 600 can be based on the one or
more recipes previously selected by the user. In particular,
interface 600 includes a list 602 including those recipes pre
viously selected by the user. As shown in FIG. 6, list 602
includes an avocado and pear Salad recipe, a steak and mush
rooms recipe, and a cheesecake recipe. As further shown in
FIG. 6, each recipe in the list 602 can include a description
listing the items that are currently in the shopping cart. For
example, description 604 lists the ingredients needed to com
plete the steak and mushroom recipe. In addition, description
604 notes that these ingredients are currently in the user's
virtual shopping cart. As also shown in FIG. 6, an interface
element 606 (e.g., a hyperlink) can be used to update the
ingredients in the shopping cart. For example, a user might
currently own or have in his or her possession, a rib eye steak.
As such, the user might click on interface element 606 to
remove the rib eye steak from the cart. As discussed with
respect to FIG. 5, a user can click on the “Place your order
button to initiate a purchase transaction. As also discussed,
any deliveries of items to the user can include physical copies
of each selected recipe with which the items are to be used. In
the event the user wishes to make a recipe again, the user can
use a suitable optical device (e.g., a mobile device camera,
webcam connected to a computer, a bar code scanner con
nected to a computer, etc.) to conveniently provide a recipe
selection to a server of the electronic marketplace.
FIG. 7 illustrates an example of an interface 700 that can be
generated for display to allow a user to provide feedback for
a recipe in accordance with one embodiment. In some
embodiments, a rating system can be provided. The rating
system can enable a user to indicate his or her relative satis
faction with a recipe that the user has previously tried. In this
example, the rating system is based on a star rating system. In
particular, a user can provide a star rating between one and
five stars. In certain embodiments, a high star rating can
indicate that a user is relatively satisfied with the recipe. The
provided rating can be used in any Suitable manner. For
example, embodiments can use the ratings to later determine
other recipes that the user might like or dislike. As another
example, the rating can be averaged with the ratings provided
by other users for the recipe. In some embodiments, interface

US 9, 165,320 B1
13
700 enables a user to provide comments or a review of the
recipe. In particular, a user can input any comments via text
field or interface element 702. The comments can be viewed

by users of the electronic marketplace in order to assess a
particular recipe.
It should be appreciated that while the interfaces shown in
FIGS. 3-7 show use of specific interface elements (e.g., text
fields, etc.), any Suitable interface elements or combination of
interface elements can be used (e.g., a list, checkboxes, radio
buttons, etc.).
In some embodiments, a user can provide input or infor
mation regarding Substitutions or alternative ingredients for a
recipe. For example, a recipe may require an apple as an
ingredient. A user may indicate, through a suitable interface,
that Substituting the apple in the recipe for a pear would make
the food dish produced from the recipe taste better. Embodi
ments can use Such information in the future to suggest Sub
stitutions or alternatives for the recipe to, for example, other

10

In one embodiment, the user can use the camera to take
15

USCS.

In some embodiments, rather than selecting and browsing
for recipes, a user can have recipes automatically selected
based on his or her user preferences and/or other data (e.g.,
purchase history, etc.). Based on the selected recipes, items
can be selected and placed in a virtual shopping cart, or
automatically purchased and delivered to the user. Illustra
tively, a user can indicate that he or she wishes to adhere to a
certain diet plan. Embodiments, can select one or more reci
pes that meet, either alone or collectively, the requirements of
the diet. For example, a user might indicate that he or she is
currently on a low carbohydrates diet plan. Embodiments can
select recipes that collectively provide a nutritional value that
meets the requirements for the diet plan (e.g., selecting reci
pes that limit carbohydrate intake to a certain level).

25

30

35

40

45

scribed to the service.

In some embodiments, a user can provide information
regarding food items the user currently owns or is available to
the user. Recipes can thereafter be presented to the user based
on the food items that the user previously input. In some
embodiments, each of the recipes can include, as an ingredi
ent, at least one of the food items. After being presented with
the recipes, the user can select one or more of the recipes. In
response to the selection, embodiments can identify those
items available from the marketplace that are needed, in addi
tion to the food items provided by the user, to complete a set
of ingredients needed to prepare the recipes. Embodiments
can thereafter either place the additional items in a virtual
shopping cart associated with the user or automatically ini
tiate a purchase transaction to purchase the additional items

50

55

60

on behalf of the user.

Embodiments can, in Some implementations, enable a user
to input the food items that he or she currently owns or has
available by using a portable device. Such as a mobile phone,
tablet device, multimedia player, hand-held Scanner, camera,

pictures of barcodes on the packages or labeling of certain
food items. The barcodes (either an image of the barcode or
information related to the barcode e.g., a barcode value) can
be matched against a bar code database stored on, for
example, data store 110 to determine the food items in the
user's possession.
In a second embodiment, the user can use the camera to

In certain embodiments, a user can Subscribe to an auto

matic recipe selection and delivery service that is based on his
or her user preferences (e.g., price range, diet plans, food
allergies, etc.) and/or other data. For example, a user Sub
scribing to the service can indicate that he or she has a budget
of S100 every week. Based on this information, embodiments
can automatically select recipes that collectively cost $100
each week to produce. Based on these recipes, food and other
items can be automatically purchased and/or delivered to the
user on a periodic basis. For example, the user can receive a
first delivery of items on Thursday and a second delivery of
items on Friday. The service may continue to automatically
select recipes and deliver items needed for those recipes until
the user indicates that he or she no longer wants to be Sub

14
etc. For example, a user can use an RFID reader module of a
mobile device to scan an RFID tag embedded in or attached to
the package of a particular food item. The information
obtained by the reader module can be received and processed
by embodiments in order to determine a food item that the
user currently has in his or her possession. As another
example, a user can use a camera module of a mobile phone
to take pictures of food items that the user currently has in his
or her possession. The pictures can thereafter be received and
processed by embodiments in order to determine the food
items that the user currently has in his or her possession. For
example, the images can be analyzed by Amazon Mechanical
TurkTM or any suitable algorithmic systems.

65

take a suitable picture of one or more food items. The picture
can be analyzed using a suitable algorithm to determine the
food items that are the subjects of the picture, and therefore in
the user's possession. For example, a user can take a picture
of a particular food item. An algorithm can determine, based
on the image characteristics of the picture that the picture
includes a banana. As another example, a user can take a
picture of his or her refrigerator. An algorithm can identify,
based on the image characteristics of the picture, each of the
individual foods in the user's refrigerator. Illustratively, the
algorithm can determine that a cartoon of milk and a stalk of
celery are in the user's refrigerator.
In a third embodiment, the algorithm use optical character
recognition (OCR) to process the labels (e.g. packaging
labels, sticker labels, etc.) of the food items in a picture.
Through processing the text of the labels, the algorithm can
identify the specific food items in a picture.
In some implementations, an algorithm can automatically
fragment a picture into multiple portions and process each
portion individually to determine the food items identified in
the picture. In other implementations, a user can individually
identify or tag regions (e.g., each region can include an indi
vidual food item) of the picture in order to enable the algo
rithm to fragment the picture for processing.
Although certain embodiments described above are
described with respect to physical items, the methods and
systems can be equally applied to items that are electronically
fulfilled. For example, a user may be presented with the
environmental impact information associated with the deliv
ery of a physical medium containing a movie (e.g., a DVD,
Blu-Ray disc, etc.) versus the environmental impact of the
delivery of a movie downloaded or streamed to a user's
selected client device over a network, such as the Internet.
As discussed above, the various embodiments can be

implemented in a wide variety of operating environments,
which in some cases can include one or more user computers,
computing devices, or processing devices which can be used
to operate any of a number of applications. User or client
devices can include any of a number of general purpose
personal computers, such as desktop or laptop computers
running a standard operating system, as well as cellular, wire
less, and handheld devices running mobile software and
capable of supporting a number of networking and messaging
protocols. Such a system also can include a number of work
stations running any of a variety of commercially-available
operating systems and other known applications for purposes

US 9, 165,320 B1
15
Such as development and database management. These

16
ware, such as applets), or both. Further, connection to other
computing devices such as network input/output devices may
be employed.
Storage media and computer readable media for containing
code, or portions of code, can include any appropriate media
known or used in the art, including storage media and com

devices also can include other electronic devices. Such as

dumb terminals, thin-clients, gaming systems, and other
devices capable of communicating via a network.
Most embodiments utilize at least one network that would

be familiar to those skilled in the art for Supporting commu
nications using any of a variety of commercially-available
protocols, such as TCP/IP. OSI, FTP, UPnP, NFS, CIFS, and
AppleTalk. The network can be, for example, a local area
network, a wide-area network, a virtual private network, the
Internet, an intranet, an extranet, a public Switched telephone
network, an infrared network, a wireless network, and any

munication media, Such as but not limited to volatile and
10

combination thereof.

In embodiments utilizing a Web server, the Web server can
run any of a variety of server or mid-tier applications, includ
ing HTTP servers, FTP servers, CGI servers, data servers,
Java servers, and business application servers. The server(s)
also may be capable of executing programs or Scripts in
response requests from user devices, such as by executing one
or more Web applications that may be implemented as one or
more Scripts or programs written in any programming lan
guage, such as Java R., C, C# or C++, or any scripting lan
guage, such as Perl, Python, or TCL, as well as combinations
thereof. The server(s) may also include database servers,
including without limitation those commercially available
from Oracle(R, Microsoft(R), Sybase(R), and IBM(R).
The environment can include a variety of data stores and
other memory and storage media as discussed above. These
can reside in a variety of locations, such as on a storage
medium local to (and/or resident in) one or more of the
computers or remote from any or all of the computers across
the network. In a particular set of embodiments, the informa
tion may reside in a storage-area network ("SAN) familiar to
those skilled in the art. Similarly, any necessary files for
performing the functions attributed to the computers, servers,
or other network devices may be stored locally and/or
remotely, as appropriate. Where a system includes comput

15

mentS.

The specification and drawings are, accordingly, to be
regarded in an illustrative rather than a restrictive sense. It
will, however, be evident that various modifications and
25

changes may be made thereunto without departing from the
broader spirit and scope of the invention as set forth in the
claims.
What is claimed is:

30

1. A computer-implemented method comprising:
receiving, by a computer system, a request for recipes from
a user device;

receiving, over a network, a picture of one or more food
items that are not needed for an order associated with the

request for recipes, the picture captured by a camera
35

module associated with the user device, the user device

in communication with the computer system via the
network;

erized devices, each Such device can include hardware ele

ments that may be electrically coupled via abus, the elements
including, for example, at least one central processing unit
(CPU), at least one input device (e.g., a mouse, keyboard,
controller, touch screen, or keypad), and at least one output
device (e.g., a display device, printer, or speaker). Such a
system may also include one or more storage devices, such as
disk drives, optical storage devices, and Solid-state storage
devices such as random access memory (“RAM) or read
only memory (“ROM), as well as removable media devices,
memory cards, flash cards, etc.
Such devices also can include a computer-readable storage
media reader, a communications device (e.g., a modem, a
network card (wireless or wired), an infrared communication
device, etc.), and working memory as described above. The
computer-readable storage media reader can be connected
with, or configured to receive, a computer-readable storage
medium, representing remote, local, fixed, and/or removable
storage devices as well as storage media for temporarily
and/or more permanently containing, storing, transmitting,
and retrieving computer-readable information. The system
and various devices also typically will include a number of
Software applications, modules, services, or other elements
located within at least one working memory device, including
an operating system and application programs, such as a
client application or Web browser. It should be appreciated
that alternate embodiments may have numerous variations
from that described above. For example, customized hard
ware might also be used and/or particular elements might be
implemented in hardware, Software (including portable soft

non-volatile, removable and non-removable media imple
mented in any method or technology for storage and/or trans
mission of information Such as computer readable instruc
tions, data structures, program modules, or other data,
including RAM, ROM, EEPROM, flash memory or other
memory technology, CD-ROM, digital versatile disk (DVD)
or other optical storage, magnetic cassettes, magnetic tape,
magnetic disk storage or other magnetic storage devices, or
any other medium which can be used to store the desired
information and which can be accessed by the a system
device. Based on the disclosure and teachings provided
herein, a person of ordinary skill in the art will appreciate
other ways and/or methods to implement the various embodi

40

45

50

fragmenting, by the computer system, the picture into plu
rality of portions and processing each portion of the
plurality of portions to identify the one or more food
items included in the picture;
providing, over the network, a listing of recipes for presen
tation to the user device, wherein a recipe included in the
listing of recipes references, as an ingredient, at least one
of the one or more food items identified in the picture:
receiving, from the user device, a selection of at least one
recipe from the listing:
identifying, by the computer system, a set of catalog items,
wherein each catalog item in the set of catalog items is
needed to complete the at least one recipe;
automatically adding, by the computer system, the set of
catalog items to a virtual shopping cart associated with a
user that does not include the one or more food items

55

60

65

identified in the picture, wherein at least one catalog
item in the set of catalog items is a non-food item;
providing, over the network, the virtual shopping cart to the
user device, wherein the virtual shopping cart includes
purchase information for each catalog item in the virtual
shopping cart; and
creating, by the computer system, the order for the set of
catalog items, the set of catalog items being based at
least in part on a first catalog item that the user has
previously purchased.
2. The computer-implemented method of claim 1, wherein
the non-food item is a kitchen appliance, a utensil, an item of
cutlery, an item of cookware, an item of bakeware, or an item
of tableware.

US 9, 165,320 B1
17
3. The computer-implemented method of claim 1, further
comprising Substituting a catalog item in the set of catalog
items with one or more alternative catalog items, wherein the
Substituting is based in part on at least one of the following:
food allergy criteria, serving number criteria, a calorie
requirement, a diet plan, an exercise plan, a religious require
ment, a meal preparation skill level, a set of preferred food
genres, price criteria, a percentage of food items to be used, an
amount of food items to be used, or a set of food exclusions.

4. A computer-implemented method comprising:
receiving, by a computer system, a request for recipes from

18
11. A computer-implemented method comprising:
receiving, by a computer system, a request for recipes from
a user device;

receiving, over a network, a picture of one or more food
items that are not needed for an order associated with the

request for recipes, the picture captured by a camera
module associated with the user device, the user device

in communication with the computer system via the
network;

10

a user device;

receiving, over a network, a picture of one or more food
items that are not needed for an order associated with the

request for recipes, the picture captured by a camera

15

module associated with the user device, the user device

in communication with the computer system via the
network;

receiving, via the user device, information identifying
regions of the picture, a particular region of the regions
including an individual food item;
fragmenting, by the computer system, the picture into a
plurality of portions and processing each portion of the
plurality of portions to identify the one or more food
items included in the picture based at least in part on the
information identifying the regions of the picture;
providing for presentation, to the user device, one or more
recipes, wherein each recipe of the one or more recipes
references, as an ingredient, at least one of the one or
more food items identified in the picture;
receiving, from the user device, a selection of at least one
recipe;
identifying, by the computer system, a set of catalog items,
wherein the set of catalog items are needed to complete
the at least one recipe; and
automatically adding, by the computer system, the set of
catalog items to a virtual shopping cart associated with a

25

30

35

user that does not include the one or more food items

40

identified in the picture, the set of catalog items being
based at least in part on a first catalog item the user has
previously reviewed, wherein at least one catalog item in
the set of catalog items is a non-food item.
5. The computer-implemented method of claim 4, wherein

45

the non-food item is a kitchen tool.

6. The computer-implemented method of claim 4, wherein
the picture includes at least one of the following: an image of
the at least one food item, an image of a barcode associated
with the at least one food item, or an image of a label associ

50

ated with the at least one food item.

7. The computer-implemented method of claim 4, further
comprising Substituting one or more catalogitems in the set of
catalog items with one or more alternative catalog items.
8. The computer implemented method of claim 7 wherein
the substituting is based in part on at least one of the follow
ing: food allergy criteria, serving number criteria, a calorie
requirement, a diet plan, an exercise plan, a religious require
ment, a meal preparation skill level, a set of preferred food
genres, price criteria, a percentage of food items to be used, an

a user device;

55

receiving, over a network, a picture of one or more food
items that are not needed for an order associated with the
request for recipes, the picture captured by a camera
module associated with the user device, the user device

in communication with the computer system via the
network;
60

amount of food items to be used, or a set of food exclusions.

9. The computer-implemented method of claim 4, wherein
identifying the set of catalog items is based at least in part on
user purchase history data.
10. The computer-implemented method of claim 4,
wherein identifying the set of catalog items is based at least in
part on training data.

fragmenting, by the computer system, the picture into mul
tiple portions and processing each portion of the mul
tiple portions to identify the one or more food items
included in the picture;
providing, over the network and via the user device, one or
more recipes, wherein each recipe of the one or more
recipes references, as an ingredient, at least one of the
one or more food items identified in the picture;
receiving, from the user device, a selection of at least one
recipe;
identifying, by the computer system, a set of catalog items,
wherein the set of catalog items are needed to complete
the at least one recipe; and
creating, by the computer system, an order for the set of
catalog items that does not include the one or more food
items identified in the picture, the set of catalog items
being based at least in part on a first catalog item a user
has previously returned, wherein at least one catalog
item in the set of catalog items is a non-food item.
12. The computer-implemented method of claim 11,
wherein the non-food item is cooking equipment.
13. The computer-implemented method of claim 11,
whereincreation of the order results in the shipment of the set
of catalog items to the user.
14. The computer-implemented method of claim 11,
wherein the picture includes at least one of the following: an
image of the at least one food item, an image of a barcode
associated with the at least one food item, or an image of a
label associated with the at least one food item.
15. The computer-implemented method of claim 11,
wherein the processing includes performing optical character
recognition on the picture.
16. The computer-implemented method of claim 11, fur
ther comprising Substituting a first catalog item included in
the set of catalog items with a second catalog item, wherein
the substituting is based in part on at least one of the follow
ing: food allergy criteria, serving number criteria, a calorie
requirement, a diet plan, an exercise plan, a religious require
ment, a meal preparation skill level, a set of preferred food
genres, price criteria, a percentage of food items to be used, an
amount of food items to be used, or a set of food exclusions.
17. A computer-implemented method comprising:
receiving at least one user preference criteria;
receiving, by a computer system, a request for recipes from

65

receiving, via the user device and over the network, infor
mation identifying regions of the picture, wherein a par
ticular region of the regions includes an individual food
item;
fragmenting, by the computer system, the picture into mul
tiple portions and processing each portion of the mul
tiple portions to identify the one or more food items
included in the picture based at least in part on the
information identifying the regions of the picture;

US 9, 165,320 B1
19
causing presentation to the user device and over the net
work, of one or more recipes based on the at least one
user preference criteria, wherein each recipe of the one
or more recipes references, as an ingredient, at least one
of the one or more food items identified in the picture: 5
receiving, from the user device, a selection of at least two
recipes;
identifying, by the computer system, a set of one or more
catalog items, wherein the one or more catalog items are
needed to complete the at least one recipe;
10
creating, by the computer system, an order for the set of
catalog items that does not include the one or more food
items identified in the picture; and
identifying, by the computer system, a first delivery date
for a first Subset of the set of catalog items and a second

delivery date for a second subset of the set of catalog 15

items.

18. The computer-implemented method of claim 17,
wherein the at least one user preference criteria includes at
least one of the following: food allergy criteria, serving num
ber criteria, a calorie requirement, a diet plan, an exercise 20
plan, a religious requirement, a meal preparation skill level, a
set of preferred food genres, price criteria, a percentage of
food items to be used, an amount of food items to be used, or
a set of food exclusions.

19. The computer-implemented method of claim 17, fur- as

ther comprising initiating a periodic purchase transaction for
the set of catalog items.
k

k

k

k

k

20