Digital_Assistants_and_Privacy

Google,'Amazon'Patent'Filings'Reveal'Digital'Home'Assistant'Privacy'Problems'

Executive'Summary'

A!new!study!reveals!that!Amazon!and!Google!have!filed!patent!applications!for!a!number!of!

technologies!that!would!dramatically!expand!surveillance!of!consumers’!private!lives.!These!

patent!applications!show!how!technology!companies!use!home!data!to!draw!disturbing!

inferences!about!households,!and!how!the!companies!might!use!that!data!for!financial!gain.!

The!technologies!detailed!in!the!patent!applications!include:!

• A!system!for!deriving'sentiments'and'behaviors'from'ambient'speech,'even'when'a'

user'has'not'addressed'the'device!with!its!“wakeword.”!In!2014,!as!Amazon!was!

preparing!to!roll!out!its!Echo!smart!speaker,!the!company!patented!an!algorithm!that!

could!listen!to!human!speech,!including!phone!conversations,!for!statements!of!interest!

such!as!“I!love!skiing.”

!The!algorithm!would!then!process!the!statements!into!key!

words!that!could!be!used!to!target!advertising.!The!patent!describes!transmitting!the!

keywords!to!Amazon!servers!as!text,!which!could!allow!the!company!to!spy!on!

conversations!while!technically!keeping!its!promise!to!only!store!and!analyze!audio!

recordings!that!a!user!intends!to!share.!!!

• Multiple!systems!for'identifying'speakers'in'a'conversation!and!building!interest!

profiles!for!each!one.!Both!Google!and!Amazon!offer!users!the!option!of!creating!

acoustic!“voice!profiles”!for!voice-activated!smart!devices!in!their!homes.

!These!

profiles!can!help!the!devices!tailor!services!to!the!person!speaking.!Patents!show!that!

both!Amazon!and!Google!could!also!use!voice!profiles!to!associate!behaviors!with!

individual!members!of!the!household,!in!order!to!better!target!ads.

• A!method!for!inferring'users’'showering'habits'and'targeting'advertising!based!on!that!

and!other!data.!Dozens!of!patent!applications!for!Google’s!smart!home!devices!detail!

scenarios!in!which!Google!may!share!data!from!smart!home!devices!with!third!parties,!

including!businesses,!who!can!then!use!the!data!to!make!inferences!about!users’!

sleeping,!cooking,!entertainment,!and!showering!schedules.

!These!inferences,!Google!

says,!“may!help!third-parties!benefit!consumers!by!providing!them!with!interesting!

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131!!

!https://www.amazon.com/gp/help/customer/display.html?nodeId=201602230,!

https://support.google.com/googlehome/answer/7323910?hl=en!!

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131,!

http://pdfaiw.uspto.gov/.aiw?PageNum=0&docid=20160259308!!

!http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-

adv.html&r=0&f=S&l=50&d=PG01&OS=+%28AANM% 2FGoogle+or+AN%2FGoo gle%29+and+SPEC%2Fcooking+and

+SPEC%2Fshower+and+SPEC%2Finferences&RS=%28%28%28%28AANM%2FG oogle+OR+AN %2FGo ogle%29+AN D+

SPEC%2Fcooking%29+AND+SPEC%2Fshower%29+AND +SPEC%2Finferences%29&TD=80&Srch1=%28% 28%28% 28G

oogle.AANM.+OR+Google.AS.%29+AND+cooking.BIS.%29+AND+shower.BIS.%29+AND+inferences.BIS.%29&StartN

um=&Refine=Refine+Search&Query=+%28AANM%2FGoogle+or+A N % 2 F G o o gle % 2 9 + an d + S P EC %2Fcookin g + an d + S

PEC%2Fshower+and+SPEC%2Finferences+and+SPEC%2F%22home+data%22!!

information,!products!and!services!as!well!as!with!providing!them!with!targeted!

advertisements.”!

• A!system!for!recommending'products'based'on'furnishings'observed'by'a'smart'home'

security'camera.!A!particularly!troubling!patent!application!describes!how!Google!could!

use!video!feeds!from!smart!devices!to!determine!users’!behaviors!and!characteristics,!

including!“gender,!age,!fashion-taste,!style,!mood,!known!languages,!preferred!

activities,!and!so!forth.”

!The!patent!application!describes!smart!devices!that!target!

advertisements!based!on!the!title!of!the!book!by!a!user’s!bedside,!the!presence!of!a!

guitar!or!basketball!in!the!room,!and!the!face!of!a!famous!actor!on!a!user’s!t-shirt.!!

• A!methodology!for!“inferring'child'mischief”!using!audio!and!movement!sensors.!

Another!Google!patent!describes!a!smart!home!system!that!monitors!the!activity!of!

every!member!of!a!household!and!reports!back!to!a!designated!“policy!holder.”

!One!

application!uses!voice!fingerprinting!to!identify!the!presence!of!children!and!motion!

sensors!to!detect!activity,!and!reports!inferred!mischief!when!children!are!both!active!

and!quiet.!This!patent!also!mentions!advertising!applications.!!

• Systems!for!inserting'paid'content'into!the!responses!provided!by!digital!assistants.!

Both!Amazon!and!Google!have!patented!methods!for!serving!ads!to!users!through!their!

smart!devices.!An!Amazon!discusses!a!system!of!“intelligent!sponsorship!based!on!

knowledge!generated!by!reference!to!the!human!being!entity!using!the!system.”!A!

Google!patent!shows!how!advertisers!could!bid!to!have!the!company’s!digital!assistant!

feature!their!products!in!its!voice-based!search!results.

!!!

While!Amazon!and!Google!may!be!focused!on!the!commercial!implications!of!their!inventions,!

these!technologies!also!have!troubling!legal!and!ethical!implications.!Smart!devices!with!

cameras!and!microphones!inevitably!capture!video!or!audio!of!people!who!did!not!consent!to!

being!recorded.

!Third!parties!can!also!access!data!from!smart!devices;!a!recent!study!found!

that!hackers!could!easily!access!insecure!wifi-enabled!“smart”!toys!and!talk!to!children.

!Law!

enforcement!has!also!begin!to!subpoena!data!from!smart!devices!in!criminal!investigations.

These!patent!applications!show!the!potentially!staggering!cost!of!a!futuristic!“smart!home.”!To!

anticipate!and!meet!consumers’!needs!in!new!ways,!digital!assistants!make!increasingly!

!http://pdfaiw.uspto.gov/.aiw?Docid=20160260135!!

!http://pdfaiw.uspto.gov/.aiw?docid=20160259308!!

!http://pdfaiw.uspto.gov/.aiw?docid=20160180848!!

http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_

EXPECTATION_OF_PRIVACY_FINAL.pdf!!

!https://www.theguardian.com/technology/2017/nov/14/retailers-urged-to-withdraw-toys-that-allow-hackers-to-

talk-to-children! !

http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE

_EXPECTATION_OF_PRIVACY_FINAL.pdf!!

invasive!forays!into!users’!private!lives.!As!users!accept!these!intrusions,!they!give!up!their!

personal!data,!and!with!it,!their!privacy!and!security.!!!

Background'

Smart!speakers!with!voice-activated!digital!assistants,!like!the!Amazon!Echo!and!Google!Home,!

promise!to!bring!a!personal!Star)Trek!computer!into!users’!homes.!The!devices!allow!users!to!

search!the!web,!play!music,!order!pizza,!and!make!phone!calls!via!a!voice!command.

!When!

connected!with!other!“smart”!devices,!the!Echo!and!Google!Home!can!also!dim!the!lights,!

control!the!thermostat,!turn!on!the!sprinkler,!and!check!the!baby!monitor.

While!these!devices!offer!convenience,!they!also!offer!tech!companies!unprecedented!insight!

into!users’!private!lives.!With!the!smaller-sized!Echo!Dot!and!the!Google!Home!Mini,!Amazon!

and!Google!have!begun!to!foray!beyond!the!living!room!and!into!the!bedroom.

!There,!they!

can!infer!from!your!interactions!with!the!device!when!you!wake!up!–!and!maybe!even!who!you!

wake!up!with.!Both!companies’!eventual!goal!is!to!have!a!device!in!every!room!of!your!home.

This!study!asks!what!user!data!Amazon!and!Google!derive!from!these!devices!and!what!the!

companies!intend!to!do!with!that!data.!This!research!draws!from!patent!applications,!which!

reflect!the!ambitious!thinking!of!companies’!research!and!development!teams.!To!be!sure,!the!

fact!that!a!company!has!applied!to!patent!a!concept!does!not!mean!that!they!will!ever!

implement!it.!Patents!do,!however,!reflect!a!company’s!ambitions.!Patent!applications!for!the!

Google!Glass!and!Amazon!Kindle!seemed!outlandish!when!they!were!filed,!but!they!led!to!the!

advent!of!very!real!products!for!the!applicants.

!The!patent!applications!detailed!in!this!memo!

provide!insight!into!the!surveillance!that!is!possible!via!smart!home!devices.!

Amazon'Echo'

Amazon’s!digital!assistant,!the!Echo,!was!the!first!major!entry!into!the!home!assistant!market.!

The!voice-activated!smart!speaker!can!make!phone!calls,!control!smart!home!and!other!

internet-enabled!devices,!retrieve!search!results,!and!–!of!course!–!order!products!from!

Amazon.com.

!!Despite!initial!criticism!about!its!Orwellian!implications,!the!Echo!has!achieved!

!http://time.com/4906071/alexa-echo-siri-google-home/!!

!https://www.nytimes.com/2016/08/08/technology/amazon-has-big-plans-for-alexa-runnin g-th e-star-trek-

home.html!!

!https://www.washingtonpost.com/news/the-switch/wp/2017/10/10/the-go ogle-home-mini-is-the-cheapest-

way-to-get-into-th e-smart-home/?utm_term=.f2e01a925bad!!

http://www.slate.com/blogs/future_tense/2017/10/05/google_s_home_speakers_are_engineered_to_lock_you

_into_google_s_ad_empire.html!!

!https://techcrunch.com/2013/02/21/new-google-glass-patent-is-the-most-comprehensive-yet-for- go ogles-fac e-

based-wearable-computer/,!https://techcrunch.com/2013/02/21/new-google-glass-patent-is-the-most-

comprehensive-yet-fo r-googles-face-based-wearable-computer/!!

!https://www.amazon.com/dp/B07456BG8N!!

widespread!commercial!success.

!Recent!estimates!suggest!that!more!than!20!million!units!

have!been!sold.

The!Echo!works!by!listening!for!a!“wakeword”!(usually!“Alexa,”!the!name!of!the!personal!

assistant!software!that!runs!on!the!Echo).

!Once!the!wakeword!is!spoken,!the!Echo!transitions!

from!what!Amazon!calls!a!“passive!listening!state”!to!a!“responsive!state.”

!In!passive!listening!

state,!the!Echo!captures!and!analyzes!all!speech!that!happens!nearby,!but!it!does!not!

permanently!record!this!speech!or!transmit!it!back!to!its!servers.

!Once!the!Echo!detects!that!

someone!said!its!wakeword,!it!enters!a!responsive!state,!in!which!records!the!audio!that!

follows!and!sends!it!to!Amazon’s!servers!for!processing.

!It!is!there!that!Amazon’s!powerful!

natural!language!processing!algorithms!determine!whether!the!user!has!asked!Alexa!to!order!a!

pizza!or!dim!the!lights.!Amazon!claims!that!it!only!stores!and!analyzes!audio!spoken!following!

an!Echo’s!wakeword.

When!users!set!up!a!new!

Echo,!they!must!click!

through!a!screen!that!

functions!as!consent!to!

data!collection.!Amazon’s!

privacy!policy!states!that!

the!company!retains!

acoustic!voice!signatures,!!

voice!queries,!music!

played!through!the!

speaker,!and!Alexa!to-do!

and!shopping!lists.

Users!can!visit!a!page!on!

Amazon’s!site!where!the!company!keeps!a!complete!history!of!the!user’s!queries.!These!

queries!can!add!up!to!valuable!information!for!Amazon.!The!same!sophisticated!algorithms!that!

Amazon!uses!to!parse!queries!from!unstructured!data!(like!speech)!can!also!be!used!for!what!

the!company!calls!“intelligent!advertising:”!

!https://www.bloomberg.com/features/2016-amazon-echo/!!

!https://www.voicebot.ai/2017/10/27/bezos-says-20-million-amazon-alexa-devices-sold/!!

!http://pdfaiw.uspto.gov/.aiw?docid=20170090864!!

!https://www.wired.com/2016/12/alexa-and-google-record-your- vo ice/!!

!https://www.amazon.com/gp/help/customer/display.html?nodeId=201602230,!

https://www.amazon.com/gp/help/customer/display.html?nodeId=201809740!

“An!additional!way!that!more!relevant!advertising!can!be!delivered!on!systems!powered!

by!embodiments!of!the!present!invention!is!via!intelligent!sponsorship!based!on!

knowledge!generated!by!reference!to!the!human!being!entity!using!the!system.![…]!For!

example,!a!user!with!a!young!child!whose!birthday!was!coming!up!in!the!next!few!days!

could!be!presented!with!a!highly-tailored!intelligent!message!directing!him!to!a!local!

bakery!which!sells!birthday!cakes.”

Although!Amazon!claims!that!it!only!saves!audio!of!speech!immediately!following!the!Echo’s!

wakeword,!a!2014!patent!application!suggests!that!it!could!also!log!a!list!of!keywords!spoken!

while!the!Echo!is!in!a!passive!listening!state.!!The!patent!application!for!“Keyword!

Determinations!from!Voice!Data”!describes!a!system!that!listens!for!not!just!for!wakewords!but!

also!for!a!list!of!words!that!indicate!statements!of!preference.

!Algorithms!described!in!the!

patent!translates!the!following!statements!into!keywords,!and!transmits!keywords!back!to!a!

remote!data!center.

!By!only!transmitting!keywords!stripped!of!context,!Amazon!could!collect!

marketing!data!from!the!Echo!while!it!is!in!passive!listening!mode!without!breaking!its!promise!

to!only!collect!and!store!audio!following!the!device’s!wakeword.!

!http://pdfaiw.uspto.gov/.aiw?docid=20140351281!!

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131!

!http://pdfaiw.uspto.gov/.aiw?docid=2 0 1 4 03 3 7 13 1 !!

The!patent!describes!a!“sniffer!

algorithm”!that!listens!for!

“trigger!words.”!In!the!patent!

application,!Amazon!describes!

trigger!words!as!“a!verb!

indicating!some!level!of!desire!or!

interest!in!a!noun!that!follows!

the!trigger!word!in!a!sentence.”

Trigger!words!can!be!positive!

(“love,”!“enjoy,”!“bought,”!etc.)!

or!negative!(“hate,”!“dislike,!

“returned,”!etc.).!Once!the!sniffer!

algorithm!detects!that!a!trigger!

word!has!been!spoken,!the!

device!can!record!the!trigger!

word!and!the!audio!that!follows.!!

Unlike!wakewords,!which!prompt!

the!Echo!to!immediately!send!

recorded!audio!to!Amazon’s!

server!for!processing,

!trigger!

words!can!prompt!the!system!to!

process!the!phrase!following!a!

trigger!word!locally!on!the!

device.

!!The!patent!describes!an!

“audio!processing!algorithm”!that!

parses!the!captured!speech.!The!

algorithm!translates!the!audio!to!text,!analyzes!“close!words”!for!context,!removes!words!that!

are!not!informative,!and!assigns!a!positive!or!negative!value!to!the!keyword.!!

The!audio!processing!algorithm!can!also!associate!keywords!with!specific!individuals:!!

“In!embodiments!where!processes!can!attempt!to!determine!keywords!for!multiple!

users,!and!where!data!before!trigger!words!are!analyzed!as!well,!a!process!might!also!

identify!the!word!"I"!as!an!indicator!of!the!user!that!should!be!associated!with!that!

keyword.!For!example,!if!the!sentence!had!instead!been!"Jenny!loves!to!ski"!then!that!

process!might!associate!the!keyword!"ski"!with!user!Jenny!(if!known,!identified,!etc.)!

instead!of!the!user!speaking!that!sentence.”

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131!!

!http://pdfaiw.uspto.gov/.aiw?Docid=20170236512!!

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131!!

!http://pdfaiw.uspto.gov/.aiw?docid=20140337131!

Illustration)from)Patent)Application)14/447487,)“Keyword)Determinations)

from)Voice)Data”

)

In!the!examples!above,!the!

audio!processing!algorithm!

identifies!the!subject!of!a!

sentence!spoken!by!a!single!

individual.!The!patent!

makes!clear!that!it!could!

also!parse!a!conversation!

between!two!speakers!and!

associate!keywords!with!

each.!If!a!person!conducts!a!

phone!call!through!her!Echo!

or!another!device!using!this!

technology,!the!audio!

processing!algorithm!could!

analyze!both!sides!of!the!

call!and!record!keywords!for!

both!users.!

While!the!patent!for!

keyword!determinations!

allows!for!the!possibility!of!

sending!audio!back!to!

remote!servers,!the!most!

thoroughly!discussed!

implementations!of!the!

technology!would!only!

transmit!keywords,!

sentiment,!and!their!

associated!users!back!to!

remote!servers.!The!device!could!transmit!keyword!data!in!near!real-time,!at!periodic!intervals,!

or!when!the!device!transfers!data!for!another!purpose.!The!patent!also!notes!that!the!device!

could!transmit!location!data!alongside!keywords!in!order!to!further!tailor!product!

recommendations.!!

Amazon!does!not!obscure!the!fact!that!the!purpose!of!this!technology!is!advertising!and!

product!recommendations:!

“Once!keyword!data!is!stored!for!a!user,!that!keyword!data!can!be!used!to!determine!

and/or!target!content!that!might!be!of!interest!to!that!user.![…]!In!this!example,!

the!advertising!entity!located!the!keyword!"wine"!associated!with!the!user!and,!based!

on!any!appropriate!criteria!known!or!used!for!such!purposes,!selected!an!ad!relating!to!

wine!to!display!to!the!user.!Similarly,!a!provider!of!an!electronic!marketplace!which!the!

user!is!accessing!has!selected!a!number!of!different!product!recommendations.”!

Illustration)from)Patent)Application)14/447487,)“Keyword)Determinations)

from)Voice)Data”)

If!the!advertiser!or!online!marketplace!(Amazon)!has!access!to!information!about!a!user’s!

friends,!it!can!use!the!keyword!data!that!it!has!derived!from!two-way!conversations!to!market!

to!people!who!do!not!own!devices!equipped!with!keyword!determination!technology.!If!you!

know!someone!with!an!Amazon!Echo,!they!may!be!able!to!use!your!conversations!with!that!

person!to!tailor!ads!to!you.!!

Amazon!has!also!developed!technology!to!identify!different!users!in!one!home.!Another!patent!

application!describes!using!voice!signatures!and!behavior!to!distinguish!between!members!of!a!

household.

!The!application!describes!a!“master!association!table”!stored!on!Amazon’s!servers!

that!“may!include!information!associated!with!individual!users,!households,!accounts,!etc.!that!

interact!with!the!system.”

!This!could!help!Amazon!determine!whether!to!advertise!birthday!

cake!to!your!spouse!or!to!your!six-year-old.!

Google'Home'and'Associated'Products'

Google!debuted!its!Echo!competitor,!the!Google!Home,!in!2016.

!As!the!name!suggests,!

Google!intends!for!the!smart!speaker!to!act!as!a!hub!for!a!variety!of!other!smart!home!

products.

!The!Google!Home!currently!works!with!Google’s!Nest!line!of!smart!thermostats,!

cameras,!and!hazard!alarms,!as!well!as!products!from!Samsung’s!SmartThings!line!and!Phillips’!

smart!lightbulbs.

!Google’s!patent!filing!history!shows!that!the!Google!Home!is!a!step!toward!

the!company’s!eventual!goal!of!collecting!data!from!every!room!of!the!home.!!

!http://pdfaiw.uspto.gov/.aiw?docid=20170236512!!

!http://pdfaiw.uspto.gov/.aiw?docid=20170236512!

!https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo -a-face-off-of-

smart-speakers.html!!

!https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo -a-face-off-of-

smart-speakers.html!!

!https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo -a-face-off-of-

smart-speakers.html! !

Illustration )fro m)Patent)Ap p lic a tio n)14/639750,)“Smart-Home)Automation)System)that)Suggests)or)Automatically)

Implem en ts )S ele c ted )Household )P o lic ies )B a s ed )o n )S e n se d )O b s er va tio n s .)

The!Google!Home!works!in!much!the!same!way!as!the!Amazon!Echo.!The!voice-activated!

speaker!is!always!listening!for!its!wakeword!(which!Google!calls!a!“hotword”),!but!it!allegedly!

only!records!and!transmits!voice!data!following!an!utterance!of!that!word.

!However,!a!

recently!disclosed!bug!in!the!Google!Home!Mini!caused!a!number!of!devices!to!randomly!

activate!and!record!conversations!even!when!users!did!not!intend!to!activate!the!device.

Google!claims!that!it!has!fixed!the!bug,!but!the!incident!shows!just!how!easy!it!is!to!transmit!far!

more!data!than!intended.!!

Like!the!Echo,!the!Google!Home!is!equipped!with!a!proprietary!digital!assistant!software.!

Google’s!digital!assistant!is!called!“Google!Assistant”!and!users!activate!it!by!saying,!“Ok!

Google.

!Like!Alexa,!Google!assistant!can!order!pizza!and!perform!web!searches,!but!it!also!

performs!an!array!of!functions!based!on!“knowledge”!drawn!from!users’!activity!on!other!

Google!platforms.

!!!

!https://www.wired.com/2016/12/alexa-and-google-record-your- vo ice/!!

!https://www.theverge.com/2017/10/10/16456050/google-home-mini-always-recording-bug!!

!https://assistant.google.com/!!

!https://www.theverge.com/2017/5/17/15651246/google-assistant-iphone-ai-fu ture - in t erf ac e - io -2017!!

The!Google!Assistant!makes!inferences!based!users’!web!browsing!histories,!email!contents,!

and!location!data!from!Google’s!mobile!apps.

!A!patent!application!for!Google’s!voice!

recognition!system!says!that!it!draws!from!“data!including!a!user's!geographic!location,!a!user's!

search!history,!user's!interests,!or!a!user's!activity”!to!

determine!the!appropriate!answer!to!a!spoken!

query.

!Google!uses!past!voice!queries!to!target!ads!

and!train!its!system!to!provide!more!tailored!answers!

in!the!future.

When!users!set!up!a!new!Google!Home,!they!must!

click!a!button!to!accept!Google’s!Terms!of!Service!and!

Privacy!Policy.!Those!documents!give!Google!broad!

license!to!use!any!data!that!passes!through!its!system!

to!profile!and!target!ads!to!users.

!The!set-up!screen!

for!Nest!devices!contains!a!similar!screen!that!will!not!

let!users!proceed!until!they!accept!the!company’s!

Terms!of!Service!and!Privacy!Statement.!Nest,!an!

Alphabet!subsidiary,!has!a!separate!Privacy!Statement!

that!purports!to!grant!users!more!control!of!the!data!

that!the!company!collects!and!shares!with!others.

The!Privacy!Statement!does!not,!however,!indicate!

whether!Google!constitutes!a!third!party!for!

information!sharing!purposes,!nor!does!it!clarify!

whether!the!policy!also!applies!to!internal!processing!

of!user!data!for!ad!targeting!purposes.

The!Google!Home!and!connected!smart!home!devices!

allow!Google!to!gather!new!types!of!information!

about!its!users.!Like!Amazon,!Google!has!filed!patent!

applications!for!its!method!of!identifying!and!profiling!different!speakers!based!on!captured!

audio.

!Google!can!supplement!these!profiles!with!data!that!it!gathers!from!the!sensors!and!

microphones!on!its!smart!home!products.!These!products!can!capture!noise!signatures,!

moisture!levels,!subtle!temperature!changes,!light!levels,!and!other!data!that!indicate!what!is!

happening!inside!the!home.!Dozens!of!patent!applications!for!Google!products!cheerfully!

announce!that!the!company’s!smart!home!devices!may!be!used!to!draw!“inferences”!about!

their!users:!

!https://www.theverge.com/2017/5/17/15651246/google-assistant-iphone-ai-future-interfa c e-io-2017!!

!http://pdfaiw.uspto.gov/.aiw?docid=20170193999!!

!https://www.google.com/intl/en-US_us/policies/technologies/pattern-recognition /,!

https://support.google.com/assistant/answer/7126196?co=GENIE.Platform%3DAndroid&hl=en!!

!https://www.google.com/policies/privacy/!!

!https://nest.com/legal/privacy-statement-for-nest-products-and-services/!!

!https://nest.com/legal/privacy-statement-for-nest-products-and-services/!

!http://pdfaiw.uspto.gov/.aiw?docid=20160293167,!http://pdfaiw.uspto.gov/.aiw?docid=20170110144!!

“[S]uch!inferences!may!include!when!are!occupants!home,!when!are!they!sleeping,!

when!are!they!cooking,!when!are!they!in!the!den!watching!television,!and!when!do!they!

shower.!The!answers!to!these!questions!may!help!third-parties!benefit!consumers!by!

providing!them!with!interesting!information,!products!and!services!as!well!as!with!

providing!them!with!targeted!advertisements.”

Many!other!patents!for!Google’s!smart!home!products!contain!boilerplate!describing!an!

“advertising/communication!paradigm![…]!that!estimates!characteristics!(e.g.,!demographic!

information),!desires!and/or!products!of!interest!of!a!user!based!on!device!usage.”

One!patent!application,!ironically!named!“Privacy-Aware!Personalized!Content!for!the!Smart!

Home,”!goes!even!further:!

“In!some!embodiments,!the!robot!69!may!use!its!sensors!to!measure!sounds!and!

appearance!of!people!and/or!objects!in!the!room!and!send!audio!and/or!visual!features!

that!can!be!used!to!assess!properties!of!the!objects!and/or!people.!For!example,!for!

people,!the!audio!and/or!visual!features!may!be!used!to!ascertain!gender,!age,!fashion-

taste,!style,!mood,!known!languages,!preferred!activities,!and!so!forth.”

The!application!goes!on!to!describe!marketing!sports!camp!to!a!15-year-old!boy!holding!a!

basketball!in!the!living!room,!combining!browser!search!history!with!an!image!on!a!user’s!t-

shirt!to!infer!an!interest!in!the!actor!Will!Smith,!and!recommending!a!TV!program!based!on!the!

book!on!a!user’s!bedside!table.

!See!e.g.!http://pdfaiw.uspto.gov/.aiw?Docid=20160260135!!

!See!e.g.!http://pdfaiw.uspto.gov/.aiw?Docid=20170300965!!

!http://pdfaiw.uspto.gov/.aiw?Docid=20160260135!!

!http://pdfaiw.uspto.gov/.aiw?Docid=20160260135!

Illustration )fro m)Patent)Ap p lic a tio n )1 4 /6 3 89 3 7 ,)“P riv a c y-Aware)Personalized)Content)for)the)Smart)Home.”)

If!that!were!not!Orwellian!enough,!another!patent!application!describes!using!a!smart!home!

system!to!monitor!and!control!screen!time,!hygiene!habits,!meal!and!travel!schedules,!and!

other!activity.!The!system!even!purports!to!“infer!mischief”!based!on!audio!and!motion!sensor!

readings!from!rooms!where!children!are!present.

!Even!this!patent!application,!which!

describes!a!system!with!near-total!insight!into!every!activity!in!users’!homes,!contains!a!proviso!

that!device!usage!data!may!be!used!for!advertising.!!

!http://pdfaiw.uspto.gov/.aiw?docid=20160259308!!

Google’s!smart!home!patent!

applications!also!describe!the!

company’s!interest!in!users’!health!

data.!Several!applications!describe!

creating!an!audio!signature!for!the!

master!bathroom!that!will!record!the!

number!of!times!that!a!toilet!is!

flushed.

!Another!application!

describes!monitoring!a!user’s!

nighttime!breathing!patterns!to!

detect!a!sleep!disorder,!and!storing!

the!information!in!a!user’s!“sleep!

profile.”

!While!these!patent!

applications!do!not!describe!systems!

for!advertising!to!users!based!on!this!

information,!Google!has!

demonstrated!a!willingness!to!

advertise!to!users!based!on!health!

data!derived!from!other!products.!

For!example,!a!2015!patent!describes!

a!scheme!to!advertise!vacations!to!

individuals!whose!heart!rates!

indicate!that!they!are!under!stress.

Google’s!vast!repository!of!information!about!its!users!has!made!the!company!the!single-

biggest!recipient!of!digital!advertising!revenues.

!True!to!form,!Google!has!applied!for!a!patent!

to!bring!its!ubiquitous!display!ad!auction!model!to!its!voice-controlled!assistant.

!The!patent!

describes!a!system!by!which!advertisers!can!bid!to!have!their!customized!voice!actions!inserted!

into!responses!to!Google!Home!users’!queries:!

“In!one!example,!the!voice!input!of!"find!restaurant!near!Mountain!View,!Calif."!may!

cause!Little!Italy!to!appear!at!the!top!of!the!list!of!restaurants!presented!on!user!device!

104.!In!another!example,!the!voice!input!of!"find!restaurant!near!Mountain!View,!Calif."!

may!cause!a!message!to!be!presented!to!the!user!suggesting!that!the!user!try!Little!Italy!

before!the!list!of!restaurants!is!presented.!“

!http://pdfaiw.uspto.gov/.aiw?Docid=20170263111!!

!http://pdfaiw.uspto.gov/.aiw?Docid=20170127980!!

!http://pdfaiw.uspto.gov/.aiw?Docid=20170112445!!

!https://www.emarketer.com/Article/Microsoft-Surpass-Yahoo-Global-Digital-Ad-Market-Share-This-

Year/1011012!!

!http://pdfaiw.uspto.gov/.aiw?docid=20160180848!!

Illustration)from)Patent)Application)14/639750,)“Smart-Hom e)

Automation)System)that)Suggests)or)Automatically)

Implements)Selected)Household)Policies)Based)on)Sensed)

Observations. )

This!makes!the!Home!a!moneymaking!prospect!for!Google!in!two!respects:!It!not!only!gathers!

data!to!serve!more!targeted!ads!on!users’!phones!and!computers,!but!it!also!creates!another!

platform!on!which!to!serve!paid!content.!!

Implications'

The!digital!advertising!of!the!last!two!decades!targeted!ads!primarily!on!the!basis!of!

demographic!information!and!interests!supplied!by!the!user!through!search!queries,!email!

contents,!and!account!information.!Modern!digital!advertising!focuses!on!inferred!behaviors,!

contexts,!and!emotions!as!better!indicators!of!what!consumers!are!likely!to!buy.

!When!

consumers!are!at!their!computers,!companies!like!Google!and!Amazon!observe!their!behavior!

and!emotions!through!browser!cookies!that!track!the!websites!that!they!visit.

!When!

consumers!are!on!their!mobile!phones,!Google!and!Amazon!can!observe!their!behavior!and!

surroundings!through!location!data.

Digital!assistants!and!smart!home!devices!allow!companies!like!Google!and!Amazon!access!to!

the!one!place!where!users!are!likely!to!spend!time!disconnected!from!their!devices:!home.!!

These!patents!show!that!smart!devices!target!moments!in!between!screen!time!to!monitor!

sleep!habits,!listen!in!on!dinner!conversations,!and!track!when!users!shower.!Access!to!this!data!

can!flesh!out!Google!and!Amazon’s!profiles!of!their!users!in!order!to!help!them!more!accurately!

server!targeted!ads.!Smart!devices!also!offer!a!new!advertising!platform!in!themselves!–!a!2013!

letter!from!Google!to!the!Securities!and!Exchange!Commission!predicted!that!in!a!few!years,!

the!company!could!serve!ads!to!refrigerators,!thermostats,!and!other!everyday!devices.

While!tech!companies!are!primarily!interested!in!commercial!uses!for!home!data,!creating!a!

repository!of!information!about!users’!private!lives!raises!the!near-certainty!that!others!will!

attempt!to!access!it.!Home!insurers!and!utility!companies!have!already!made!deals!with!Nest!to!

put!smart!devices!in!their!customers’!homes.

!Other!businesses!that!would!benefit!from!home!

data,!like!lenders!and!health!insurers,!could!be!next.!!!

Law!enforcement!has!already!found!a!use!for!smart!devices.!An!Amazon!Echo!made!headlines!

last!year!when!police!investigating!a!murder!sought!to!subpoena!recordings!made!by!the!

device.

!Investigators!in!the!same!case!also!managed!to!obtain!data!from!a!smart!water!meter!

!http://adage.com/article/digitalnext/bem-a-approach-audience-targeting/3 08 38 3/!!

!https://www.amazon.com/gp/help/customer/display.html?nodeId=202075050,!

https://www.google.com/policies/technologies/types/!!!

!https://www.google.com/policies/privacy/example/may-collect-and-process-information .h tml,!

https://www.amazon.com/gp/help/customer/display.html/ref=hp_terms_us?nodeId=2016042 00! !

!https://www.sec.gov/Archives/edgar/data/1288776/000128877613000074/filename1.htm!!

!https://nest.com/insurance-partners/,!https://nest.com/energy-partners/!!

http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE

_EXPECTATION_OF_PRIVACY_FINAL.pdf!!

that!suggested!that!the!crime!scene!had!been!hosed!down!before!police!arrived.

!In!2015,!

Popular!Mechanics!reported!that!police!had!requested!data!from!Nest!cameras!on!multiple!

occasions,!though!it!is!unclear!whether!Google!has!complied!with!the!requests.

Hackers!or!other!malicious!actors!could!also!access!to!user!data!through!their!smart!devices.!

Google!Home’s!FAQ!contains!the!following!disclaimer:!

“Anyone!who!is!near!your!Google!Home!device!can!request!information!from!it,!and!if!

you!have!given!Google!Home!access!to!your!calendars,!Gmail!or!other!

personal!information,!people!can!ask!your!Google!Home!device!about!that!information.!

Google!Home!also!gets!information!about!you!from!your!other!interactions!with!Google!

services.”

A!recently-disclosed!Bluetooth!vulnerability!could!allow!hackers!to!completely!take!over!both!

the!Echo!and!the!Google!Home,!allowing!access!to!all!private!information!that!passes!through!

the!device.

!Another!recent!study!showed!that!some!smart!devices!are!so!insecure!that!

hackers!can!use!them!like!intercoms!to!talk!to!children.

Digital!assistants!and!other!smart!devices!also!raise!thorny!issues!surrounding!the!

constitutionally-guaranteed!reasonable!expectation!of!privacy.!Guests!to!Echo-!or!Google!

Home-enabled!homes!may!not!necessarily!consent!to!being!recorded.!Legal!scholars!argue!that!

owners!of!a!smart!home!device!should!have!a!duty!to!disclose!the!presence!of!the!device!to!

guests.

!Nest’s!terms!of!service!explicitly!place!liability!for!failure!to!obtain!consent!before!

recording!on!the!users!of!its!devices.

!In!practice,!however,!smart!device!owners!are!unlikely!to!

explicitly!warn!their!dinner!guests!or!trick-or-treaters!that!they!might!be!recorded.!!

Digital!assistants!like!the!Amazon!Echo!and!the!Google!Home!greatly!expand!the!collection!of!

personal!data,!magnifying!the!risk!that!someone!will!learn!something!about!you!that!you!would!

rather!keep!private.!!

http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE

_EXPECTATION_OF_PRIVACY_FINAL.pdf!!

!http://www.popularmechanics.com/technology/security/a14171/dropcam-law-enforcement-video-requests/!!

!https://support.google.com/googlehome/answer/7072285?hl=en!

!https://www.armis.com/blueborne-cyber-threat-im p a cts - am azon-echo-google-home/!!

!https://www.theguardian.com/technology/2017/nov/14/retailers-urged-to-withdraw-toys-that-allow-hackers-

to-talk-to-ch ildren!!

http://www.americancriminallaw re vie w .c o m/files/5114/ 9 51 5 /4 1 8 8/ A LE X A _ AND_THIRD _ P A R T IES _ R E A SONABLE

_EXPECTATION_OF_PRIVACY_FINAL.pdf!!

https://nest.com/legal/terms-of-service/