a» United States cz Patent Application Publication co, pub. No. US 2023/0245657 Al LIANG et all (a3) Pub. Date: Aug. 3, 2023 (64) KEYWORD DETECTION USING MOTION Gut 2506 (2006.01) SENSING G01 15706 (2006.01) GOL 15/183 (2006.01) (71). Applicant: Apple Ine., Cupertino, CA (US) G01 1518 (2006.01) GU6E 3716 (2006.01) (72) tnventors: Eddy Zexin LIANG, Seattle, WACUS! (52) US CL Madhu CHINTHAKUNTA, Ssratoga, ce Gte1 1825 (2013.00); GF 20n1 caus) (2013.01); G20L, 1522 2013.01), GIOL 2576 (2013.01); 6101. 15/063 (2013.01); GI0L 18/183 (201501); GIOL 181818 (2013.01) GU6F 3/168 (2013.01); GIOL 2015223 2013.01) (21) Appl. Now 18/103,427 (22) Filed: Jan, 30, 2023 Related US. Application Data (60) Provisional application No, 68/305.535, filed on Peb on ABSTRACT Systems and processes for operating an intelligent auto 1, 2022 rated assistant are provided, Ia one embodiment, data is i received from motion sensor, for instance, recording the Nee erie) motion of a user as the user utters a spoken input A 6) mec. determination is made whether a portion ofthe motion data GIOL 1525 (2006.01) matches reference data fora set of one or more words (eg. GOOF 301 G10L 1522 006.01) f word or phrase) Iso, a task associated with the one oF (2006.01) ore words i nite in response 108 Sytem Serve Sr ‘0 Dita Assistant Sever 112 11 Prosessng Modes baat Sot WO tetra ta ES i 1 Nagaton Serves) essing Seniests) aledar inerron Serves Senices) Enema ServeesUS 2023/0245657 Al Aug. 3, 2023 Sheet 1 of 14 Patent Application Publication aang 2BDUIED i i seoiag feUIEIK3 0 De HaW O) wai yao ancy on au Januag queIsissy FeN5IG wr Sot wash sang waisks eonPatent Application Publication Aug. 3, 2023 Sheet 2 of 14 US 2023/0245657 AI Memory 202 Portable Multifunction Device 200 (Operating System L~ 226 [Applications (continued) po 238 [Communication Module L228 | frtap Module oar [Contactfotion Module [-20 | i i g 232 | [Browser Module 28 [Graphics Module 1533 | [Calendar Module 249 Haptic Feedback Module + aq | |WidgetModules 1 209.4 etree 238 [Weather Widget() Pr vso2 ule Digi Assist Oiontodue 17 >re annie ta Les gi [231 [Calculator Widget ae lUser Data and Models 236 [Alarm Clock Widget im [Applications ie z Tae 237 [Dictionary Widget stags Noe {238 249-6 [Felephone Module [7230 User Created Widgets) >. |Video Conference Module 240 | [Widget Creator Module Ln [E-mail Client Module oat | [Search Module L251 linstant Messaging Module a leven Wes oe sets ee [7-242 | [Online Video Module 2a | [Camera Module 1 944 |DeviceiGiobal internal State bX limage Management Module} 95, 262 Nido & Music Player Module |7~>,, [Ewer Sistem} JNotes Module Ke ‘extemal Port} 218 RF Circuitry —— pan 203 28 8 Speaker 204 1] : 203 | “Ratio =a 2m —T] _ Corvoler Circuitry 8 | eioerats | | ao, = iomphone Interface Proximity me? sor ie 220 7] Provessor(s) ° ‘eoelerometers) 2065 205 = WO Subsystem Display Optical intensity Haptic Other Input Controle 256 |] _Sensor(s) Sensor(s) Feedback |} Controler(s) 7 Controter 258 | | Controller 259 | | Controller 261 260 Yo 203 Fin Fas Fm Fos Touch Optical Contact Tactile Output |] Other Input _ Sensitve Sensor(s) Intensity 1] Generator(s)_ |] Conto! Devices Disa Sytem 264 ae, 267 FAG. 2AUS 2023/0245657 Al Aug. 3, 2023 Sheet 3 of 14 Patent Application Publication ‘Rianjeg wang a7 Dd eeperey 7 wens Twang \ ‘SUORIUYEG BAZ Beg wong, ee ‘Ig ewe voneondy 062 seIpueH wang B__ Jeep ino Zz s8¥e86n w9I90 3H -aepdn eed sojesed09 DBZ sezwbooay eng 7oajeoey Wen O62_seppuey wong FOzIU6009y WAAT DBZ sezwBooay wena ‘TSZ warp vogeonddy we | ‘npoyy seyoiedsiq wang ee INDO UORRURLIA}OG serjtiovey W943 ang wu | ‘aIMPOWY UOREUMLIOTIG MANA HH iwt| Joyo eH 4 JeuOS Wang TZ maip uoneoyddy uopeaddy © sore uoneansdyPatent Application Publication Aug. 3, 2023 Sheet 4 of 14 US 2023/0245657 Al Portable Multifunction Device 200 310) Speaker 211 300 310s SIM card slot | 312is headphone jack Touch Screen 2 ¢ Contact intensity) Sensor(s) 28 7 “Tactile Output ~ y '\ Generator(s) 267.) ‘Microphone Home ‘Accelerometer(s} 28 304 268 [__ Extemal Port 228 FIG, 3Patent Application Publication Aug. 3, 2023 Sheet 5 of 14 US 2023/0245657 Al [Operating System [x 226 Memory 470-— [Communication Medule L- 7 (ContactMotion Module oo Graphics Module x3 Haptic Feedback Module a [Fext Input Module L- ba Digital Assistant Client Module Lb User Data and Models JApplications 37 [Contacts Module a a Device 400 [Telephone Module 29 \ [Video Conference Module oan E-mail Cent Module Bm instant Messaging Module os 410 cuts) Workout Support Module 043 [Camera Module aon 420 image Management Module a7 Ee [Browser Module 248 ES rig - Weather Widget — {VO interface eater He rr 22 Display = A 440 [Calculator Widget aoe [Alarm Clock Widget 405 Keyboardiiiouse H— 450 Dictionary Widget 1 [Tat Has lene Tactile Output [Widget Creator Module 1 951 Generators) t~ 457 [Search Module 252 [video & Music Player Module 1” jon Sensor(s)__}}—- 459 Drawing Module 482 Presentation Module 1 ‘eh TENGE [Word Processing Module ae 48Q_| Communications [Website Creation Module 188 Interface Disk Authoring Module a Spreadsheet Module a FIG. 4 |DevceiGiobal internal State [57Patent Application Publication Aug. 3, 2023. Sheet 6 of 14 Portable Multifunction Device ZB 200 ™~ 20g (seaerat ) aS) 500 0 73 Weather 538 Settings 546 O00 ©00 eo0 Uiities Bod 522 Touch Serer ‘Recelerometeris) 258 FIG, 5A US 2023/0245657 AlPatent Application Publication Aug. 3, 2023 Sheet 7 of 14 US 2023/0245657 Al 550, é Ra (Tactile Output 1 1 562 \_ Sensors 459 | FIG. SBPatent Application Publication Aug. 3, 2023 Sheet 8 of 14 US 2023/0245657 Al DEVICE 600 608 os 606 FIG. 6AUS 2023/0245657 Al Aug. 3, 2023 Sheet 9 of 14 Patent Application Publication sna g9 OT WOSNAS NOLLOW ' S YOSNAS: TWNOLLOSMIG dODSOYAD SLEWOUTTEOOV] WOSNAS Sd9 NOLLV JINAWINOO| K l I si9-7] AUOWSIN (s)¥ossa00¥d] 919. $297] ALISNALNT HONOL AVTASIC ¢ us INGNI gePatent Application Publication Digital Assistant System 700 Memory 702 \ Aug. 3, 2023 Sheet 10 of 14 ‘US 2023/0245657 AI |Operating System Lx 718 |Communication Module [- 720 [User Interface Module Lr 722 _ 724 [Applications | an [Digital Assistant Module I 208 iO Processing Module } 730 ISTT Processing Module 1 758 [ASR Systems fee Natural Language Processing Module }~ 760 [Onictogy im 744 Vocabulary 748 [User Data a TA jalogue Flow Processing Module} 75. Task Flow Processing Module 1 754 [Fask Flow Models I~ 238 Service Processing Module 1 I [Service Models i L740 7 processors) [Speech Synthesis Processing Module TOL, uN 10 Interface WO Devices =H 716 moe | Network Communications 1 interface Wy Wired Communications 714 Nw Wireless Circuitry FIG. 7AUS 2023/0245657 Al Aug. 3, 2023 Sheet 11 of 14 Patent Application Publication ai DIT ‘S}aPOW = MOL ainpo| sasuodsay yse] Buiss2001g pateag sisauyukg 7 yooadg sasu0dsey 98 Sapoyy ¥5L poy Supse00ig Busse001d le—|-__» Wo} EL ap-nOHes 1 x - Busse001g Of ae AIUD 08 ‘ampoyy Buissa90id BbL vel vonewesaidey zeL pst ONBUOU suogejuasaiday 3x2 || > ‘annoy vorsianuog ‘ainpowy al L-| Pe yeqeudiy aauoud Buisse00ld 11S [pas rduy yosods tee: oes: BL qweysissy feuSigUS 2023/0245657 Al Aug. 3, 2023 Sheet 12 of 14 Patent Application Publication IL DIA 7 abuey 20d / ar 277 weueg ~ 7 epuwey | azig hued auiyelea urewog uojenesay uasneyse,Patent Application Publication Aug. 3, 2023. Sheet 13 of 14 US 2023/0245657 Al 7 808 si0a-t “HEY SIRI" nN sion] a ds ‘NEXT SONG FIG. 8BPatent Application Publication Aug. 3, 2023 Sheet 14 of 14 US 2023/0245657 Al 900-7 902 OBTAINING A FIRST LANGUAGE MODEL INCLUDE ae PLURALITY OF REFERENCE DATA FOR A RESPECTIVE FIRST PLURALITY OF ONE OR MORE WORDS i 1 PLURALITY OF SAMPLE DATA {| LANGUAGE MODEL INCLUDING A SECOND PLURALITY OF { REFERENCE DATA FOR A RESP! ‘OND PLURALITY OF t SETS OF ONE OR MORE WORDS, FIRST DATA FROM A MOTION SENSOR RECEIVING i VA DETERMINING WHETHER A PORTION OF THE FIRST DATA, MATCHES REFERENCE DATA FOR A SET OF ONE OR MORE PERFORMING A CROSS-CORRELATION BETWEEN THE, PORTION OF THE DATA AND THE REFERENCE DATA FOR THE SET OF ONE OR MORE WORDS TO DETERMINE A SET OF CORRELATION COEFFICIENTS J | ~s DETERMINING WHETHER THE SET OF CORRELATION COEFFICIENTS SATISFIES CORRELATION CRITERIA a IN ACCORDANCE WITH A DETERMINATION THAT THE PORTION OF THE FIRST DATA MATCHES THE REFERENC! DATA FOR THE SET OF ONE OR MORE WORDS, CAUSING TASK ASSOCIATED WITH THE SET OF ONE OR MORE WORDS aE TAe Lolo FIG.9US 2023/0245657 Al KEYWORD DETECTION USING MOTION SENSING ‘CROSS-REFERENCE TO RELATED APPLICATIONS, 10001] This application claims priority to US. Provisional Patent Application 63/305,535, entitled “USING ACCEL EROMETER FOR KEYWORD SENSING,” filed Feb. 1, 2022. The contents of this application are incorporated herein by reference in their entirety. FIELD 10002] This relates gencrally to intelligent automated assistants and, more specifically, recognizang voice inputs ‘sing motion sensing BACKGROUND. [0003] Intelligent automated assistants (or digital assis fants) can provide & beneficial interface between umn users and electeonic devies, Such assistants ean allow users to interact with deviees or stems using natural language in spoken andior text forms. For example, user can provide ‘speech input containing a uscr request oa digital sistant ‘operating on an electronic device. The digital assistant can Jmerpret the vser’s intent from the speech inpat and oper ‘ionalize the users intent ito tasks. The tasks ean then be performed by executing one or more services of the elee- ‘tonic device, and a relevant output responsive tthe user request can be returned to the user. [004] “An electronic device may be controlled using voice inputs detected using an audi sensor, such asa microphone incorporated in a phone, hoadset, of smart speaker. Some voice control systems require sdtional user inputs 10 sctivate, such as bolding down a buton or tapping 2 touch: seastive surface proe to speaking in order 10 begin detect- ing audio withthe dio sensor. Other voice contro systems ‘continuously detect and process aio inorder to respond 10 voice inputs vvithout the latency or user effort associated ‘with additional activation inputs. However, continaously ‘detecting and. processing audio data expends power and processing capacity even when the users not actively using Voige contol. Additionally, voice coatrol systems ean rest in false positive responses if the audio sensor picks up ‘ambient noise or specch from an unintended use. SUMMARY [0005] Example methods are disclosed herein. An ‘example method includes, tan electronic device having one fo more processors, receiving fist dats from mation sensor: determining whether a portion of the fist data matches reference data for a set of one or more words ad jin accordance with a determination thatthe portion of the first data matches the reference data for the set of one OF ore words, causing a task associated withthe set of ane oF ‘more worde to he iitinted, [0006] Example non-ansitory computer-readable media fate disclosed herein. An example notranstory computer ‘eadable storage medium stores one or mere programs, The ‘one or more programs comprise insiructions, whieh when executed by one or more processors ofan electronic deviee, cause the clecttnie device to recive fist data fom a ‘motion sensor determine whether «portion ofthe fist data mmtches reference data for a set of one o more words; and Aug, 3, 2023 in accordance with a determination thatthe portion of the fiat data matches the seference data for the set of one oF more words, cause a task associated with the set of ome oF ‘ore word to be initsted 10007] Example electronic devices are dislosed herein ‘An example electronic device comprises one oe more pro= jestors 4 memory: and one oF more programs, sere The fone of more programs are sfored in the memory and eon: ‘gure tobe executed bythe one oF more processors, the one cor more programs inching instrctions for receiving fst data froma motion sensor, determining whether portion of the fist data matches reference data fora set of one or more words; and in accordance with a determination that the portion ofthe first data matches the reference data fr the sot fone or mare words, causing a task associated withthe set ff one oF more words to be initiate. 10008] An example clectnic device comprises means for receiving fist data from a motion senso, determining ‘whether a portion ofthe fist data matches reference data for ‘set of one oF more words; and in accordance with ‘determination thatthe portion ofthe first data matches the reference dala for the set of one oF more words, easing @ task associated with the set of one or more words to be initiated. 10009] Recognizing a voice input using motion sensing provides an intuitive and eficent approsch for contelling an electronic device. When a user speaks, the user's mouth, ice, head, and neck move and vibrate, Motion sensors sch as aecelerometers and gyroscopes can detect these motions, ‘while expending relatively litle power eompared to audio Sensors such as micropbones. Additionally, motion sensors are Tess ikely than auto sensors to detect ambient sound, ‘och as another person speaking or voices from television fr stereo, By comparing motion daa 0 reference data for a eC of one or mor’ relerence words or phrases (e.. voice commands, in-vocabulary Words, or the like), a matching reference word or phrase can be determined, Accordingly, a task associated with the recognized (eg. matching) refr- fence word of phrase can be initiated based on the motion data, without nooding to detet of process audio dat BRIEF DESCRIPTION OF THE DRAWINGS 10010] FG. 1 is 2 block diagram ilustating a system and environment for implementing a digital assistant, according to various examples [0011] FIG. 24 isa block diagram illustrating a portable ‘multifunction device implementing the client-side portion of 1 digital assistant, according to various examples. [0012] FG. 2B isa block diagram illsteating exemplary components for event handling, “aeconting #0” various exannpls. 10013] FIG. 3 illustrates a portable multifunction device Implementing the cliet-side portion of a digital assistant, sccorling to varioas examples [0014] FG, 4a block diagram of an exemplary multi- Jnetion device with a display and touch-sensitive surfiee, socording to Various examples. 10015] FG, SA illustrates an exemplary wser interlace for ‘ment of applications on @ portable mullfunetion device, sccording to various examples 10016] FIG. 8 illustrates an exemplary user interlace for ‘a multfumetion devie witha toveh-tensiive surface tt is Separate from the display, according t various examples.US 2023/0245657 Al [0017] FIG. 6A ihstates a personal electronic device, ‘ocorting to various examples. [0018] FIG. 6B is block diagram illustrating a personal clectronie device, according 10 various examples. [0019] FIG. 7A is a block digram ilusteaing 2 digital ‘sistant system or a server portion thereof, according 10 various examples [0020] FIG. 72 illuswates the funetions of the digital assistant shown in FIG. 7A, according to various examples [0021] FIG. 7C ilusirates @ portion of an ontology, secording to various examples. [022] FIGS. 84-88 ilustrte systems for recognizing voice inputs sing motion sensing, according to various examples. [0023]. FIG. 9 illustrates 2 Now diagram for a method for recognizing voice inputs using motion sensing, acording 10 various examples DETAILED DESCRIPTION [0024] Inthe following deserpvion oFexamples, reference is made tothe accompanying drawings in which are shown by way of illustration specific examples that can be proc= jee, It is to be understood that other examples can be weed and sictral changes ean be made without departing from the scope ofthe various examples [0025] Voiee inputs may be recognized using motion sensing. Using data receive from a motion senso, a dete mination is made whether a portion of the received data ‘matches reference data fr st of one or more words, the portion of data matches, «task arsocated with the ct of one fF more words is initiated [0026] Aldhough the following description uses terms “irs.” “Second,” et, to deseribe various clement, these ements should not be limited by the terms. These terms are nly used to distinguish one clement from another. For ‘example, a fist input could be termed a second input, and, Similarly, 2 second input could be termed a fist input ‘without departing from the seope of the varions described examples. The frst input and the second input are both inputs and, in some eases, are separate and different inputs 1027] The terminology used in the description of the various described examples herein is forthe purpose of ‘describing particular examples only andi aot intended to be Timiting. As used inthe description of the various described examples and the appended elms, the singular Forms “s,” an.” and “the” are intended 1 ince the plural Toms 38 wel, unless the context clearly indicates otherwise. It will also be understood thatthe term “andr” as used herein refers to and encompasses any and all possible combinations ff one or more of the associated listed items. It will be Tuchee understood that the terms “includes,” “neluding, “comprises.” andor “comprising,” when used in tis specie ‘cation, specify the presence of stated features, integers, steps, operations, elements, and/or components, bt Jo act preclude the presence or addition of one or more ether Tstures, integers, steps, operation, elements, components, andor groups thereat 0028] "The term "i" may be construed mean “when” or “upon” of “in response to determining” ee “in response 10 ddetceting.” depending on the context. Simialy, the phrase “if tis determined” or “if [a stated condition or event] is detected” may be construc to mcan “upon determining” or “in response fo determining” or “upon detecting [the stated Aug, 3, 2023 condition or event)” or “in response to detecting [the stated condition or event,” Jepending on the context 1, System and Environment 10029] FG, 1 iostates a block diagram of systems 100 according (0 Various examples. In some examples, system 100 implements a digital assistant. The tems “digital assis tan" “Viral assistant.” “intelligent automated assistant.” for “automatic digital assistant” refer to any information Processing systems that interprets natural langage inp in Spoken and/or texto oem to infer user intent, and performs sctons based on the infered user intent. For example, to act ‘onan infered user inten, the system perfomns one or more of the following: identifying a task low with steps and parameters designed to accomplish the inferred user intent, ‘npuiting specific requirements rom the inferred ws intent into the tark flow: executing the task low by iavoling programs, methods, services, APIs, or the like; and gener sing ouptesponss tothe ter inan able Pest) 10030] Specifically digital assistant is capable of accept ing user request at Teast partially in the form of a natal Jagugge command, request, statement, narrative, and/or nguiy. Typically. the user request seek either an informa tional answer or perfomance of a task by the digital sssstant. satisfactory response to the wer request includes a provision ofthe requested informational answer, a perfor- ‘mance of the requested task, of « combination of the fo. For example, user asks the digital assistant a question, such as “Where am I right now?" Based on the user's current Jocation, the digital assistant answers, “You are in Central Park near the west gate.” The user also requests the perfor- mance ofa task, for example, “Please invite my friends 19 iy girlfriends birthday party next week.” In respons, the digital assisiant can acknowledge the request by saying “Yes. rght away” and then send a sitable calendar invite on behalf ef the user to each of the use's frend listed in the user's electronic adress book, During performance of reaested tsk, the digital assistant sometimes interacts with the user in a continuous dialogue involving multiple exehanges of information over an extended period of time, There are numerous other ways of interacting with a digital assistant to request information of performance of various tasks. In adtion to providing verbal responses and taking programmed actions, the digital assistant also provides responses in other visual o aio forms, eg 8 ext, sees, rnc, videos, animations, ee [0031] As shown in FIG. 1, in some examples, a digital sistant i implemented sceording to client-server mel The digital assistant includes client-side portion 102 (here alter “DA client 102") executed on user device 104 and senerside portion 106 (hereafter “DA server 106°) exectted om server system 108. DA client 102 vommuni- tes with DA server 106 through one or more networks 110 DA client 102 provides client-side functionalities such a ‘serfacing input and output processing and communication With DA server 106, DA server 106 provides serverside unetionaltes for any number of DA clients 102 each residing on a respective user device 104 0032] In some examples, DA server 106 includes elient- Tacing UO interface 112, one oF more processing modules ANd, data and models 116, and VO interface to external scrvicos 118, The cient-facing VO interface 112 facilitates the clentfacing input and output processing for DA serverUS 2023/0245657 Al 106, One or more processing modules 114 utlize data and models 16 0 process speech input and determine the user's intent based on natural language input. Further, one or more processing modules 114 perform task execution based on Inferred ser intent, In some examples, DA server 106 ‘communicates with extemal services 120 through network (6) 110 for tsk completion of information acquisition. UO Interface to extemal servis I18foclifates seh com cations 10033] User device 194 can be any’ suitable electronic Udevioe. In some examples, user device 104 is « portable ‘multfunetional device (eg, device 200, described below With reference to FIG. 24), mulifinetional device (eg, Advice 400, described below with reference to FIG. 4), or a personal eleetonic device (ea, device 60, described below ‘ith reference o FIGS. 64-68). portable multifuntional ‘device i, for example, a mobile elephone that also eoatains bother functions, such as PDA andor muse player Tanctions Spevitie examples of portable multifunction devies include the Apple Watch, iPhone, iPod Tou, and iPadto devices from Apple ine. of Cupertino, Cali, Other examples of portable multifunction devices include, without limita ton, earphonesTbeadphones, speakers, and laptop or tablet, computers. Further, in some examples, user device 104 s a fon-portable mltfusctional device, In particuls, user device 104s a desktop compute, a game console a speaker, 1 television, ora television set-op box. In some examples, tiser device 104 includes « touch-sensitive surface (ea, touch screen displays andor touchpads). Further, user device 104 optionally includes one of more ther pysical userinterface doviees, such as a physical keyboard, mouse, andior «joystick. Varows examples of electronic devices, such as) multifuetionsl devices, are described below in preater del [0034] Examples of communication network(s) 110 Include local area networks (LAN) and wide area networks (WAN), eth Internet. Communication network(s) 110 is Implemented using any known network protocol, inclading various wired or wiraless protocols, such as, for example, Fihemet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Favironment (EDGE), code division multiple access (CDMA), time division multiple access (IDMA), Bla tooth, Wi-Fi, voice over Internet Protocol (VoIP), Wie MAX, or any other suitable communication protocol [0038] | Secver system 108 is implemented on one or mone Standalone data processing apparatus ora distributed et ‘work of computers. In some examples, server systems 108 also employs various virtual devicos andlor services of third-party service providers (eg, think-paryclovd service providers) to provide the underlying computing resources andlor infastracture resources of server system 108 10036) In some examples, user device 104 communicates With DA server 106 via second user device 122, Second user device 122 is similar or identical to user device 104. For ‘example, second user device 122 is similar to devices 200, 400, oF 600 described below with reference to FIGS. 2A, 4, land 6A-6B. User device 104 is configured to communica: ‘ively couple w second user device 122 via direct com- ‘munication connection, such as Bluetooth, NFC, BILE, oF the lke, oF via a wired or wireless network, sic as Toca WiFi nctwork. In some examples, second user device 122 is configured to at as a proxy betwen user device 104 and DA server 106. For example, DA client 102 of user device Aug, 3, 2023 104 is configured to transmit information (ex, a user request received at user device 104) to DA server 106 via second ser device 122, DA server 106 processes the infor ‘mation and returns relevant data (eg, data content respon: sive to the user request to user device 104 via sccond user device 122, 10037] In some examples, user device 104 is configured 19 communicate bbreviated requests for data to second user Advice 122 to race the amount of information transmitted fom user dovice 104. Second user device 122 i configured to determine supplemental information to add tothe abbee- ated request to generate complete request to trast 10 DA server 106. This system architecture can advantageously allow user device 104 having limited! eommunication capa bilities andor fimited battery power (eg, a watch or a similar compact electronic device) to access services pro- vided by DA server 106 by using second user device 122, having grester communication capabilities andor battery poser et, a mobile phone, lapiop compurer tablet con- puter, orth lke), a a proxy to DA server 106. While only {wo user devices 104 a 122 are shown a FIG. 1, i shoul be appreciated that system 100 in some examples, includes any number and typeof user devices configured in his proxy configuration to communicate with DA server system 106, 0038] Although the digital assistant shown in FIG. 1 Includes both a client-side portion (¢, DA client 102) and a serverside portion (@£, DA server 106), in some examples, the Tunetions of @ digital assistant are imple mented as a standalone application installed on a user device, In ixition, the divisions of Fuetionalities beeen the client and server portions ofthe digital assistant can vary jn different implementations, For instance, in some examples, the DA client is a thin-lient that provides only tweer-facing input and output processing functions, and del cegates all other functionalities of the digital assistant 10 4 backend server. 2, Bletronie Devices 10039] Attention is now directed toward embodiments of clectronie devices for implementing the client-side portion of a digital assistant. FIG. 2A is a block diagram illustrating Portable multifunction device 200 with touch-sensitive dis lay system 212 in aocordance with some embodiments Toveh-sewsitive display 212 is sometimes called a “toveh screen” for convenience and is sometimes known as or called a “toucheseasitve display system.” Device 200 Jnchides memory 202 (which optionally iachides one or ‘more computer-readable stomge mediums), memory con twoller 222, one or more processing units (CPUs) 220, Peripherals interface 218, RF cirevitry 208, aio ciruitey 210, speaker 211, microphone 213, inputoutput (UO) sub- system 206, exer input control devices 216, and external port 224, Device 20 optionally includes one or more optical Sensors 264. Device 200 optionally includes one or more contact intensity sensors 268 for detecting intensity of contacts on device 200 (eg. a touch-serstve surface such touch-sensitive display “system 212 of device 200) Device 200 optionally incides one or more tactile ouput enerators 267 for generating tactile ourpus on device 200 {eas generating tactile ontpits on a touch-sensitive sure such as touch-sensitive display system 212 of deviee 200 or touchpad 488 of device 400). These components optionally communicate over one oF more communication buses oF signal fines 20.US 2023/0245657 Al [0030] As used inthe specification and clams, the term “intensity” ofa contact on a touch-sensitive surface refers the force or pressure (fore per unit area) of a contact (eg, 1 finger contact) on the touch-sensitive surface, of 10. Substitute (proxy) Toe the force oe pressure ofa contact on the touch-sensitive surface, The intensity of @enatact has 4 range oF vals that includes atleast Fou dstnet valves and ‘ore typically eludes hundreds of distinct values (6, at Teast 256), tensity ofa contact s, optionally, determined (ce measured) using various approaches and various sensors br combinations of sensors. For example, one cr more force sensors underneath or adjacent tothe touch-sensitive sures ar, optionally, used to measure force at Various poins on the touch-sensitive surface. In some implementations, force measurements from mulple force sensors are combined (ea weighted average) to determine an estimated fore of 1 contact. Similarly. a pressure-seastive tip of a stylus is, ‘optionally, sed to determine a pressure ofthe stylus on the touch-sensitive surlace. Alteraively, the sizeof the contact area detected on the touch-sensitive surface and/oe changes ‘hereto, the capacitance ofthe touch-sensitive surface proxi mate To the contact andlor changes thereto, andlor the resistance of the touch-sensitive sirface proximate to the contact andor changes thereto are, optically, use as substtme forthe force of pressure of the contact on the ‘ouch-scastive surface. In some implementations, the sub- stitute measurements for contact foree or pressure are used directly to determine whether an intensity threshold hasbeen exceeded (ez, the intensity threshold is described in nits forresponding to the subsite measurements). In some Implementations, the substitute measurements for contact force of presture are converted to an estimated force or pressure, and the estimated force or pressure is used 19 ‘termine whether an intensity threshold has been exceeded (eg. the intensity threshold is 4 pressure thresbold met sured in units of pressure). Using the intensity of contact 28 an attribute of a user input allows for user access 0 additonal device functionality that may otherwise not be aocessble by the user ona reduced-sze device with limited real estate for displaying affordances (eg., on a touch: Sensitive display) andor receiving user input (eg. via @ touch-sensitive display. a touch-sensitive surlae, ora physi- callmechanical contol such asa knob or 2 button). [0081] As used inthe specification and claims, the tem “getle ouput” refers to plisical displacement of a device relative toa previous position of the deviee, physeal dis- placement ofa component (eg. a touch-sensitive surlace) of ‘dvice relative to another component (e.g, housing) ofthe ‘device, or displacement ofthe component relative ton center ‘oftmass ofthe device that willbe detected by a user with the user's sense of touch, For example, in situations where the device or the component ofthe device isin eontact with a surface of a user tht is sensitive to toveh (ea finger, palm, of other part of # user's hand), the toile ouput tzenerated hy the physical displacement will be interpreted by the ter a8 toctile sensation corresponding to. per coived change in physical characteristics ofthe dove of the component of the device. For example, movement of touch-sensitive surface (eg, a touch-sensitive display or ‘eackpad) i, optionally, interpreted by the user as a “dovsn click” or “up click” of a physica actuator button. In some ‘cases, user wil fool a tactile sensation such as an “down Click or “up click” even when there is a0 movement of physical actuator button associated with the touch-sensitive Aug, 3, 2023 surface that is physically pressed (eg, displaced) by the tser's movements. As another example, movement of the touch-semsitive surfce is, optionally, interpreted or sensed by the user as “roughness” of the touch-sensitive surface, even when there is no change in smoothness of the touch sensitive surface, While such interpretations of touch by a ‘user wil be subject tothe individualized sensory perceptions ofthe use, there are many sensory pereptions of touch that arecommon oa large majonty of users. Ths, when a tele ‘output is described as corresponding toa particular sensory perception of @ user (ean “up clic,” a “down click” “roughness”, unless otberwise stated, the generated tactile contpat comresponds to physical displacement ofthe devie or ‘component thereof tha wll generate the described sensory perception fora typical (or average) user, 10042] Itshoukd be appreciated that device 200 is only one cexaniple of portable multifunction device, and tat devise 200 optionally has more or lower components than shossn, optionally combines two or more components, or optional has a diferent configuration or arrangement of the compo= nents. The various components shown in FIG. 2A are ‘implemented in hardware, software, or 4 combination of both hardware and software, including one or more signal processing andlor aplication-specifc integrated cir [0043] | Memory 202 includes one or more eomputer-read: able storage mediums. The eomputer-readable storage med ‘usar, for example, tangible and non-transitory. Memory 202 includes high-speed random aceoss memory and als) Jncludes non-volatile memory, such 2s one or more magnetic disk storage devices, lash memory devices, or eller non- volatile solid-state memory devices. Memory controller 222 controls access 19 memory 202 by other components of device 200, 10044] In some examples, a non-ransitory eomputer-read- able stomge medium of memory 202 i8 used to store instructions (ee, for perlonning aspecis of processes described below) for use by of in connection with an Instruction execution system, apparatus, or device, such as a computer-based. system, processorcontaining system, oF ‘ther system that can fetch the instructions from the instr tion exceution system, apparatus, or device and execute the instrtions. In other extumples, the instructions (ee, for performing aspects of the processes deserbed below) are SHored on a non-ransitory computerteadable slonige ‘dium (aot shown) ofthe server system 108 or are divided between the non-tnusitory “computer-readsble songs medium of memory 202 and the non-transitory computer: readable storage medium of server system 108 10045] Peripherals interface 248 is used 1 couple input and output peripherals of the device to CPU 220 and ‘memory 202, The one or more processors 220 run or execute various software programs andr sets oF instructions stored ‘memory 202 (0 perform various functions for device 200 and to process data. In some embodiments, peripherals Interface 218, CPU 220, and memory controller 222 are Implemented on a single chp, such as chip 204. In some other embodiments, they are implemented on separate chips [0046] RF (radio frequency) circuitry 208 receives and fends RF signals, also called eloctomagnetie signals, RF circuitry 208 converts electrical signals to/from eletromag- netic signals and communicates with communications net works and other communictions devices via the clecro- ‘magnetic signals. RF circuitry 208 optionally includes wel- known eireuitry for performing these functions, includingUS 2023/0245657 Al bc not inte an antenna system, an RF transceiver, one crmmore amplifiers, uae, one oF sar seit, digit Slgnal processor CODEC chipst subseber ideality tmodule (SIM) car, memory, and so Forth. RE ciety 208 ionally communicates wih newer, such the Inte: flan refer to as the World Wide Web (WWW), 28 inanet andor a witeless network, suchas a cea ele phone network, wiles lea area network (LAN) aor 8 metropolitan area network (MAN), and other devies by Wwirelest communication, The RF crt 208 optionally includes well-known eeu for detecting nea Hel com ‘usicaton (NFC) lds, sucha by a shor-apge comm cation ado. The wisest commniation optionally cs any of pry of communications andar, potas, and ecologies incling but mot intel o Glob System, for Mobile Communications (GSN), Fahaneed Data GSM Environment (EDGE) high-speed dowalink packet access (CISDPAY, high-speed uplink pocket acess (HSDPA), Ev lotion, DatrOnly (EV-DO), HSPA. HSPAs, Dual FHSPA (DC-HSPDA), long tc evolution (CTE), nee Held communication (NPC), wideband code dvs lite access (W-CDMA), code vison muiple access (CDMA), time division multiple seces (TDMA), Bluetooth, Bit etwoth Low Enemy (BTL), Wireless Fidelity (WEE) (ex, TSE 8021s, IEEE 802.16, IEEE 802.11, IEEE 8021, ane IEEE 802 1a), wiee ove Internet Protocol (VP), WeMAX, a protocol fore mal (eg. Item message access protacl IMAP) anor post ofce protocol POP) instam messaging (eg, extensible messaging and presence protocol (XMPP}, Sesion Tritton Proweol Tor Tastant Messaging and Presence Leversaing Extensions SIMPLE, Instant Mesuing and Presence Service (IMPS), andor Short Messe Service (SMS). cr any le suitable com nication protocol including communication protocols fot yet developed at ofthe filing date of thi document. [0087] Audio ciruiy 210, speaker 211, nd microphone 213 provide an aio interface between 8 user and device 200, Nalin cicuity 210 receives suo data fram pexph= tral interce 218, converts the ado data to an electrical Signal, and transmits the electrical signal 1 speaker 211 Speaker 21 converts the clerical inet human-uile sound waves. Audio circuitry 210 also revives electical Signals converted by microphone 213 fom sound waves Adio ezeuiey 210 covers the electrical signal o aio data and rans the aio dato pripiras ntrface 218 fr processing. Audio data are retseved rm sno trans ried 9 memory 202 andor RF circuitry 208 by pecnh- eras interface 218. In some embodiments, andi cireey 210 aso iacudes a Ieadset jack (e312, IG. 3). The ict jack provides a interface betwoon aio cicitey 210 and remable aio iapatoutpa peripheral, soc as uiitcoly hemes ra ede wit hot out (ea, ‘headphone fr one or both eas) and iat (eg, 8 mie phone) {0048} 10 subsystem 206 couples inpuoutpt periper aly om devie 20, much as touch Sreen 212 an othe input onto devies 216, o peripherals interes 218. 10 sub System 206 optionally includes display controller 256, op Gal sensor conor 258, iatensty sensor controler 259, haptic feedback controller 261, and one or moe input conrller 260 footer input or contol devices. The une ur ‘nore input contllers 260 reccivelsend cletrial signal fromto other it contol devices 216. The oer input contol davies 216 optionally include physical butions Aug, 3, 2023 (ean push burtoas, rocker buttons, ete), dials, slider switches, joysticks, fick wheels and so forth. In. some alternate cmbodiments, input contrller(s) 260 are, option- ally, coupled 10 any (or none) ofthe following: a keyboard, an infrared port, USB por, and a pointer device such as 4 mouse, The one or more buttons (€2, 908, FIG. 3) option- ally inelide an upidown bution for volume eon) of speaker 211 and/or microphone 213. The one or more buttons optionally include a push button (¢g, 306, FIG. 3). 10049] _ quick press of the push button disengages a lock ff touch sergen 212 or hegin a process that uses gestures on the touch screen to unlock the device, as desribed in US. patent aplication Ser No.11/322,549, “Unlocking a Deviee by Performing Gestures on an Unlock Image,” fled Dee, 23, 2005, US. Pat. No. 7,657,849, which is hereby incorporate by reference in its entirety. longer pres ofthe pus button (Gu 306) tums power ta device 200 on or of. The user is able to customize a functionality of one er more of the buttons. Touch sersen 212 is wsed to implement vital or soft batons and one or more salt keyboards. 0050] Tovch-sensitive display 212 provides an_ input fnterfae and an output interfce between the deve snd user Display controller 256 receives andr sed electrical signals fromito touch sersen 212. Touch sercen 212 displays ‘stl ouput tothe were visual output incidesgropt- jes, text, icons, video, and any combination thereof (colle tively termed “graphics”, In some embodiments, some or all ofthe visual output correspond to user-interface objects 10051] Touch sereen 212 has a touch-sensitive surfiee, Sensor, or set of sensors that accepts input fom the user bused on hapic andr tactile contac, Tech sreea 212 ad display controler 286 (along with any associated modules andlor sets of instructions in memory 202) detect contact (and any movement or breaking of the contaet) on touch sereen 212 and convert the detected contac into interaction with userinterface objects (eg, one oF mone soft Keys, Jcons, web pages, or images) that are displayed on touch screen 212. nan exemplary embodiment, point of contact between touch soreen 212 and the user comesponds to a finger of the user. 10052} Touch screen 212 uses LCD (gui crystal display) technology, LPD (light emitting polymer display) tecno} ogy, of LED (light emitting diode) technology, although other display technologies may be wsed in other embod ments. Touch sereen 212 and display controller 286 detect contact and any movement or breaking thereof using any of | plurality of touch sensing technologies now known or later developed, including but not limited to eapacitive, resistive, Infrared, and surface acoustic wave technologies, 38 well 3s other proximity sensor arrays oF other elements for deter mining one or mor: points of contact with touch sereen 212, Inan exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the ‘hone and iPod Touch from Apple Inc. of Cupertino, Cait 10053] touch-sensitive display in some embodiments of ‘ouch sereen 212 is analogous 0 the mul-toueh sensitive ‘ouchpods described in the Tallowing U.S, Pat. No. 6.323, 846 (Westerman ea.) U.S. Pat, No. 6,570,857 (Westerman eta), andior US. Pat No. 6,677,932 (Westerman), and/or US. Patent Publication 200200150241, cach of which is hereby incorporated by reference in its entirety. However,US 2023/0245657 Al ‘ouch sereen 212 displays visual output from deviee 200, whereas touelisensiive touebpads Jo not provide visual output 10084] A touch-sensitive display in some embodiments of touch sereen 212 is as described in the following applica ‘ions: (1) U.S, patent application Ser. No, 11/381,313,-"Mul- ‘ipoint Toveh Surface Controller US. patent application Sex. No. 10840,862, ‘Touehseroen," led May 6, 2004: (3) U.S, patent appicstion Ser. No. 10/903,964, “Gestures For Touch Sensitive lapt Devices.” file Jul. 30,2004: (4) U'S. patent application Ser. No,_ 11/048,264, Deviees," fled Jan. 31, 2008; (5) US. patent application Ser. No. 11038,90, "Mode-Based Graphical User Intorfaces For Touch Sensitive Input Deviees." led Jan, 18, 2005; (6) US. pateat application Ser, No, 1/228.788, "Viral Input Device Placement On A Touch Sereen User interface,” filed Sep. 16, 2005: (7) US. patent application Sor. No. 11/228, 700, “Operation OF Computer With A Touch Sereon Interface,” fled Sep. 16, 2005; (8) U'S. patent application Ser. No, 11/228,737, “Activating Viral Keys OF A Touch Sereen Virtual Keyboard,” filed Sep. 16, 2005; and (9) US. patent application Ser. No. 11/367,749, “Multi-Function Hand-Held Device,” fled Mar. 3, 2006 AIL of these appli cations are incorporated by reference herein i thei entirety [0055] Touch sercen 212 has, for example, a video rs Iution in excess of 100 dpi. some embodiments, the tovel screen has a video resolution of approximately 160 dp The User makes contact with touch sereen 212 using any sbitable ‘objector appendage, such as a stylus, a finger, and so forth Tinsome embodiments, the user interface is designed to work primarily with finger-hsed contacts and gestures, which ean be less precise than styls-hased input dv to the larger aa ‘of contact ofa finger on the touch screen. In some embodi- ‘ments, the device translates the rough finger-basod np ito ‘aprevse pontercursor postion or command fo performing the actions desired by the user. 10056] In some embodiments, in addition to the tovch screen, device 200 includes « touchpad (not shown) for fctivating or deactivating. particular functions. In some tembodiments, the touchpad is touch-sensitive area of the ‘device that, unlike the touch sere, does not display visual output. The touchpad is @ touch-sensitive surface tht is Separate from touch sereen 212 or an extension of the touch-sensitive surface farmed by the touch sereea 10057] Device 200 also includes power system 262 for powering the various components. Power system 262 Includes a power management systom, one or more power souroes (eg, battery, altemating eurent (AC)), recharging system, a power failure detection ctcit,@ power converter br imerter, a power status indicator (ea Hab-emitting ttode (LED)) and any other components associated with the teneration, management and distribution of power in por able devices, [0058] Device 200 also includes one or more optical sensors 264, FIG, 2A shows an optical sensor coupled to ‘optical seasor contoller 258 in LO subsystem 206. Optical sensor 264 includes charge-coupled device (CCD) or ‘onplementary’ metal-xide semicondnctor (CMOS) pho- ‘otransstors. Opiical sensor 264 receives ligt from the fenviroament, projected through one oF more lenses, and ‘converts the light to data representing an image. In conjune- fon with imaging module 243 (also called a camera mod tle), optical sensor 264 captures stil images or video. In Aug, 3, 2023 some embodiments, an optical sensor is located on the back of device 200, opposite touch sereen display 212 on the front ofthe device so thatthe touch sercen display ie used as ‘iewiinder for sill andor video image soquistion. In some embodiments, an optical sensors keaton the frat of the device so that the user's image is obtained for video con- ferencing while the user views the other video conference participants on the toveh sereen display. In sonie embodi- ments, the postion of optieal sensor 264 can be changed by the user (eg. by rotating the lens and the sensor in the device housing) so that a single optical sensor 264 is use slong with the touch screen display for both video confer- fencing and sill andor video image acquisition, 0059] Device 200 optionally also includes one or more contact intensity sensors 268, FIG, 2A shows a contact Intensity sensor coupled to intensity sensor controller 289 in VO subsystem 206, Contact intensity sensor 268 optionally Jnchides one or move piezoresistive stain gauges, capacitive force sensors, electne Toree sensors, piezneletrie force sensors, optical force sensors, capacitive touch-sensitive surfaces, oF otier intensity sensors (eg Sensors Used 10 measure the force (or pressure) of a contact on a touch sensitive surfce). Contact intensity sensor 268 receiver contac intensity information (eg. pressure information ora proxy for pressure infomation) from the environment. In Some embodiments, atleast one contact intensity sensor is collocated with, or proximate to, 2 touch-sensitive surfice (touch-sensitive display system 212). In some embodi- ments atleast one contact intensity sensor is located on the back of device 200, opposite touch screen display 212, ‘whic i located on the Iroat of device 20, 10060] Device 200 also includes one oF more proximity sensors 266. IG, 2A shows proximity sensor 266 coupled to peripherals interface 218, Altematly, proximity sensor 266 is couple to input controller 260 in 1 subsystem 206. Proximity sensor 266 is performed as described in US. patent application Ser. No. 11/261,839, “Proximity Detector Jn Handheld Device", Ser. No. 11/240,788, “Proximity tector In Handheld Device”; Ser. No. 11/620,702, “Us Ambient Light Sensor To Aygmeat Proximity Sensor Out- put"; Ser, No, 11/886,862, “Automated Response To Aad Sensing Of User Activity In Potable Devices"; and Ser. No. 11/638,281, "Methods And Systems For Aulomatie Con figuration Of Peripherals” which are herehyincorporatd by reference in their entirety. In som embodiments, the prox- Jmity sensor tums off and disables touch sereen 212 when the multifunetion device is placed near the users ea (€2 when the user is making 4 phiooe call) 0061} Device 200 optionally also includes one or more tacile output generators 267. FIG. 2A shows tactile ouput enerator coupled to haptic feedback controller 261 in iO subsystem 206, Tactile output generator 267 optionally Jnchides one or more electronic devices such #s speak cs of other audio components andor eleciromechsnical devices that convert energy into linear motion such as a rotor, solenoid, electroactive polymer, piezoelectric actua- tor, elctostatie actuator, of other tactile output generat component (e., a component that converts electrical sig nals into wetife outputs on the device). Contact intensity sensor 265 receives tactile feedback generation insivetions fom haptic feedback module 233 and. generates tactile cotpats on device 200 that are capable of being sensed by a twscrof device 200, In some embodiments at least one tactile foutpot generator is collocated with, oF proximate to, aUS 2023/0245657 Al touch-sensitive surface (e touch-sensitive display system 212) and, optionally, generates a tactile ouput by moving the touch-sensitive surface vertically (eg. inut of sur face of device 200) or laterally eg, back and forth in the same plane a & surface of device 200), ln some embod ‘ments at least one tactile omtput generator sensor is located fon the back of device 200, opposite touch sereen display 212, whichis located on the front of device 20, [0062] Device 200 also inchides one or more acelerom: ser 268, FIG. 2A shows accelerometer 268 coupled 10 Peripherals interface 218. Allemately, aooclerometer 268 is ‘ouped fo an input contmller 260 in LO subsystem 206, Accelerometer 268 performs, for example, as described in US. Patent Publication No. 20050190059, "Acceleration bused Theft Detection System for Portable Electronic Deviees,” and US. Patent Publication No. 20060017692, “Methais And Apparatuses For Operating A Portable Device Based On An Accelerometer” both of which are incorporated by reference herein in their entirety. In some embodiments, information is displayed om the touch sercen Aisplay in portrait view ora landscape view based on at lysis of deta received from the one or more aeelerom ters. Device 200 optionilly includes, in addition to accel ‘erometers) 268, a magnetometer (not shawn) and 8 GPS (or GLONASS or other global navigation system) receiver (oat shown) for obiining information concerning the location tnd orientation (eg, portrait or landscape) of device 200, 10063] Ia some embodiments, the software components ‘sored in memory 202 include operating sytem 226, com ‘munication module (or set of instructions) 228, contac! ‘motion module (or set of instrctions) 28, graphics module (or set of instructions) 232, text input module (or set of instructions) 234, Global Positioning System (GPS) module (6 set of instructions) 238, Digital Assistant Client Module 229, and applications (or ses of insrtions) 236, Further, ‘memory 202 stores data and models, sich as user data ad ‘models 231. Furthennore, in rome embodiments, memory 202 (HG. 2A) of 470 (FIG. 4) stores device’ global internal State 257, as shown in FIGS. 2A and 4, Devicelglobal Jnteral state 257 includes one or more of: ative application state, indicating which applications, if any, are curently active; display ste, indicating what applications, views or ther information occupy various regions of touch sereen Aisplay 212; seasor state, including information obtained trom the device's various sensors and input control devices 216; and location information eonceming the device"s hoe tion andlor atiude [0064] Opersting system 226 (e.g. Dania, RIXC, LINUX, UNIX, OS X, 108, WINDOWS, of an emboddod ‘operating system such as VxWorks) inclades various sot- ‘ware components andlor divers for controling and manag- ing general system tasks (eg, memory management, stomge components, 10068] Communication module 228 facilitates commnni- cation with other devices over one or more extemal ports 224 and also includes various software components fOr handling data received by RF eteutry 208 and/or exec pon 224, External port 224 (eg., Universal Serial Bus (USB), FIREWIRE, et.) is adapted for coupling directly 10 other devices or indirectly over a network (€ the Inter, Wireless LAN, etc). In some embodiments, the extemal port js a multipin (eg, 30-pin) connector that is the same 28, or Aug, 3, 2023 sinilarto andioe compatible with, the 30-pin connector used ‘on iPod® (trademark of Apple Inc.) Jeviees. 10066] Contaevotion module 230 optionally detects con- tact with touch screen 212 (in conjunction with display controller 286) and other touch-sensitive devices (e@., 8 touchpad or physical click whl). Contact motion made 230 includes various software components for peroeming various operations related to detection of contact, such 2 determining if contact has occured (e. detectng finger ‘dvin event), determining an intensity of the contact (eg, the force or presture of the contactor a substitute foe The Torce or pressre of the contet), determining if there is ovement ofthe contact and tracking the movement scrost the touch-sensitive surface (eg. detecting one or more finger-dragging events), and determining ifthe contact has cussed (eg, detecting & fingerup event or a Break in ontaet) Contactimotion module 230 roveives contaet data liom the touch-sensitive surface, Determining movement of the point of contact, which is represented by a series of contact data, optionlly includes determining speed (m0 nitude), velocity (magnitude and direction), andor an accel eration (a change in magnitude andor direction) ofthe point ffcontaet These operations are optionally, applid to single contacts (eg, one finger contsets) or to moltipl simulta- neous contacts (eg. “multitouch” multiple finger contacts). Tn some embodiments, contaeVmation module 230 aad display controller 256 detect contact on a touchpad 10067] In some embodiments, contactimotion module 200 Uses «set of one oF more inensly thresholds to detennine ‘whether an operation has boon performed By @wser (©. 19 determine whether a user bas “clicked” on an icon) In some embodiments at last a subset of the intensity thresholds are determined in accordance with software parameters (€, the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hanlware of device 200), For example, a mouse “click” threshold ofa trackpad or touch sereen display can be sct to any ofa lange range of | predefined threshold values without canging the trackpad fr touch sereen display hardware. Additionally. in some implementations, a user of the doview is provided with software settings for adjusting one oF more of the set of intensity thresholds (e., by adjusting individual intensity thresholds andr by adjusting a plurality of intensity thresh- olds at once with system-level click “intensity” parameter). 10068] Contaetimotion module 230 optionally detects gesture input by a user. Different gestires on the touch ensitive surlice have difleent contact pattems (eg. dif erent motions, timings, andor intensities of detected! con- tacts). Thus, a gesture is, optionally, detected by detecting a particular contact pattem. For example, detecting a finger tap gesture includes detecting a finger-down event followed Hingee-up (fil) event atthe same pos the same postion) asthe finger-down event (@g., at the positon of an icon). AS another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-drapging events, and subse- quently followed by detecting a finger-up (lift?) event 10069] Graphics module 282 includes various known sot- ware components for rendering and displaying graphics on touch seroen 212 or other display, incloding components for changing the visual impact (e., brightness, transparency, Saturation, contas, or ther visual property) of aaphics iUS 2023/0245657 Al are displayed, As used herein, the term “graphics” includes any object that ean be displayed toa user, including, without limitation, text, web pages, icons (such as user-interface objees including soft keys), digital images, videos, anima tions, and the like 10070] In some embodiments, graphics module 232 stores data representing graphics to be used. Bach graphic is, ‘optionally, assigned a coresponding code. Graphics module 252 rocowves, irom applications ete, one oF more codes ‘positing graphs 1 be displayed along with, if necessary, soordinate data and other graphie property data, and then tzeneatessereen image data to output to display’ controller 256. [0071] _Hepsic feodback module 233 includes various soft- ‘ware components foe generating instructions used by tactile output generator(s) 267 t produce tate outputs al one or ‘more loations on device 200 in response to user interactions swith device 200, [0072] Text input module 284, which is, in some ‘examples, a component of graphies module 232, provides sof keyboards for entering text in various applications (eg. ‘contacts 297, email 240, M241, browser 247, and any oer sppliction that needs text ips), 10073] GPS module 288 determines the location of the device and provides this information for use in various applications (eto telephone 238 for use in location-based ating: to camera 243 as picturevideo metadata; and 10 ‘applications that provide location-based services sich a Weather widgets, local yellow page Widgets, and mep! navigation widgets). [0074] Digital assistant client module 229 includes va ‘ous client-side digital assistant instructions to provide the client-side funetionaliGes of the digital assistant. For example, digital assistant client module 229 is capable of sccepling voice input (eg. speech input, text input, touch Input, ander gestural input through various user interfaces (©, microphone 213, accelerometers) 268, 1och-sensi ive daplay system 212, optical sensors) 264, other input control deviees 216, te) of porable multifunction device 200, Digital assistant cleat module 229 is also capable of| providing output in_ audio (eg. speech output), visual, andlor tactile forms through various output interfaces (eg. Speaker 211, touch-sensitive display system 212, tactile futput generators) 267, ete.) of porable multifunction device 200, For example, utpot is provided as voice, sound, ers, text messages, menus, graphies, videos, animations, ‘bration, andlor combinations of two oF more ofthe above. During operation, digital assistant client module 229 com municates with DA server 106 using RE circiry 208 10075] User data and models 281 include various data associated with the user (eg, user-specific vocabulary data, user proference data, userspecified name pronunciations, data from the user's electronic address book, to-do list, shopping lists, et.) to provide the client-side functionalities of the digital assistant. Further, user data and models 231 include various models (eg. speech recognition models, Matistical Langwge models, natural language processing ‘models, ontology tak low models, service models, et) for processing use input and determining. usr ten [0076] In some examples, digital assistant cient module 229 ullizes the various sensors, subsystems, ad peripheral ‘devices of portable mutifametion device 200 to gather ‘ditional information from the surrounding environment of the portable multifunction device 200 to establish a context Aug, 3, 2023 associated wih a ser, the current use iteration, andlor the current use put In some exaapls, digital asistant clout module 229 provides the context information or a Subset hereof will the user inp to DA sever 106 to bp forthe o's inal, Ia somcexanpes, the digital assistant 0 snes the conten information to determine how fo prepare an deliver outputs othe use. Contextual informa tion i refered to as context dita {0077 "Insome examples the contextual infomation tat accompanies the user input includes sensor information, eng, ambient nose, ambien eroperstrs, images or ‘idos of the surrounding ‘covionment, cc. Ia some xampes, the context infomation ean also include the piel state ofthe device, e., device orientation, device Fheation, device temperature, power lve, sped, acceler ‘ion, moion patents, cellular signals seength, et. In some examples, infomation rated wo the software site of DA Server 106, nnning process, installed programs, past fnd preseat nctwork atic, background services. for toga resources usage, ete, and of porable mutation device 200 is provided to DA server 106 at contextial information asocited with 2 user np [0078] In some example, the digital assistant clint mod le 229 seletvely provides information ser dts 231) stred oa te portable mulfuntion device 200 in response to raqests fmm DA server 106. In some eximpes, digit sistant clint mole 229 alsa elicits tonal np om the mer via a natural langinge dislogne or oer ser interfaces upon request by DA server 106. Digit assistant client module 229 passes the additonal input to DA server 196 to belp DA server 106 in intent dedvetion andor Fallment othe users intent expres inthe usr reqs {0079} A more deuiled description fa Ug asstan is desribed blow with reference FIGS, ZAC. ltshould Be recognized that digital assistant clit module 229 can inclade any number ofthe subsmoduls of digital assistant module 726 described below [0080] Applications 236 inchnde the following modules (or ses of ststretions), or 2 subset of superset hero 10081) Coniets module 237 (sometimes called sn dress book or conc st} 0082] Telephone module 238: 0083] Vidco conference module 239: 0084] E-mail elieat module 240, 008s] Instant messaping (IM) module 241: {0086} Workout support module 242: 0087] Camera module 243 for sill andlor video images: 0088] Image manggement module 244; [0089] Video player module; [0090] Music player module: [0091] Browser module 247; 10092] Calendar module 248; 10093] Widget modules 249, which includes, in some ‘examples, one or more of: weather widget 249-1, Stocks widget 249-2, calelator widget 249-3, larm ‘lock widget 249-4, dictionary widget 249-5, and other Widgets oblained by the user, ak well as user-reated widgets 249-6; [0094] Widget ereator module 250 for making. wer created widgets 249-6; 0095] Search module 254; 10096] Video and music player module 282, which merges video player module and musie player module;US 2023/0245657 Al (0097) {oo} Notes move 253; ‘Map module 254; andor [089] Online vikeo module 258 {0100} Examples of otber applications 286 that are stored sn memory 22 inclade other word processing apliations, oor ings editing applications, drawing applications, po Seatation aplicatons, JAVA-emabledapplicadons, encnyp- tion, digital rights management, oie revogiton, and voice replication [0101] To conjunton wih touch sereon 212, display contol 286, ontsev motion mexile 230, graphics module 252, and text input module 234, conscs medile 297 are ‘si fo manage an adress book or contact list (ea stored in application internal state 292 of eontaets module 297 ia ‘memory 202 or memory 470), including adding name(s) 10 the dates book deleting name(s) from the aes book assocaing telephone numbers), eal adres(s), ps Cal address(es) rater information with name associating, fn image wih a name, eatgorizing snd sorting names, providing telephone nombers oF email aes 4 ite noe fiat communications by telephone 238, video enferenve module 239, emai 240, or IM 241 and so Forth [0102] In conjunction with RF circuitry 208, ai ee Cairy 240, speaker 21, microphone 213, touch sree 212, Asp conte 256, contact motion module 29, graphics module 232, and text input module 24, telephone module 238 wed to euer a sequence of darcterscomespoaling to telephone number sees one oF more telephone ai bers in contac module 257, modi telephone number that has Boon entre, il rspotve telephone numb, conve a convertion, and disconnect or hangup when he conversation is completed. As noted above the wireless emmonicaton ase any of «plurality of commaniatons Standards, protocols, and techaologies. {0103}. Ia conjuacion with RE ciety 208, audio ci cuitry 240, speaker 21, microphone 213, touch sereen 212, splay controler 286, opal sensor 268, optical snsor conoler 28, contseUnotion male 230, graphics module 232, text input module 234, contact: mode 297, ad ‘elephone module 238, video conference module 239 Jncindes executable iasirictions (0 iaiate, conde, ad terminate video conference between ter end on OF thor ober partcpantsin accordance wilh User insructions [0105] To conjunstion with RF ciety 208, ouch sereen 212, ispay conoller 256, contaciiovion module 230, tropics module 282, and text input module 244, e-mail Slieut module 240 icles execwube instetions were, Sead, ecive and mung eal in response tower insu dons. In conjunction wih image management module 248, int module 240 makes it very cay to era and Send emails with sill or video images taken with camera module 248, {0105] Ta conjunction with RE icity 208, ouch screen 212, display coniller 256, conaciimovion module 230, fripice modle 232, and text input mode 234, the instant tnewsaging mode 241 includes executable insrctions 10 enter & sequence of characters conesponding 10 an instant tmesoge, © modify previously ener characters, t ats tia respective instant message (or example using 2 Shore Message Service (SMS) or Multimedia Mesage Senice (QMS) proco! for tlepbony-base instant messopes oF using XMPP, SIMPLE, or IMPS for Intel-based stant tmesoges) to eceive instant messages, and to view sive instant messages. In some embodiment, ansnited andor Aug, 3, 2023 received instant messages include graphies, photos, audio fle, video fle andr oer stachmens as are supported in ty MMS and/or an Enhanced Messaging Service (EMS)-As used herein, “instant messaging” refers to both telephony bused messiges (eg., messages sont using SMS or MMS) and Intemet-based messages (€, mesages sent using XMPP, SIMPLE, or IMPS). 0106] In conjunction with RF eireuitey 208, tonch sroen 212, display controller 286, contacvimation module 230, tgrsphies module 232, text input module 234, GPS module 235, map module 284, and music player module, workout ‘support module 242 includes executable instructions 10 create workouts (eg., with time, distance, ander calorie burning goals), communicate with workout sensors (sports devices): receive workout seasor dat; calibrate sensors used to monitor a workout: sclect and play music for a workout; and display, store, and transmit workout data 0107] In’ conjunction with touch sereen 212, display controller 286, optical sensors) 264, opical sensor contol Jer 258, contacvmotion module 280, graphics module 292, and image management module 244, camera module 243 Jnchides executable instructions to capture still images or Video (ineluding video stream) and sore them into ‘memory 202, modify characteristics of a slill image or video, a delete a sill imoge of video from memory 202. 10108]. In conjunction with touch sereen 212, display controller 256, contscVnotion mele 290, grapes made 252, text input module 244 and camera module 243, image ‘mangement module 244 includes executable instructions 19 arrange, modily (eg. edit, or otherwise manipulate, label, delete, present (eg ina digital slide show or album), and ‘ore will andor video images. 10109] In conjunction with RF eiteuitey 208, touch sroen 212, display: conteoler 286, contaevnovion module 230, bruphies module 282, and text input module 234, browser ‘modlle 247 includes exccutable instrvtions to browse the Internet in accordance with user instrctions, including scorching, linking to receiving. and displaying web pages or Portions thereof, aswell a atachments and other files inked fo web pages [0110] In conjunction with RF circuitry 208, touch sreen 212, display controler 256, contactimotion modsle 230, pics module 232, text input module 234, e-mail client ‘modile 240, and browser module 247, calendar module 248 inchudes executable insrotions to crete, display, modify, and store calendars and data associated with calendars (eg, calendar entries, to-do lists, ete.) in accordance with user instrctions 10111] In conjunction with RF circuitry 208, touch srcen 212, display conoler 256, contaevnotion module 230, tripies module 2¥2, text input module 234, and browser ‘mostole 247, widget modules 249 are mini-appications that can be downloaded and used by a user (e., weather widget 249-1, stocks widget 249-2, calculator widget 249-3, alam lock widget 249-4, and dictionary widget 249-8) or created by the user (eg, uscrcresiod widget 249-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) fis, CSS (Cascading Style Sheets) file, and a TavaScrpt file. In some embodiments, a. widget Jnchudes an XML (Extensible Markup Language) file and a JavaSerip file (eg. Yahoo! Widgets). 10112] a conjunction with RF circuitry 208, touch screen 212, display controller 286, contacvimotion module 230, ‘rapies module 2¥2, text input module 234, and browserUS 2023/0245657 Al 10 module 247, the widget ereator module 250 are used by 4 user fo create widgets (turning a userspeified portion of a web page into a widget [0113] In conjunction with touch socen 242, display con- twoller 256, contactimation movie 230, graphics module 232, and tex input module 234, seach module 281 includes executable instructions to seach for text, musi, sound, mage, vdeo, andlor other files in enemory 202 tat match ‘one or more search criteria (eg, one of more user-specified Sscarch terms) in accordance with user instretions. [0118] In conjunetion with touch seen 212, display con- twoler 256, contactimosion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, and browser module 247, video and musie player module 252 Jncdes executable insiretions that allow the user o down Jad and play back recorded music and other sound files stored inane or mare file formats, such as MPS or AAC fies, and executable instructions to display, present, or otherwise play back videos (eg, on touch soon 212 oron an extemal, comected display vi external port 224). In some embodi- ments, device 200 optionally ineludes the functionality of an MP3 player, such as an iPod (trademark of Apple Ine.) [0113] In conjunetion with touch soreen 212, display con wwoller 286, contactmotion movil 230, graphics module 232, and text input module 234, notes module 253 includes executable instetions to create and manage nots, to-do lists, andthe like in accondance with wser instructions [0116] In conjunction with RF circuitry 208, touch sereen 212, display controler 286, contacvmetion module 230, {raphies module 232, text input module 234, GPS module 235, and browser module 247, map module 254 are used 19 receive, display, modify, and soe maps and data associated with maps (ez. driving directions, data on stores and ater points of interest ator nea a particular eatin, and ater location-based data) in acordance with use instruction. [0117] In conjunetion with touch seen 212, display con twolee 256, contactimasion movie 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, text ‘input module 234, e-mail client movdle 240, and browser ‘movule 247, online video module 285 includes insiections that allow the user 10 acoess, browse, reoive (eg. bY streaming and/or download, play back (eg, on the touch Screen or on an extemal, connected display via external port 224), sond an e-mail with a link to particular online video, fand otherwise manage online videos in one oF more fle Toemats, seh as 11.264, Ta some embodiments, iastant messaging module 241, rather than e-mail client module 240, is used to send a'link to a particular online video. Additonal description of the online video application can be Found in U.S. Provisional Patent Application No. 60936, 562, "Portable Muhifunetion Device, Method, and Graph al User Interface for Playing Online Videos,” fled Jun. 20, 2007, and US. patent application Ser. No, 11/968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed Dee. 31, 2007, the contents of whick are hereby incorporsted by ‘reference in thet entirety. [0118] Fach of the above-identfied modules and applica ‘ions comesponds to a set of executable instructions for performing one oF more funetionsdeseribed above and the methods described in this application (eg, the computer Jimplemented methods and ther information processing methods described herein). These modules (eg. set of Instructions) aed not be implemented as separate software Aug, 3, 2023 programs, procedures, or modules, and dus various subsets Of these modules can be combined or otherwise rearranged in various embodiments. For example, video player module can be combined with music player modole into a single ‘module (eg. video and music player module 52, FIG. 24). In some embodiments, memory 202 stores a subset of the rmoddules and data strctures identified above. Furthermore, memory 202 stores additional modules and data stevetures rot described above [0119] Ia some embodiments, device 200 is 2 device ‘where operstion of a predefined set of functions on the device iS perlammed exclusively through 2 touch screen fndor a touchpad. By using a touch sereen andor a touch ppd as the primary input control device for operation of device 200, the number of physical input contol devices (uch as push buttons, dials, and the Tike) on device 200 is reduced, 10120] The predetined set of fonctions that are performed exclusively through a touch screen andlor a’ touchpad optionally include navigation between user interfaces. In some embodiments, the touchpad, when touched by the use, hvigates device 200 to a min, home, or root ment from ‘any user interface that is displayed on deviee 200, tn such temibadimeals, "men buon” is implemented using. a touchpad. In Some other embodiments, the menu button i 4 physical push button or athcr physical input control device fnstead ofa touchpad, [0121] FIG. 2B is a block diagram iltusteating exemplary components for event handing in accontance wih some embodiments. In some embodiments, memory 202 (FIG. 2A) or 470 (FIG. 4) includes event sorter 270 (eg. in operating system 226) and a respective application 2361 (x. any ofthe aforementioned applications 237-281, 288, 450-490), 10122] vent sorter 270 receives event information and determines the application 236-1 and application view 291 ‘of application 236-1 w which to deliver the event informs tion: Event sorter 270 jnludes event monitor 271 and event dispatcher module 274, In some embodimenis, application 236-1 includes application internal state 292, which indi- cates the current application view(s) displayed on touch: Sensitive display 212 when the application is active or executing. In some embodiments, devicelghibal internal Sate 287 is used by event sorter 270 to determine which application(s) is (are) currently active, and application iner- nal state 292 is used by event sorter 270 t0 detennine application views 291 to which to deliver event information. 10123] In some embodiments, aplication internal sate 292 includes aitonal ifomnatin, such as one of more of: resume information to be used when application 236-1 restimes exectton, user interface slate information that indicates information being displayed or that is ready for display by application 236-1, sate queve for enabling the user t go back oa prior sate of view of application 236-1, and a redavundo queue of previous ations taken by the use, 10124] Event monitor 271 receives event information from peripherals interface 218. Event information includes infor mation about a sub-event (eg, a user foueh on touel- sensitive display 212, as part of @ multitouch geste). Peripherals iterfce 218 transmits information it receives fom UO subsystem 206 or a sensor, such as proximity sensor 266, socelerometer(s) 268, andior microphone 213 (Ghrough audio eieuitry 210), Information that peripherals