Shwetha Rajaram, Hemant Bhaskar Surale, Codie McConkey, Carine Rognon, Hrim Mehta, Michael Glueck, Christopher Collins
Reality Labs Research, Meta
Conference paper · CHI '25: ACM Conference on Human Factors in Computing Systems, Yokohama, Japan
PDF ACM DL Publication BibTeX
A woman wearing glasses, a ring, and a wristband asks: Remind me, what happened in last week's meeting? Along a timeline, the assistant's spoken answer highlights the word activities, a haptic pulse on the wrist lines up with that word, and a thumb tap on the ring selects it; the assistant then describes two activities.
Gesture and audio-haptic techniques let users rapidly navigate and manage the timing of a conversation with a voice assistant. Here a haptic pulse on the wrist lines up with the keyword "activities" in the spoken answer, a tap on the ring selects it, and the assistant goes on to specific activities. Figure from the paper.

Quick facts

Voice assistants powered by large language models can now hold long, open-ended conversations, but speech is linear: you have to sit through long answers, interrupt out loud, and time your pauses so the assistant does not drop the conversation. We kept voice and audio for the content of the conversation and added a ring for gestures and a wristband for haptics to control its flow. The work was led by Shwetha Rajaram (University of Michigan School of Information) during a research internship at Reality Labs Research, Meta. Overall, our work contributes:

  1. A suite of multimodal interaction techniques that combine voice, gestures, audio, and haptics to facilitate conversations with voice interfaces, built into a working system.
  2. An exploratory study with 14 participants showing the benefits of these techniques for rapid access to information and flexible navigation within conversations.
  3. Design recommendations for better role-based collaboration between the input and output modalities.

Abstract

Advances in large language models (LLMs) empower new interactive capabilities for wearable voice interfaces, yet traditional voice-and-audio I/O techniques limit users’ ability to flexibly navigate information and manage timing for complex conversational tasks. We developed a suite of gesture and audio-haptic guidance techniques that enable users to control conversation flows and maintain awareness of possible future actions, while simultaneously contributing and receiving conversation content through voice and audio. A 14-participant exploratory study compared our parallelized I/O techniques to a baseline of voice-only interaction. The results demonstrate the efficiency of gestures and haptics for information access, while allowing system speech to be redirected and interrupted in a socially acceptable manner. The techniques also raised user awareness of how to leverage intelligent capabilities. Our findings inform design recommendations to facilitate role-based collaboration between multimodal I/O techniques and reduce users’ perception of time pressure when interleaving interactions with system speech.

The idea: talk for content, tap for control

Gestures and haptics carry less information than speech, but they are more efficient to perform and to process. So we gave each modality a role. Voice and audio carry the content of the conversation: your prompts and the assistant’s answers. Ring gestures steer the flow (go deeper, skip, replay, wrap up, tell me more), and wristband haptics confirm your input and point out what you can do next. Because the two pairs of channels run in parallel, you can act while the assistant is still talking.

conversation content: voice and audio Assistant audio keyword answer deeper answer more You voice prompt conversation control: gestures and haptics Wristband haptics nudge confirm Ring gestures wake go deeper tell me more wrist roll Tap A Tap B ×2 I/O modality properties expressiveness high med low high time cost content voice input audio output control gesture input haptic output
The timeline shows one stretch of a conversation: a wrist roll wakes the mic, a haptic nudge marks a keyword while the assistant talks, Tap A cuts in to go deeper on that keyword, and a double Tap B asks for more detail without interrupting. The chart shows why each modality got its role: voice and audio are expressive but slow, while gestures and haptics are quick but less expressive.

One gesture set covers many functions because the system knows who is talking. A gesture can mean one thing while the assistant is speaking, another while you are speaking, and a third during silence.

System

Techniques. We built techniques for two kinds of flow. For navigational flow (the order of the conversation): Select a follow-up topic that the assistant reads out, Go Deeper into a keyword while the assistant is speaking, and Replay the current response or Skip to the next sentence. For temporal flow (its timing): Wake the assistant with a quick wrist rotation instead of a wake word, which also interrupts it; Restyle Response to regenerate it with new instructions; and Wrap it Up or Tell me More to end a response at the next sentence or extend it. On the output side, audio-haptic cues show when the system is listening or processing, warn with a subtle haptic after 5 seconds of silence that the conversation is about to time out (speaking within another 5 seconds keeps it going), confirm gestures, and nudge users toward keywords they can explore. Nudges are haptic only, so they do not compete with the assistant’s speech.

System diagram: an illustrated person wearing Ray-Ban Meta glasses, a ring, and a wristband, with photos of the ring and wristband hardware, connected to a screenshot of the Unity voice assistant client, which connects to GPT-4o and the Meta Voice SDK.
System architecture. The LLM-enabled voice assistant is a Unity application (A) that uses GPT-4o (E) and the Meta Voice SDK (F). Audio goes to Ray-Ban Meta glasses over Bluetooth (D), and two in-house wireless wearables handle gesture input (B) and haptic feedback (C). Figure from the paper.

Implementation. Four GPT-4o agents power the assistant: an Informational Audio Assistant that answers prompts, a Keyword Identifier that highlights important words in each answer, a Follow-up Generator that suggests subtopics, and a Response Extender that adds detail at the end of the current answer. The Meta Voice SDK (version 66.0) handles dictation, text-to-speech, and voice commands, and its word events give the timing of every spoken word, which lets nudges and gestures line up with specific words. The ring uses electrodes to sense when the index finger touches skin, plus an IMU and an optical proximity sensor, to tell apart two thumb-tap positions: A (the distal phalanx) and B (the proximal phalanx). The wristband has 4 vibrotactile actuators, and its haptic patterns match the frequency and amplitude of audio cues made with a professional sound designer. All cues were designed to be just noticeable.

Study

Conditions. Voice-Driven interaction, the baseline: speak prompts, hear answers, and use the wake word “Hey Roger” to start or interrupt. Gesture+Haptic-Guided interaction: the same voice and audio, plus gestures, a Wrist Roll to wake or interrupt, audio-haptic feedback, and haptic-only nudges on keywords.

Eight hand illustrations, each labeled with a function and a gesture: Wake Mic, Wrist Roll; Select Keyword or Follow-up Topic, Tap A; Play Follow-up Topics, Tap B; Replay, Tap A Hold; Skip, Tap B Hold; Wrap it Up, Tap A twice; Tell me More, Tap B twice.
The gesture set used in the study: three poses (Wrist Roll, Tap A, Tap B), with holds and double taps for more functions. Tap B sequences move the conversation forward, and Tap A sequences backtrack or stall it. Gestures marked (*) were introduced in Task 2. Figure from the paper.

Tasks. After a training session, participants did two tasks, each with one 2-minute conversation per condition. In Task 1 (Learning), they learned about an unfamiliar topic they chose from a list (such as “how manual clocks work” or “uses of coriander”) and then taught it back to the researchers. In Task 2 (Meeting Prep), they prepared for an upcoming meeting in a scenario of commuting to work, by asking about two meeting transcripts: one on social media strategy for a new breakfast cereal and one on a library summer reading program for kids. Task 2 added Replay, Skip, and Wrap it Up for faster-paced conversations. Condition order was counterbalanced and reversed between tasks.

Participants. 14 people (5 women, 7 men, 2 not reported), with an average age of 31 (range 21-61). 11 used voice interfaces daily. Sessions lasted 90 minutes.

Measures. Timestamped logs of user and system speech, gestures, and haptic nudges; the Subjective Assessment of Speech User Interfaces (SASSI) questionnaire after each condition; and interviews. We compared conditions with Wilcoxon signed-rank tests.

What do we learn?

1. With gestures, people did more and interrupted more

Participants performed 248 interactions in the Gesture+Haptic-Guided condition versus 137 with voice alone (Z = -3.296, p = 0.001), and interrupted the assistant 58 times versus 8 (Z = -3.18, p = 0.002). Error rates did not differ significantly: 0.137 for voice input and 0.203 for gestures (Z = -1.538, p = 0.132). With voice, people let the assistant finish even when it was repeating itself, because cutting it off felt rude: “it felt like I’m talking to somebody… so I let him finish the sentence” (P4). A gesture felt different: “I don’t have to commit the act of interrupting by voice. I just kind of send a signal” (P13).

Two conversation timelines from one participant. The voice-driven conversation about the rules of curling alternates evenly between spoken questions starting with Hey Roger and the assistant's answers. The gesture and haptic-guided conversation about how mirrors are made has a single spoken prompt followed by many gestures (Select Keyword, Tell me More, Play Followups, Select Followup) and denser assistant speech.
P6's two Task 1 conversations. Voice-driven turns were balanced and slower-paced, while gestures gave quicker access to information and more interruptions. Light blue bubbles are user speech, white bubbles are system speech, and blue dotted lines are interruptive gestures. Figure from the paper.

2. Gestures made information easier to reach, but the assistant felt in charge

Participants described the gesture condition as “having information at your fingertips” (P7), and the conversations had more system speech surfacing new details. The scoped gestures and nudges also showed people what the assistant could do: “knowing that the assistant can add a follow up helped me realize what all I can ask” (P2). Several participants then used the same moves by voice, saying “Tell me More” and “Replay”. On the other hand, conversations felt system-driven (“the leader in the conversation is the voice assistant,” P5), and people stuck to gestures instead of speaking: on average only 28.36% of gestures were used to wake the mic. With gestures, some participants were also unsure how the system had interpreted them, especially when selecting keywords.

3. Haptic nudges added time pressure

Combined with fast gestures, the nudges made participants feel they had to act right away or miss an opportunity. P6 described being “in a chase” with the system, and P13 felt like “multi-tasking between hearing and feeling”. On SASSI, participants rated Voice-Driven interaction significantly higher for Likeability (mean 6.016 vs 5.071; Z = 2.731, p = 0.007) and Cognitive Demand (5.329 vs 4.114, where higher is more favorable; Z = 2.605, p = 0.01), with no significant differences for System Response Accuracy, Annoyance, Habitability, or Speed. Participants attributed the higher voice ratings to their familiarity with voice interfaces and expected gestures to get easier with practice.

Box plots of SASSI scores for voice-driven (blue) and gesture plus haptic (orange) interaction across six dimensions. Voice is significantly higher for Likeability and Cognitive Demand; the other four dimensions overlap.
SASSI ratings on a 7-point scale, where higher is more favorable. Asterisks mark the two dimensions where Voice-Driven scored significantly higher. Figure from the paper.

Design implications

  1. Move between gestures and voice automatically. For example, open the mic right after a gesture so users can clarify with a few words, and use output to suggest the right input: audio prompting voice, haptics prompting gestures.
  2. Confirm how the system interpreted the input, not just that it arrived. During Go Deeper, the assistant could repeat the keyword it thinks the user picked, so mistakes are easy to correct.
  3. Give advance notice and allow delayed actions. Haptics could ramp up before a keyword arrives, and gesture variants could separate “do it now” from “queue it for later”.

Limitations

We used a ring that supports subtle on-skin gestures, and the finding that gestures are a socially acceptable way to interrupt may not hold for camera-based gesture recognition. The study ran in a controlled lab, while speech and gesture recognition would be less reliable in noisy, busy settings. Most participants were experienced voice interface users, and people with hearing or motor impairments still need to be studied. Results may include novelty effects, although participants rated the voice baseline higher on two SASSI dimensions, which suggests no bias toward the new techniques.

Publication

Shwetha Rajaram, Hemant Bhaskar Surale, Codie McConkey, Carine Rognon, Hrim Mehta, Michael Glueck, and Christopher Collins. 2025. Gesture and Audio-Haptic Guidance Techniques to Direct Conversations with Intelligent Voice Interfaces. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ‘25), Yokohama, Japan. ACM, 1-20. https://doi.org/10.1145/3706598.3714310

BibTeX

@inproceedings{rajaram2025gesturellm,
  title     = {Gesture and Audio-Haptic Guidance Techniques to Direct Conversations with Intelligent Voice Interfaces},
  author    = {Rajaram, Shwetha and Surale, Hemant Bhaskar and McConkey, Codie and Rognon, Carine and Mehta, Hrim and Glueck, Michael and Collins, Christopher},
  booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '25},
  location  = {Yokohama, Japan},
  pages     = {1--20},
  year      = {2025},
  publisher = {ACM},
  doi       = {10.1145/3706598.3714310}
}

Contact

Questions? Feel free to contact:

  • Shwetha Rajaram, Reality Labs Research, Meta: shwethar [at] umich.edu
  • Hemant Bhaskar Surale, Reality Labs Research, Meta: hemant.surale [at] gmail.com

Figures are from the paper (© 2025 copyright held by the owner/author(s)) and are reproduced here by the authors.