Junxiao Shen1,2, Roger Boldu1, Arpit Kalla1, Michael Glueck1, Hemant Bhaskar Surale1, Amy Karlson1
1Reality Labs Research, Meta  ·  2University of Bristol
Journal article · IEEE Transactions on Visualization and Computer Graphics (presented at IEEE ISMAR 2024)
PDF IEEE Xplore Publication BibTeX
Four line drawings: a person in AR glasses with a ring on the index finger facing a virtual keyboard; the hand pinching with the cursor on a key; the hand moving as a red trajectory crosses the keyboard; and the hand released, with hello world typed.
Typing a word with RingGesture: (a) turn the wrist to place the cursor on the first letter, (b) pinch thumb and index finger to start the swipe, (c) trace the word by moving the wrist, and (d) release the pinch. Score Fusion then predicts the top words, with the first one preselected. Figure from the paper.

Quick facts

Lightweight AR glasses, built for all-day wear, can’t fit the many cameras that wide-field hand tracking needs, so text entry needs another input device. Led by Junxiao Shen (Reality Labs Research, Meta and University of Bristol), we built RingGesture: a ring that tracks the hand with an IMU and senses pinches with electrodes, so people can gesture-type in mid-air with one hand. Overall, our work contributes:

  1. RingGesture, a fast, accurate, and easy-to-learn ring-based mid-air gesture typing system, with entry rates (27.3 WPM on average, 26.4 WPM in the first block, 32.5 WPM in the last) comparable to gesture typing on a phone.
  2. Score Fusion, a lightweight deep-learning word prediction framework that combines a word-gesture decoding model (trained with a new layout transformation), a spatial spelling correction model (using a keyboard-layout-aware edit distance), and a pre-trained contextual language model.
  3. Two studies: one comparing word-level and phrase-level gesture typing (32 participants), and a longitudinal study (16 participants) showing that Score Fusion improves uncorrected character error rate by 28.2% and entry speed by 55.2% over a conventional framework, Naive Correction.

Abstract

Text entry is a critical capability for any modern computing experience, with lightweight augmented reality (AR) glasses being no exception. Designed for all-day wearability, a limitation of lightweight AR glass is the restriction to the inclusion of multiple cameras for extensive field of view in hand tracking. This constraint underscores the need for an additional input device. We propose a system to address this gap: a ring-based mid-air gesture typing technique, RingGesture, utilizing electrodes to mark the start and end of gesture trajectories and inertial measurement units (IMU) sensors for hand tracking. This method offers an intuitive experience similar to raycast-based mid-air gesture typing found in VR headsets, allowing for a seamless translation of hand movements into cursor navigation. To enhance both accuracy and input speed, we propose a novel deep-learning word prediction framework, Score Fusion, comprised of three key components: a) a word-gesture decoding model, b) a spatial spelling correction model, and c) a lightweight contextual language model. In contrast, this framework fuses the scores from the three models to predict the most likely words with higher precision. We conduct comparative and longitudinal studies to demonstrate two key findings: firstly, the overall effectiveness of RingGesture, which achieves an average text entry speed of 27.3 words per minute (WPM) and a peak performance of 47.9 WPM. Secondly, we highlight the superior performance of the Score Fusion framework, which offers a 28.2% improvement in uncorrected Character Error Rate over a conventional word prediction framework, Naive Correction, leading to a 55.2% improvement in text entry speed for RingGesture. Additionally, RingGesture received a System Usability Score of 83 signifying its excellent usability.

The idea: a ring draws the swipe, three models pick the word

Mid-air gesture typing tends to be fastest when the cursor maps directly to body movement. Vulture reached 20.6 WPM with hand control, but it relied on OptiTrack markers along the arm. RingGesture gets the same direct mapping from one ring: its IMU turns wrist rotation into cursor motion, and its electrodes detect when the thumb touches the index finger. IMU tracking adds noise and drift to the swipe, so rather than trusting the swipe alone, Score Fusion weighs it against what the keyboard layout and the conversation make likely.

1. The ring draws the swipe q w e r t y u i o p a s d f g h j k l z x c v b n m release: end pinch: start pinch release Electrodes thumb touches index finger IMU wrist rotation moves the cursor (gain 1.8) one pinch-to-release swipe is one word 2. Score Fusion Gesture decoding model swipe → candidate words Spatial spelling correction keyboard-aware edit distance Contextual language model history, tags, typed text blended score for each word (1 − a − b) · decoder + a · spelling + b · context rerank: Top-1 preselected
The ring view shows how one word is entered: a pinch starts the swipe, wrist rotation sensed by the IMU moves the cursor, and the release ends it. The Score Fusion view shows how the word is chosen: each candidate gets three log-probability scores (from the swipe, from a keyboard-aware spelling model, and from a contextual language model), and words are ranked by their weighted sum. The baseline, Naive Correction, instead runs decoding, spelling correction, and n-gram reranking one after another.

System

Ring. We built the ring from the ElectroRing reference design (Kienzle et al.). IMU readings are converted to quaternions, then to polar coordinates, then to a 2D cursor position, with a control-display gain of 1.8. Electrodes detect the touch and release of a pinch from changes in an electrical signal.

Steady pinches. A pinch can nudge the cursor right when it matters, so the selection point lands in the wrong place (a Heisenberg effect of crosstalk between input modalities). To counter it, an exponential smoothing filter sets its own filtering level by trading off noise rejection against tracking error (λ = 0.75).

Gesture decoding model. An attention-enhanced bidirectional LSTM with CTC loss, trained on the public How We Swipe phone dataset (1,338 users, 11,318 unique words). That dataset used a different keyboard layout, so we mapped its swipes onto ours with a piecewise affine transformation anchored at key centers and six points around the keyboard. This raised the model’s test accuracy on the Study 1 data by 76% compared with simply normalizing by keyboard width and height.

Three plots of the same swipe for the word available over a keyboard: raw input on the input layout, temporally resampled on the model layout, and spatially transformed on the model layout, each with anchor points labeled TL, TR, ML, MR, BL, and BR.
Converting a swipe for "available" from the dataset's keyboard layout to ours: first resampled in time, then transformed in space using key centers and six anchor points around the keyboard. Figure from the paper.

Spatial spelling correction. A probabilistic edit distance that knows the keyboard layout. Substituting an adjacent key (like q for w) has probability 0.17, while a non-adjacent key has 0.01; omitting or adding a character each has 0.06. We estimated these from earlier mid-air gesture typing data.

Contextual language model. A bidirectional LSTM that predicts the next word from the previous words, the conversation history, and context tags such as location and time. Adding context lowered perplexity from 70.91 to 42.82, and pre-training on public comments and posts before fine-tuning lowered it to 37.16. After quantization, the model is 7 MB, small enough to run in real time on a mobile phone processor.

Experiment

Study 1: word or phrase? First, we asked whether removing the pause between words would speed up gesture typing with a directly mapped cursor. 32 participants (18 men, 13 women, 1 undisclosed; mean age 33) swiped 40 phrases per condition on a Sensel touchpad, which stands in for perfect tracking, with an idealized decoder. In the word-level condition they lifted the finger after every word; in the phrase-level condition, only after the whole phrase.

Study 2: RingGesture over two weeks. 16 participants (9 women, 6 men, 1 undisclosed; ages 21 to 49.5) wore the ring on the index finger and typed on a virtual keyboard shown on a monitor, which stands in for the fixed virtual screen of lightweight AR glasses. Over four sessions on separate days, they completed 16 blocks; each block had 10 phrases with Score Fusion and 10 with Naive Correction, in counterbalanced order. Phrases came from the ConvAI2 dataset, and each phrase’s conversation history and persona tags were fed to the language model automatically.

A participant seated at a desk with a ring on the index finger, hand raised in front of a monitor showing a keyboard and a phrase to type.
Study 2 setup: participants could rest their arms on the chair's armrests and move the wrist freely to swipe. Figure from the paper.

Measures. Words per minute, uncorrected and corrected character error rate, Likert ratings of six aspects (such as ease of learning and hand fatigue), and the System Usability Scale.

What do we learn?

1. Swiping word by word was as fast as swiping whole phrases, and most people preferred it

Word-level swiping reached 24.7 WPM and phrase-level 25.5 WPM, with uncorrected error rates of 7.8% and 9.2%; neither speed (F = 2.14, p = 0.13) nor accuracy differed significantly. 19 of 32 participants preferred word-level typing, citing familiarity, thinking in words, less fatigue, and immediate feedback; 8 preferred phrase-level typing and 5 had no preference. RingGesture therefore uses one swipe per word.

2. Score Fusion made ring typing 55% faster

With Score Fusion, participants averaged 27.3 WPM, against 17.6 WPM with Naive Correction (F = 30.276, p = 0.001), a 55.2% improvement. The advantage held across all 16 blocks (no condition by block interaction, p = 0.196). With Score Fusion, entry rate rose from 26.4 WPM in the first block to 32.5 WPM in the last, and peak performance reached 47.9 WPM.

Line chart of words per minute over 16 blocks. Score Fusion stays between about 24 and 33 WPM; Naive Correction stays between about 12 and 20 WPM. Power-law learning curves are fitted to both.
Entry rate over 16 blocks with Score Fusion and Naive Correction, with fitted power-law learning curves. Figure from the paper.

3. Every model in the fusion pulls its weight

Score Fusion also lowered the uncorrected character error rate by 28.2% and the corrected one by 29.0% relative to Naive Correction. Replaying the swipes logged in Study 2, the gesture decoding model alone had a character error rate of 27.86%; adding spatial spelling correction brought it to 12.12%, and adding the contextual language model brought it to 5.56%.

Two box plots comparing Score Fusion and Naive Correction. Score Fusion has a lower median and a tighter spread for both uncorrected and corrected character error rate.
(a) Uncorrected and (b) corrected character error rate under the two conditions. Figure from the paper.

4. People found it easy to use

RingGesture scored 83 on the System Usability Scale, where anything above 80 indicates excellent usability. Participants found it easy to learn, reported little hand fatigue, and were neutral about eye fatigue. Learning curves suggest they started faster with Score Fusion (fitted initial rate 23.9 vs 14.1 WPM) but improved faster with Naive Correction (learning rate 0.115 vs 0.069), possibly because they had to adapt to its weaker predictions. Two participants occasionally triggered a false pinch when the middle finger touched the index finger.

Limitations

We used a monitor as a stand-in for lightweight AR glasses, which show a virtual screen at a fixed distance with only 3DOF. That holds in a controlled lab but may not extend to real-world use, especially on the move; testing in the wild (typing while walking, in a car, or lying in bed) waits for glasses with longer battery life. The conversational context was predefined with each phrase rather than coming from participants’ own conversations. We have not yet tested Score Fusion with other text entry methods, such as touch typing.

Publication

Junxiao Shen, Roger Boldu, Arpit Kalla, Michael Glueck, Hemant Bhaskar Surale, and Amy Karlson. 2024. RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction Framework. IEEE Transactions on Visualization and Computer Graphics 30, 11 (November 2024), 7441-7451. https://doi.org/10.1109/TVCG.2024.3456179

BibTeX

@article{shen2024RingGesture,
  title     = {RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction Framework},
  author    = {Shen, Junxiao and Boldu, Roger and Kalla, Arpit and Glueck, Michael and Surale, Hemant Bhaskar and Karlson, Amy},
  journal   = {IEEE Transactions on Visualization and Computer Graphics},
  volume    = {30},
  number    = {11},
  pages     = {7441--7451},
  year      = {2024},
  publisher = {IEEE},
  doi       = {10.1109/TVCG.2024.3456179}
}

Contact

Questions? Feel free to contact:

  • Junxiao Shen, Reality Labs Research, Meta and University of Bristol
  • Hemant Bhaskar Surale, Reality Labs Research, Meta: hemant.surale [at] gmail.com

Figures are from the paper (© 2024 IEEE) and are reproduced here by the authors.