One swipe decoder for many XR keyboards

Quick facts

Swipe typing (word-gesture typing) now shows up all over XR: poking a virtual keyboard, pinching to drag a ray across it, or swiping on a surface. Each mode produces differently shaped swipes, which makes a neural decoder trained on one system hard to reuse on another. Led by Junxiao Shen (Reality Labs Research, Meta and University of Bristol), we built one decoder that works across them by turning each swipe into a coarse sequence of key regions before decoding. Overall, our work contributes:

1. Trajectory coarse discretization, which makes it possible to pre-train a word-gesture neural decoder that generalizes to on-surface and mid-air keyboards in AR and VR. The resulting Pre-trained Neural Decoder is fast, easy to deploy, and needs no extra data collection for training or fine-tuning.
2. A validation on four AR/VR keyboard datasets (three of them collected for this work), where the Pre-trained Neural Decoder reached 90.4% Top-4 word accuracy and performed significantly better than conventional decoders.
3. A real-time test on Quest 3: after quantization the model is 4 MB and decodes with a latency of about 97 ms.

Abstract

Text entry with word-gesture keyboards (WGK) is emerging as a popular method and becoming a key interaction for Extended Reality (XR). However, the diversity of interaction modes, keyboard sizes, and visual feedback in these environments introduces divergent word-gesture trajectory data patterns, thus leading to complexity in decoding trajectories into text. Template-matching decoding methods, such as SHARK² [32], are commonly used for these WGK systems because they are easy to implement and configure. However, these methods are susceptible to decoding inaccuracies for noisy trajectories. While conventional neural-network-based decoders (neural decoders) trained on word-gesture trajectory data have been proposed to improve accuracy, they have their own limitations: they require extensive data for training and deep-learning expertise for implementation. To address these challenges, we propose a novel solution that combines ease of implementation with high decoding accuracy: a generalizable neural decoder enabled by pre-training on large-scale coarsely discretized word-gesture trajectories. This approach produces a ready-to-use WGK decoder that is generalizable across mid-air and on-surface WGK systems in augmented reality (AR) and virtual reality (VR), which is evident by a robust average Top-4 accuracy of 90.4% on four diverse datasets. It significantly outperforms SHARK² with a 37.2% enhancement and surpasses the conventional neural decoder by 7.4%. Moreover, the Pre-trained Neural Decoder’s size is only 4 MB after quantization, without sacrificing accuracy, and it can operate in real-time, executing in just 97 milliseconds on Quest 3.

The idea: snap the swipe to keys, then pre-train once

Most decoders read a swipe as a sequence of x, y coordinates. Those coordinates depend on the keyboard’s size, the input mode, and how noisy the tracking is, so a swipe on a phone and the same swipe in a headset look quite different. Instead, we assign each point of the swipe to a coarse region around a key and give the decoder a sequence of one-hot key vectors. The input then lives in the same discrete space as the output (letters), and small offsets in where the finger or cursor went stop mattering.

The decoder

Discretization. We divide the keyboard into square regions, one per key, each larger than the key itself. The region size comes from how widely touch points spread around keys in the phone dataset, and the larger “pixels” make the input more tolerant of spatial noise and ambiguity. A mapping function assigns each trajectory point to its region, and the resulting sequence is one-hot encoded.

Pre-training data. 95,649 swipes from the public “How We Swipe” phone dataset (1,338 users), plus 100,000 synthetic swipes from a GAN-based generator trained on that same phone data: 32,347 unique words in total. No XR data is used.

Model. Two stacked bidirectional LSTM layers (hidden size 222, dropout 0.3), a fully connected layer, and a CTC beam-search decoder that turns per-time-step letter probabilities into word candidates. We picked this configuration from a sweep of over 100 configurations while keeping the model under 5 MB.

Evaluation

Keyboards. Four XR keyboard systems that the decoder never saw in training: mid-air poking in AR (public HoloLens 2 data from two earlier studies, 25,513 swipes), mid-air poking and mid-air pinching in VR (Quest 2, about 51,400 swipes each), and on-surface swiping on a Sensel touchpad with the keyboard shown on a monitor (33,594 swipes).

Participants. For the VR datasets, 200 volunteers (100 women, 100 men) typed at home on a retail Quest 2, about 30 minutes per condition. For the on-surface dataset, 100 volunteers typed 160 words each in sessions of about one hour.

Baselines. SHARK², the template-matching decoder most XR gesture keyboards use, and a conventional neural decoder with the same architecture as ours but raw Cartesian coordinates as input. We used leave-one-subject-out testing.

Measures. Top-1 and Top-4 word accuracy: how often the intended word is the first suggestion, or among the first four.

What do we learn?

1. Every XR keyboard leaves a different fingerprint

On a unit-width keyboard, touch points for AR mid-air poking sat farther from key centers (0.042 on average) than for VR poking (0.030) or VR pinching (0.028). We attribute this to less accurate hand tracking on HoloLens 2 and to the lack of hand occlusion in optical see-through AR. Phone (0.15) and on-surface (0.12) swipes were spread even more widely, a “fat finger” effect of small touch surfaces. The shapes differ too: VR swipes were straighter and more varied, and on-surface swipes were the most curved.

2. One pre-trained decoder beats both baselines out of the box

Averaged over the four XR datasets, the Pre-trained Neural Decoder reached 83.3% Top-1 and 90.4% Top-4 accuracy without any data from those keyboards. That is 37.2 percentage points above SHARK² in Top-1 accuracy (37.6 in Top-4) and 7.4 points above the conventional neural decoder (6.9 in Top-4). Top-4 accuracy was 89.8% for AR poking, 91.9% for VR poking, 90.2% for VR pinching, and 89.5% on-surface. Adding a bigram language model and auto-correction raised the average Top-1 accuracy to 94.5%.

3. Fine-tuning helps, but only a little

Fine-tuning on each dataset’s training split raised Top-1 accuracy by 3.4 points on average (for example, from 85.1% to 88.0% for VR poking). That gain is much smaller than the gap between the pre-trained decoder and either baseline, so for research use the pre-trained decoder can be dropped in as is.

4. The encoding matters more than the region shape

One-hot encoding beat integer encoding on every dataset (83.0% vs 64.2% Top-1 on-surface, for example), likely because integers imply an order between keys that does not exist. Square and elliptical regions performed about the same, so we kept squares because they are cheaper to compute. Replacing the LSTM with a transformer kept Top-1 accuracy below 60%. After quantization, the model is 4 MB and runs with a latency of 97 ms on Quest 3, with Top-1 word accuracy unchanged.

Limitations

The decoder is built for QWERTY and may not work out of the box on other layouts such as Metropolis, OPTI, or Dvorak, because discretization assumes the QWERTY key arrangement. We validated it on four keyboard systems but did not test less common designs, such as curved keyboards or eye-gaze-based gesture typing. A transformer-based decoder performed poorly, likely because of too little training data; training one on a larger combined dataset is future work.

Publication

Junxiao Shen, Khadija Khaldi, Enmin Zhou, Hemant Bhaskar Surale, and Amy Karlson. 2024. Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and Pre-Training. IEEE Transactions on Visualization and Computer Graphics 30, 11 (November 2024), 7118-7128. https://doi.org/10.1109/TVCG.2024.3456198

BibTeX

@article{shen2024Gesture2Text,
  title     = {Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and Pre-Training},
  author    = {Shen, Junxiao and Khaldi, Khadija and Zhou, Enmin and Surale, Hemant Bhaskar and Karlson, Amy},
  journal   = {IEEE Transactions on Visualization and Computer Graphics},
  volume    = {30},
  number    = {11},
  pages     = {7118--7128},
  year      = {2024},
  publisher = {IEEE},
  doi       = {10.1109/TVCG.2024.3456198}
}

Contact

Questions? Feel free to contact:

• Junxiao Shen, Reality Labs Research, Meta and University of Bristol
• Hemant Bhaskar Surale, Reality Labs Research, Meta: hemant.surale [at] gmail.com

Figures are from the paper (© 2024 IEEE) and are reproduced here by the authors.