What happens inside a 3D click

Quick facts

A VR selection is not a single instant. Between press and release, which took about 0.17 seconds on average in our study, the ray, eyes, head, and hand all keep moving. Overall, our work contributes:

1. Eight measures that characterize pointer movement, gaze, head, and hand movement, and the alignment between gaze and pointer during a selection.
2. A validation study with 36 participants, five 3D pointing techniques, and 13,885 trials, showing that the measures track accuracy, throughput, task load, exertion, and user experience.
3. Design implications for 3D pointing: tolerate less stable pointing, prioritize speed and low latency, and design for coordination across modalities.

Abstract

Measures of target selection in virtual reality (VR) have focused mainly on overall performance, such as speed and accuracy. However, a selection is not atomic: numerous things happen during the short “click” when a target is selected. In this work, we formalize a set of measures that provide insight into what happens during this process. The measures characterize the pointer movement, user behaviors (gaze, head, hand and finger), and coordination between modalities. To validate the measures, we collected pointing data from 36 participants using five different 3D pointing techniques. Our results show that these measures effectively characterize pointing performance, task load, perceived exertion, ease of use/learning, comfort, and sense of agency. We found that tolerating less “stable” pointing and considering alignment between modalities both improve pointing performance. Our findings suggest that 3D pointing should prioritize speed, allow for more movements during selection, and consider coordination of different modalities.

The idea: look inside the click

When we compare pointing techniques in VR, we usually report speed and accuracy, which treat a selection as a single instant. Instead, we recorded every sensor sample between selection start and selection end (about 12 samples per click) and computed measures over that window.

We started from 31 candidate measures and kept the eight that correlated with performance and user ratings and were easiest to interpret. They fall into three groups:

• Pointer: how much the ray direction jitters (pointer direction variability) and how far the cursor wanders on the target plane (hit point variability).
• Single modality: how far the eyes and head turn (gaze and head direction range), how far the palm moves (palm position drift), and how much the index fingertip moves (index fingertip variability).
• Alignment: how far apart the pointer and gaze land on average (mean alignment distance), and where the pointer was at the moment it agreed most with gaze (alignment-adjusted endpoint).

The measures only need the sensor samples from inside the click, so they are cheap enough to compute at runtime.

Experiment

Techniques. We chose five techniques that cover different pointing modalities (controller, gaze, hand, finger) and selection triggers (button, pinch, thumb tap), from widely used controllers to a novel wearable.

Apparatus. A Unity testbed on a Meta Quest Pro. Gaze, head, and controller were recorded at 72 Hz and hand tracking at 60 Hz. The ring prototype (ElectroRing) combines a 9-axis IMU with electrodes that detect the thumb tap.

Task. Gray spherical targets appeared one at a time at random positions within ±35° of the center of view, 2 m away, in two sizes (5° and 10°). Each technique and size combination had 5 practice and 40 test trials.

Participants. 36 people (14 women, 22 men, mean age 36.1), 35 of them right-handed. After removing outliers, we kept 13,885 valid trials (98.4%).

Measures. Hit or miss for every trial, plus ratings after each block: NASA-TLX task load, Borg CR-10 perceived exertion, and ease of use, ease of learning, comfort, and sense of agency.

What do we learn?

1. Steadier clicks are more accurate

Across individual trials, the less the ray moved during the click, the more often it hit the target.

2. But techniques that let you hold still were slower and more tiring

Ranked by throughput, which combines speed and accuracy, the order was Controller (2.49 bits/s), Gaze-Contr (2.23), Gaze-Pinch (1.94), Ring (1.75), and Handray (1.34). Handray had the least pointer movement during the click and the highest physical demand. Controller had the most movement and the lowest physical demand.

The same pattern held for task load, perceived exertion, ease of use, comfort, and sense of agency: less movement during the click went with worse ratings. Holding a ray perfectly still in mid-air takes effort in a way that resting a finger on a touchscreen does not, so speed mattered more than accuracy for overall performance.

3. Eyes and hands work as a team

For the three techniques that don’t point with gaze (Controller, Handray, Ring), we scored each click using the alignment-adjusted endpoint instead of the release position. The hit rate went from 91.8% to 93.6%: 196 misses became hits and 52 hits became misses. We also found correlations across modalities. For example, gaze range was linked to hand and finger exertion, and mean alignment distance was linked to task load.

Design implications

1. Tolerate less stable pointing. Don’t make users hold a ray perfectly still to select.
2. Prioritize movement speed over pointing accuracy.
3. Prioritize low latency over sensing accuracy. Participants called Handray “laggy” and felt more in control when the pointer “actually followed the movement”.
4. Design for coordination. Gaze, head, and hands move together during a click, and using that alignment can rescue near misses.

Limitations

We tested spherical targets at a fixed 2 m distance on a vertical plane, in fully virtual environments only, using the Quest Pro’s default eye and hand tracking. Handray was sampled at 60 Hz versus 72 Hz for the others, which may account for some of its differences. Extending the measures to AR, other target layouts, and real-world tasks is future work.

Publication

Junhan Kong, Jacqui Fashimpaur, Michael J. Proulx, Hemant Bhaskar Surale, Jacob O. Wobbrock, and Amy Karlson. 2026. Quantifying 3D Pointing: Characterizing What Happens During Pointing Selection in Virtual Reality. In Proceedings of the 28th International Conference on Multimodal Interaction (ICMI ‘26), Napoli, Italy. ACM, 273-289. https://doi.org/10.1145/3776574.3831138

BibTeX

@inproceedings{kong2026quantifying,
  title     = {Quantifying 3D Pointing: Characterizing What Happens During Pointing Selection in Virtual Reality},
  author    = {Kong, Junhan and Fashimpaur, Jacqui and Proulx, Michael J. and Surale, Hemant Bhaskar and Wobbrock, Jacob O. and Karlson, Amy},
  booktitle = {Proceedings of the 28th International Conference on Multimodal Interaction},
  series    = {ICMI '26},
  location  = {Napoli, Italy},
  pages     = {273--289},
  year      = {2026},
  publisher = {ACM},
  doi       = {10.1145/3776574.3831138}
}

Contact

Questions? Feel free to contact:

• Junhan Kong, University of Washington: junhank [at] uw.edu
• Hemant Bhaskar Surale, Reality Labs Research, Meta: hemant.surale [at] gmail.com

Figures from the paper are reproduced under its CC BY 4.0 license.