Switching modes with bare hands in VR

Quick facts

In VR, the same hand movement can draw, select, or move things depending on the current mode, so people switch modes constantly. We measured how fast and how reliably eleven bare hand techniques do that, in two experiments with 28 participants. Overall, our work contributes:

1. Dominant hand postures with large motions (fist, palm, point) are error prone and less preferred, but a subtle variation of the pinch is comparable to the fastest non-dominant techniques.
2. Comparable to touch. Except for a few dominant techniques, bare hand mode-switching times are comparable to most touch methods.
3. A pinch can replace a button. Using a dominant hand pinch as the manipulation trigger performs about the same as a controller button.
4. A full ranking of all techniques, from the non-dominant controller button and dominant middle finger pinch (fastest) to dominant pointing (slowest).

This paper is the mid-air follow-up to our study of mode switching on touchscreens, and it uses the same experimental protocol so the two can be compared.

Abstract

We present an empirical comparison of eleven bare hand, mid-air mode-switching techniques suitable for virtual reality in two experiments. The first evaluates seven techniques spanning dominant and non-dominant hand actions. Techniques represent common classes of actions selected by a methodical examination of 56 examples of prior art. The standard “subtraction method” protocol is adapted for 3D interfaces, with two baseline selection methods, bare hand pinch and device controller button. A second experiment with four techniques explores more subtle dominant-hand techniques and the effect of using a dominant hand device for selection. Results provide guidance to practitioners when choosing bare hand, mid-air mode-switching techniques, and for researchers when designing new mode-switching methods in VR.

The idea: isolate the cost of one mode switch

With bare hands there are no buttons, so something has to tell the system whether a hand movement means “draw in the default mode” or “draw in the other mode”. In our setup, a thumb-index pinch starts and stops drawing, and a held hand posture (with either hand) selects the mode. The system reads the posture at the instant the pinch starts a line.

The measure comes from the “subtraction method” used before for mouse, pen, and touch. Every line drawing includes some overhead for moving to the next start point, so comparing single lines would mix that overhead into the result. Instead, we compare full cycles: two lines in the compound task (switch in, draw, switch out, draw) against two lines in the baseline task (both default). Subtracting the mean baseline cycle from the mean compound cycle leaves the cost of the switch.

To choose what to test, we collected 40 bare hand actions from 56 publications and device manuals, and kept those that are:

• Independent: recognizable in a single sensor frame, without timing or a gesture path (so no dwell, and no drawing an “X”).
• Kinesthetic: held by the user for as long as the mode is needed, and released to end it.
• Unconstrained: the dominant hand stays free to reach any position while the mode is held.

That left 29 actions in six classes (pinch, extend finger, close hand, open hand, raise hand, touch body), and we tested the most popular action from each class.

Experiment 1

Techniques. Four non-dominant hand postures (fist, palm, raising the hand into the field of view, touching the side of the head) and three dominant hand postures (fist, palm, pointing). In all non-dominant techniques the dominant hand draws with a pinch. As a non-bare-hand comparison, an eighth condition used the button of an HTC controller held in the non-dominant hand.

Apparatus. An HTC Vive (1080 × 1200 px per eye, 90 Hz) and a Unity application at 90 FPS. A LEAP Motion on the headset tracked the hands. Its built-in recognizer often confused fist, palm, and pinch during rapid switching, so we trained a simple per-user classifier on fingertip features (almost 99% accurate in a 6-person pilot) and taped force-sensitive resistors to the thumb and to the side of the headset to detect pinches and head touches.

Task. 3D line drawing, standing. Five pairs of spheres (100 mm diameter) are stacked with the top pair at shoulder height, and each pair is connected by pinching inside the first sphere, moving to the second, and releasing. In the baseline task all pairs are blue and drawn with the plain pinch. In the compound task, pairs alternate blue and red, and red pairs must be drawn while holding the mode-switching posture.

Participants. 16 people (5 women, all right-handed, mean age 28.5), 7 with prior VR experience.

Measures. Mode-switch time from the subtraction method, and error rates for start errors (line started outside the sphere), end errors, and mode errors (wrong mode), split into mode-in and mode-out errors. Each participant completed 720 line drawings across 8 techniques, and rated each technique afterwards.

Experiment 2

The first experiment raised two questions: would dominant hand actions that are closer to the pinch itself work better, and does the bare hand pinch trigger hurt performance compared to a controller button?

Techniques. Oriented pinch and middle finger pinch, both with the plain pinch as the default. We also re-ran the non-dominant palm and the non-dominant controller button, but this time with a controller in the dominant hand as the drawing trigger.

Participants. 12 people (5 women, 1 left-handed, mean age 29.6). Apparatus, task, and measures were the same as in Experiment 1, with 360 line drawings per participant.

What do we learn?

1. The non-dominant hand is fast and accurate

Ranked by mode-switch time in Experiment 1: non-dominant controller button (331 ms), non-dominant palm (459 ms), non-dominant fist (513 ms), head touch (528 ms), dominant palm (729 ms), raising the hand into view (793 ms), dominant fist (846 ms), and dominant pointing (955 ms). The non-dominant palm and fist were not significantly different from the controller, and their error rates were close to the 8% of the un-moded baseline task.

Even so, almost half of the participants picked the controller as their favorite.

2. Big dominant hand postures cause confusion, and pointing is the worst

Overall error rates ranged from 5.3% (controller) to 18.3% (dominant pointing), with the dominant fist at 15.3% and the dominant palm at 13.6%. Pointing also had the highest mode-in error rate (7.6%), meaning people often failed to enter the mode. We had chosen fist, palm, and point because they look very different from a pinch, but switching quickly between such different postures seemed to confuse people: “I feel Index, Fist, and Pinch are all same” (P7).

3. A subtle pinch fixes the dominant hand

In Experiment 2, the middle finger pinch (233 ms) was as fast as the non-dominant controller button (226 ms), with no significant difference, and faster than the oriented pinch (669 ms). Its only weakness was a higher end error rate (5.0%, against 0.6% for the non-dominant palm). For overall preference, 42% of participants liked the middle finger pinch and the controller.

4. A pinch works as well as a controller button

Holding the non-dominant technique constant, swapping the pinch trigger for a dominant hand controller made no significant difference to mode-switch time (controller button: p = .07; palm: p = .94). Baseline line drawing was actually slightly faster with the pinch, by about 41 ms.

Design implications

1. Use non-dominant actions for accuracy-focused tasks, such as raising a palm or touching the head. Slower dominant postures like fist or palm may suit infrequent switches where accuracy matters less, such as panning. A mix of both may work best, since the non-dominant hand may tire over long periods.
2. Avoid the pointing posture for frequent mode switching. It is a common mid-air action, but it was the slowest and most error-prone technique here.
3. If the dominant hand must switch modes, make it subtle, like a middle finger pinch that switches and starts the action at once.
4. Avoid rapid switching between very different dominant postures. Variations on the pinch avoided the confusion we saw with fist, palm, and point.
5. A pinch is a practical alternative to a controller button for triggering manipulation.

Limitations

We focused on absolute, single-point input for direct manipulation, and did not consider non-hand input such as feet, voice, or gaze. To make sensing reliable for the experiment, we added force sensors and a per-user posture classifier instead of relying only on the hand tracker. Sessions were designed to take under an hour, and non-dominant techniques may become fatiguing over longer periods of use. More subtle gestures, such as single-hand microgestures, remain to be tested.

Publication

Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2019. Experimental Analysis of Barehand Mid-air Mode-Switching Techniques in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19), Glasgow, Scotland, UK. ACM, Paper 196, 1-14. https://doi.org/10.1145/3290605.3300426

BibTeX

@inproceedings{surale2019experimental,
  title     = {Experimental Analysis of Barehand Mid-air Mode-Switching Techniques in Virtual Reality},
  author    = {Surale, Hemant Bhaskar and Matulic, Fabrice and Vogel, Daniel},
  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '19},
  location  = {Glasgow, Scotland, UK},
  articleno = {196},
  pages     = {1--14},
  year      = {2019},
  publisher = {ACM},
  doi       = {10.1145/3290605.3300426}
}

Contact

Questions? Feel free to contact:

• Hemant Bhaskar Surale, University of Waterloo: hemant.surale [at] gmail.com

Figures are from the paper (© 2019 ACM) and are reproduced here by the authors.