Jeremy Hartmann*, Hemant Bhaskar Surale*, Aakar Gupta, Daniel Vogel
University of Waterloo
*Authors have contributed equally to this work.
Workshop paper · CHI 2018, Montreal, QC, Canada
PDF arXiv Publication BibTeX
A chart with reality-virtuality on the horizontal axis and conformity from none to full on the vertical axis, with four areas A, B, C, D. Beside it, four sketches of a person at a table with four blocks: B in AR and C in VR with two real and two virtual blocks, A using a tablet with real blocks on the table, and D in VR where the table and blocks are all virtual.
The conformity spectrum and the four usage contexts plotted on it. A uses a tablet next to real blocks (no conformity), B is in AR with two virtual and two real blocks (high conformity), C is in VR where two virtual blocks and the table are aligned with real ones (high conformity), and D is in VR with everything virtual (no conformity). The sketches are arranged in the same positions as on the chart. Figures 1 to 3 from the paper.

Quick facts

Jeremy Hartmann and Hemant Bhaskar Surale (University of Waterloo) contributed equally to this four-page workshop paper. It proposes a vocabulary rather than reporting a study: a second axis for describing XR environments, and a way to reason about what breaks when people in different environments work on the same physical objects. Overall, our work contributes:

  1. The conformity spectrum: an axis orthogonal to Milgram et al.’s reality-virtuality continuum that describes how closely the virtual content adheres to the real-world physical characteristics surrounding the user.
  2. Four block states (owned, shared, unowned, orphaned) that describe, from one collaborator’s view, whether an object is physical or virtual for each person.
  3. An analysis of six collaboration scenarios between four usage contexts, using awareness and synchronization to find where collaboration breaks down and what could mitigate it.

Abstract

The concept of a conformity spectrum is introduced to describe the degree to which virtualization adheres to real world physical characteristics surrounding the user. This is then used to examine interaction challenges when collaborating across different levels of virtuality and conformity.

The idea: track where each block is real

When two people collaborate across XR, the same block can be a physical object for one of them and a virtual one for the other. Moving a block is then not symmetric: if the block is physical for someone else, only that person can actually move it in the real world.

Four block states, from one collaborator's view real for them virtual for them real for me virtual for me Shared me them Owned me them Unowned me them Orphaned me them physical block virtual block When A moves B's real block 1. In sync real and virtual match 2. A moves it on a tablet B is out of sync 3. B moves the real block in sync again
Each block's state depends on who has its physical counterpart. In Scenario 1, A (on a tablet) moves a block that is physical for B (in AR). The shared virtual block moves, but B's real block stays put, so B is out of sync until B moves the real block by hand.

What we propose

A second axis for XR. Milgram et al.’s reality-virtuality continuum classifies XR technologies (AR, VR, MR) by how virtual they are. We add an orthogonal axis, conformity, for how closely interaction entities in the user’s locus of attention adhere to the physical characteristics of their surroundings. We focus on conformity to time, space, and geometry, but material texture, tactile texture, mass, sound, and smell could be included as well. The motivation is a missing name for environments where the geometry of virtual objects is mapped directly onto real objects occupying the same space, which we call a virtual-conforming reality.

The corners of the space. Bottom-left: digital interactions with no connection to the immediate physical environment, such as conventional computers. Upper-left: perfect Spatial Augmented Reality, where every digital interaction maps one-to-one to physical geometry. Upper-right: a fully virtual environment that perfectly conforms to the immediate real environment. Bottom-right: what most people consider current VR, which does not conform to the real environment at all.

A shared task. Two remote users coordinate moving four blocks into a final configuration, and either can move any block. The task stands in for real-world collaborations around a physical artifact, such as assembling furniture, repairing a kitchen sink, or getting a 3D printer working. We examine each pairing with two aspects of remote collaboration:

  • Awareness of each other’s actions, to communicate movements and to prevent simultaneous access to the same block.
  • Synchronization of the shared task state across both collaborators.

Our hypothesis is that the collaborative experience is a function of the relative conformity between the usage contexts and the objects in them. The paper’s Table 1 rates awareness and synchronization for each context and block state. For orphaned and unowned blocks, both are good in every context. The problems are with owned and shared blocks:

Context Owned and shared blocks
A (tablet) not applicable
B (AR) partial awareness, poor sync
C (VR, conforming) good awareness, poor sync
D (VR) not applicable
Awareness and synchronization when manipulating owned or shared blocks, summarized from Table 1 of the paper.

What the scenarios show.

  • A and B (tablet and AR). If A moves a block that is real for B, B’s real block and its moved virtual version diverge. B needs the displacement shown clearly (for example with a colour change or ghosting) and a prompt to move the real block. A has to re-sync all of A’s own real blocks at the end: low conformity means synchronization is postponed until after collaboration.
  • A and C (tablet and conforming VR). C stays aware, since the moved block is now visibly virtual, but synchronization is harder: C may not know a block no longer conforms until C tries to manipulate it. Visual cues need to show that a block has moved.
  • C and D, B and D (high conformity with fully virtual VR). When D moves a block, the shared virtual space updates, but the high-conformity partner’s physical blocks fall out of sync. D’s action is closer to a suggestion: agency over the physical blocks stays with the partner, who decides whether to apply it.
  • B and C (AR and conforming VR). Moving an owned block keeps the shared state in sync, but moving a shared or unowned block desynchronizes the other person. One option is to let each collaborator move only owned blocks; otherwise some mechanism is needed to maintain the shared state.
  • A and D (no conformity on either side). Awareness and synchronization in the shared virtual space are trivial, but once both agree on the result, each has to leave the interaction and arrange the physical blocks separately.

Open questions

  1. Other tasks. The scenarios cover a single block-arrangement task. Many other interactions with different constraints could be analyzed the same way.
  2. Evaluation. Future work will evaluate tasks defined on this space, with usability experiments on remote and co-located collaboration under different levels of conformity.
  3. Re-establishing conformity. How should a system show that a block no longer conforms, prompt someone to move it, or limit who can move which blocks?
  4. More dimensions. The paper considers time, space, and geometry. How conformity to texture, mass, sound, or smell would change the analysis is open, and depends on whether virtualizing technologies can synthesize them.

Publication

Jeremy Hartmann*, Hemant Bhaskar Surale*, Aakar Gupta, and Daniel Vogel. 2020. Using Conformity to Probe Interaction Challenges in XR Collaboration. Workshop paper, CHI 2018, April 21 to 26, 2018, Montreal, QC, Canada. arXiv:2004.05235. https://arxiv.org/abs/2004.05235 (*equal contribution)

BibTeX

@misc{hartmann2020using,
  title         = {Using Conformity to Probe Interaction Challenges in XR Collaboration},
  author        = {Hartmann, Jeremy and Surale, Hemant Bhaskar and Gupta, Aakar and Vogel, Daniel},
  year          = {2020},
  eprint        = {2004.05235},
  archivePrefix = {arXiv},
  primaryClass  = {cs.HC},
  url           = {https://arxiv.org/abs/2004.05235},
  note          = {Workshop paper, CHI 2018, Montreal, QC, Canada. The first two authors contributed equally}
}

Contact

Questions? Feel free to contact:

  • Jeremy Hartmann, University of Waterloo: j3hartma [at] uwaterloo.ca
  • Hemant Bhaskar Surale, University of Waterloo: hemant.surale [at] gmail.com

Figures from the paper are reproduced under its CC BY 4.0 license.