Time
Jan - Apr 2022
Industry
Mixed Reality
team
Haoran Xu, Carina Geng,
Duoning Zheng, Zoe Zhang
Role
MR Designer
MR Gesture Library in a Nutshell...
Problem
Current MR interactions lack intuitive, consistent, and efficient gesture controls, leading to high learning costs and limited usability.
Solution
The MR Gesture Library introduces a set of natural, universal gestures for MR environments, enhancing user experience by improving intuitiveness, efficiency, and consistency across applications.
What if...
...things with hands in Mixed Reality?
People want to select & multi-select
People want to group & ungroup
People want to Copy & Paste
People want to delete
People want to undo
What does Metaverse & Mixed Reality mean in our eyes?
Why will the Metaverse become a reality,
and why is MR the access point?
Currently, a lot of VR interactions are focused on gestures and behavior that people familiar with. However the metaverse can go beyond that. What if I can go beyond the traditional ways of interaction and improve the VR interaction?
However, What are the interaction barriers in metaverse (MR interaction)?
Based on our research of the XR interaction, I found that the user experience is insufficient, and we summarized them into the following bullet points.
AS A RESULT
I aim to explore the possibilities of universal gestural interfaces to make interactions more intuitive, usable, and efficient.
And users can apply interactions everywhere as simple as sending emojis.
target user
I researched current VR/AR/MR users and designed a persona based on the findings.
concept
I made a game to help game designers try out our gestures library, experiencing them in an immersive environment.
User needs to use various interactions to escape from 2 rooms.
Scenario 1 room
(Entry interaction)
Scenario 2 factory
(advanced interaction)
design principles
According to Don Norman's article: Gestural Interfaces: A Step Backwards In Usability, I set design principles.
Visibility
Perceptible & Obvious signifiers.
Instant Feedback
Immediately, no matter it is a change of the status or showing the result of interaction.
Non-destructive operations
Hence the importance of "undo"
Discoverability
All operations can be discovered by systematic exploration of menus or Guidance
Reliability
Operations should work. Period. And events should not happen randomly.
Consistency
Logical and systematical interface. Gestures should be applied consistently across systems to minimize cognitive load.
Develop
Gesture Library
Establishing design principles inspired by current OS interactions and adapting them for VR.
Natural Gestures
Select
Rotate
Move
Delete
Drag in space
Gestures for editing
Multi-select
Multi-selected
Group/Ungroup
Slicer
Selection
Scan
Gestures for virtual tools
Menu
Picker/Eyedrop
storage
making the interaction more intuitive
I led research observing people's real-life task completion behaviors, analyzing their reactions to identify themes that informed the creation of our final gesture set.
improve the efficiency of the gestures
In order to improve the efficiency of gestures, I studied the interactions in the current interfaces and improved the efficiency of each gestures in XR. As a result, I collaborated with the team to develop the following gestures."
what i learned & The Impact
The Metaverse requires an infrastructure to support seamless interactions. Just like the current screen based OS, I hope the work inspires the creation of universal interaction standards for all XR applications
© Copyright 2025 • All Rights Reserved
haoranfuture@gmail.com