We are very much still in the experimental phase. Facebook Reality Labs, the social giant’s AR/VR division, says it sees AR taking off as a platform when contextually-aware AI that understands what you’re trying to do and then provides the appropriate commands and actions converges with an “ultra-low-friction input.”
The Intelligent Click is this AI + input inside of a wrist-worn prototype that Facebook has created to interface with augmented reality. Instead of tapping or swiping a piece of glass or using a hand controller (like with VR), the wrist gadget essentially gives you virtual hands to control AR objects.
This wrist-worn gadget uses EMG (electromyography) to “translate electrical motor nerve signals that travel through the wrist to the hand into digital commands.” The wearable isn’t detecting motion, it’s actually turning neuron signals inside of your muscles into actionable commands.
“What we’re trying to do with neural interfaces is to let you control the machine directly, using the output of the peripheral nervous system — specifically the nerves outside the brain that animate your hand and finger muscles.”
Thomas Reardon, Director of Neuromotor Interfaces at Facebook Reality Labs
Facebook tells me you could even add more fingers and control them with ease. Or, you could give a person with disabilities who may not have fingers virtual ones. In one demo, a person who was born without fingers was able to virtually grab objects using their own hand and the wrist-worn input device.
Then, there’s haptics. With a mouse, you feel a click, and with touchscreens, you might feel a buzz. Facebook says its wristband device can vibrate and pulse depending on different interactions. While it’s no PS5 Dual Sense controller, Facebook says it's able to “approximate the sensation of pulling back the string of a bow” for a virtual bow and arrow.
“The intelligent click gives you the ability to take these highly contextual actions in a very low-friction manner because the interface surfaces something that’s relevant based on your personal history and choices, and it allows you to do that with minimal input gestures.”
Tanya Jonker, Research Science Manager at Facebook Reality Labs
I know what you’re saying: Facebook care about privacy? Nonsense! While we don’t have any idea how AR and the Intelligent Click would be used or integrated with Facebook’s existing platforms, wrist-based gesture controls are more private than, say, using voice controls that everyone can hear.
As exciting as Facebook’s wristband and Intelligent Click sound, they’re both still prototypes. A working prototype — but one that’s still fluid and being worked on. It’s not a finished product and there’s no telling if it is the right form of input for AR. Facebook’s simply sharing its creation in an effort to create transparency and get feedback from the community for further development.