Design

Facebook wants to give you Force-like powers inside of AR... using this wristband

Facebook Reality Labs, the social giant’s AR/VR wing, has created a wrist-worn device that translates your neural signals into digital commands inside of the AR world. It’s the closest thing to the Force or having a superpower.

Despite years of development and hundreds of millions of dollars spent on research and development, no tech company has “cracked” AR in any meaningful way. Not Apple, not Google, not Microsoft, not Snapchat, and not Facebook.

Kim Kulish/Corbis News/Getty Images

We are still many years (if not decades) away from a reimagined world where augmented reality blends virtual objects and interfaces seamlessly into our lives. And that’s if AR ever becomes a viable computing platform like desktop and mobile.

🎥: Facebook

What’s the input for AR?

Headsets will eventually shrink to the size of glasses (at least that’s the hope), but the most important question is input. How will we interface with the AR world? For AR to succeed as a platform, there needs to be a universal input that’s easy to understand like the mouse and keyboard and touchscreen and multi-touch. 🎥: Apple

We are very much still in the experimental phase. Facebook Reality Labs, the social giant’s AR/VR division, says it sees AR taking off as a platform when contextually-aware AI that understands what you’re trying to do and then provides the appropriate commands and actions converges with an “ultra-low-friction input.”

Getty

The Intelligent Click

Facebook’s AI is some of the smartest there is when it comes to anticipating your needs. But the “ultra-low-friction input” still has a long way to go before we’re interfacing with AR naturally. For now, Facebook Reality Labs has a near-term input it’s calling the Intelligent Click.

Facebook

The Intelligent Click is this AI + input inside of a wrist-worn prototype that Facebook has created to interface with augmented reality. Instead of tapping or swiping a piece of glass or using a hand controller (like with VR), the wrist gadget essentially gives you virtual hands to control AR objects.

Facebook

This wrist-worn gadget uses EMG (electromyography) to “translate electrical motor nerve signals that travel through the wrist to the hand into digital commands.” The wearable isn’t detecting motion, it’s actually turning neuron signals inside of your muscles into actionable commands.

Facebook

1mm

The amount of precise finger motion that Facebook says the EMG wearable can detect.

🎥: Facebook

“What we’re trying to do with neural interfaces is to let you control the machine directly, using the output of the peripheral nervous system — specifically the nerves outside the brain that animate your hand and finger muscles.”

Thomas Reardon, Director of Neuromotor Interfaces at Facebook Reality Labs

Facebook

The application of the Intelligent Click with the wristband is pretty limited right now. You can “pinch and release the thumb and forefinger” or “click your fingers together” to manipulate AR objects and buttons and such.

🎥: Facebook

Facebook tells me you could even add more fingers and control them with ease. Or, you could give a person with disabilities who may not have fingers virtual ones. In one demo, a person who was born without fingers was able to virtually grab objects using their own hand and the wrist-worn input device.

🎥: Facebook

Becoming the keyboard

Another prototype Facebook involves a virtual QWERTY keyboard that works on any interface. Using machine learning and AI for autocorrection, you could potentially type on any surface without a physical keyboard. With practice, Facebook says it’s possible to touch type as fast as using a virtual keyboard that’s personalized to your typing style. Naturally, I’m skeptical. 🎥: Facebook

FEELING THE UNREAL

Then, there’s haptics. With a mouse, you feel a click, and with touchscreens, you might feel a buzz. Facebook says its wristband device can vibrate and pulse depending on different interactions. While it’s no PS5 Dual Sense controller, Facebook says it's able to “approximate the sensation of pulling back the string of a bow” for a virtual bow and arrow.

Facebook
🎥: Facebook

Couple this wrist-controlled gesture input with contextual software that shows you what you need when you need it (i.e. showing a timer when you’re cooking instead of asking an assistant to set one) and you have the potential for a powerful new platform.

🎥: Facebook

This contextual AI is essentially Siri Suggestions or Android’s App Actions taken to the next level. It’s a platform that anticipates your computing needs instead of one that’s waiting for user action.

Facebook

“The intelligent click gives you the ability to take these highly contextual actions in a very low-friction manner because the interface surfaces something that’s relevant based on your personal history and choices, and it allows you to do that with minimal input gestures.”

Tanya Jonker, Research Science Manager at Facebook Reality Labs

Facebook

More Private (sorta)

I know what you’re saying: Facebook care about privacy? Nonsense! While we don’t have any idea how AR and the Intelligent Click would be used or integrated with Facebook’s existing platforms, wrist-based gesture controls are more private than, say, using voice controls that everyone can hear.

Facebook

Work in progress

As exciting as Facebook’s wristband and Intelligent Click sound, they’re both still prototypes. A working prototype — but one that’s still fluid and being worked on. It’s not a finished product and there’s no telling if it is the right form of input for AR. Facebook’s simply sharing its creation in an effort to create transparency and get feedback from the community for further development.

Facebook

If Apple ever releases its long-rumored AR headset/glasses, make sure to pay attention to the input. It’s going to be make-or-break for the product. Ditto for Facebook, Snapchat, or any other tech company that’s building an AR headset.

Getty

Share