blocks in VR, but what was immediately noticeable was how accurate and dexterous Leap Motion's sensor is at tracking hand movements. Rather than a simple series of gestures, the virtual hands are able to mimic movements down to individual finger movements – all without the use of any hand-worn sensors.
Murati said what the team at Leap Motion has done is essentially solve a very difficult computer vision problem ... all with a controller made with off-the-shelf parts. The infrared sensor is able to track hand and finger movements (with a 180-degree field of view), which are then filtered into the controller's software API, and converted into virtual hand movements. All-in the unit comes in at about 8-mm in height according to Leap Motion's specs. “You have an IR window, beneath that two cameras that see hands and reconstruct a model of each hand,” Murati said.
The real leg work is done on the software end. In 2016, Leap Motion released an update to Orion, the hand-tracking software that powers its controller, with the aim of improving performance, particularly toward mobile VR. “Since we already had the hardware out we could make dramatic updates to the software,” Murati said. “That's what we did with Orion. We took a look at the tracking pipeline and asked ourselves what would this look like in VR? And we re-did it from the ground up.”
In the past several months, Leap Motion has shifted its focus toward mobile VR – the next wave of hardware that will require no external sensors to track user movements. “To do this we needed the controller to have higher performance, lower power, and at least ten times faster hand tracking, while also being smoother and more accurate,” Murati said. “We wanted to focus on mobile VR because ultimately we want VR to be accessible. Mobile VR experiences are considered second class to PC-based VR right now and that comes down to the fact that input is very limited with things like tapping on your head to control the [Samsung] Gear VR, for example.”
Leap Motion is currently working with OEMs to get its sensor embedded into the next generation of VR and AR headsets coming to the market. The company also maintains a strong presence in the DIY community – encouraging developers at all levels to undertake projects, both in and outside of VR, using its sensor technology. Researchers at The Burke Medical Research Institute in White Plains, NY are using the Leap Motion controller for physical therapy for stroke patients. Mercedes-Benz has designed a concept car that uses the controller for infotainment and dashboard control. Developers have used the controller in conjunction with AutoCAD. And, of course, all manner of independent game development has come about through Leap Motion's annual 3D Jam competition for projects developed for the controller.
With so much engagement with its developer community, and ongoing discussions with OEMs, likely the biggest question on anyone's mind is how soon will Leap Motion's controller allow for haptic feedback and touch sensitivity? Unfortunately, Murati said Leap Motion