Google develops cheap, AI-assisted controller tracking for VR headsets

Though it’s currently available in tethered VR headsets using position-tracking sensors for controllers, six degrees of freedom (6DOF) controller tracking is a holy grail of sorts for standalone VR headsets. As discovered by Adobe’s Dimitri Diakopoulos and noticed by RoadtoVR, Google has come up with an inexpensive AI-assisted way to add 6DOF controller tracking capabilities to a standalone headset — full details of which are included in a recent Google research paper.

The trick begins by using the same two fisheye cameras already found in the standalone VR headset to additionally track the positions of the user’s hands and arms. On the software side, it relies on a neural network trained with hand, arm, and controller positions to estimate the real-world controller’s unknown location within the user’s known grasp. A dataset of 547,000 stereo image pairs was used to train the network, compiled from 20 users doing 13 movements in varied lighting.

“Our key observation is that users’ hands and arms provide excellent context for where the controller is in the image,” wrote the Google researchers, “and are robust cues even when the controller itself might be occluded. To simplify the system, we use the same cameras for headset 6-DoF pose tracking on mobile HMDs as our input. In our experiments, they are a pair of stereo monochrome fisheye cameras. We do not require additional markers or hardware beyond a standard IMU based controller.”

At the moment, Google is able to determine the controller’s position within a mean 33.5 millimeter error rate — only 1.32 inches — when it knows hand and arm positions. Presently, the system runs at 30FPS on a single mobile CPU core and doesn’t require any object to be glowing to track its location properly. But the company believes it can optimize the system for faster and more accurate tracking, as well as to expand it to controller-less hand tracking. Google is expected to have more to say on this at its I/O developer conference, which starts on May 8.