Brown University researchers open source mobile AR hand-object interaction library

As the metaverse buzz continues to heat up, discussions about the technologies that will power it are also becoming more mainstream. In a previous article published by VentureBeat, Gary Grossman, senior vice president of the technology practice at Edelman and Global Lead of the Edelman AI Center of Excellence, said AI, AR, VR, 5G and blockchain may converge to power the metaverse. However, other experts like Louis Rosenberg, CEO and chief scientist at Unanimous AI, believe augmented reality (AR) — not virtual reality (VR) — will be the heart of the metaverse.

A new chapter in the AR conversation is smartphone AR, which allows everyday users to interact with the virtual environment. Many companies in the space are focusing on AR as a tool to improve the effectiveness and efficiency of the virtual sales processes. According to Gartner, current trends show 75% of capital-equipment-intensive industries will use AR as a key component for cost reduction and avoidance among frontline workers.

Enterprise-facing AR and VR solutions are evidence of the ever-increasing power of AR, with Google’s powered Project Tango and Apple’s IR sensors showing the potentials of AR to turn simple devices into genius supercomputers capable of understanding, predicting and anticipating the use of space. AR’s promises are exciting, but the cost of devices and the expertise it requires hinder its development and deployment at scale.

While AR seems to be gaining an inroad into the enterprise, Jeff Huang, academic researcher and Computer Science professor at Brown University, told VentureBeat that the enterprise is an escape route for failed consumer AR. Huang said certain Chinese manufacturers will launch wireless AR glasses in 2024 – 2026 that will be more accessible than any enterprise-grade AR by big tech.

However, Huang and other collaborators on Brown HCI’s Portalble concept believe smartphone AR will become ubiquitous and want to aid the race to a digital-first future by open sourcing the project, which the research team has been building for years. The open-source mobile AR hand-object interaction library currently runs on phones, without any extra hardware needed at all, but Huang said there will eventually be a more premium version with a Leap Motion controller attachment.

As an open-source documentation project, Portalble allows developers to use its codes in their projects. Without the needed gizmos that usually accompany the use of AR, Portalble enables a normal android phone to become a centerfold for AR action. 

Time is ticking for a convergence of worlds 

With AR, users can interact with an object in the augmented space using their body parts. This creates a distinctness from the metaverse, where the entire experience happens in a space outside the physical world as we know it. Perhaps the AR experience reveals the need for a convergence of worlds — a shift from the norm — but merging technologies and bringing together devices to actualize the experience is often problematic. 

Aleissia Laidacker, global director of creative technology at The Mill, wrote in a Forbes article, that due to the cost, there is a long way to go before people can fully tap into the potential of AR. But Huang seems to think the time for consumer AR is nearer than we think, noting that 2025 – 2026 will be the year when consumer AR becomes impressive for everyday use. He, however, noted that we won’t begin to see AR everywhere until 2029. He cited the issues of technological advancements and the need to build and adapt devices for an overall seamless experience as part of the hurdles to be crossed before AR can become commonplace. Not only that, but he also noted that light field cameras will become a part of AR glasses to enable better tracking in the near future. 

This timing is in line with expert predictions for the bloom of the metaverse. Gartner predicts that the metaverse will not be fully mature before 2030 and advocates for immediate testing of devices and technologies that are mission-critical to the metaverse. AR is one of those technologies. Similarly, a Forrester article identifies a blend of VR, mixed reality (MR) and importantly AR as the keys to unlocking the best of the metaverse.   

With time ticking for the convergence of the different technologies into the building block of a bold new world, Huang is confident that the scope of work done on Portalble is sufficient to fit it into the metaverse discussion. Although he notes current challenges with battery, overheating and field of view (FOV) must be solved before that can be actualized. 

AR for all

Portalble AR can turn any android phone into an AR portal, allowing users to transcend the real world and move virtual objects with their hand. Several actions can be performed like sketching, manipulating and throwing objects, as demonstrated on Portalble’s repository page.  The difference between what Portalble offers and what is popular right now in the AR space is that Portalble enables direct interactions with the objects, rather than the usual hands-in-the-air gesture. 

With the open-source nature of Portalble’s AR infrastructure, security might become a concern for developers and researchers. Open source is a double-edged sword: while serving as a backbone of digital innovation, it also poses unique threats. However, Huang defended the decision to open source Portalble, claiming it’s best to leave the project so, in the spirit of academic research and fair belief in the untapped potential of every other researcher in the space. He hopes that someday, other researchers will be able to build on the work that has already been done.

Huang said while there are other players in the space like Manomotion, he claims Portalble is the best phone-only hand-object interaction technology. He said a major edge Portalble has over other AR projects is that it enables developers to draw on the framework of Portalble and include it in their games and apps. Huang’s goal is to make AR available everywhere, knowing that others are building for other devices — hence why AI for mobile phones has been a considerable focus for the company. Developers who are interested in Portalble’s AR solutions can check out the GitHub repository.