This AI teaches robots how to walk

Artificially intelligent (AI) systems have imbued robots with the ability to grasp and manipulate objects with humanlike dexterity, and now, researchers say they’ve developed an algorithm through which machines might learn to walk on their own. In a preprint paper published on Arxiv.org (“Learning to Walk via Deep Reinforcement Learning“), scientists from the University of California, Berkeley and Google Brain, one of Google’s artificial intelligence (AI) research divisions, describe an AI system that “taught” a quadrupedal robot to traverse terrain both familiar and unfamiliar.

“Deep reinforcement learning can be used to automate the acquisition of controllers for a range of robotic tasks, enabling end-to-end learning of policies that map sensory inputs to low-level actions,” the paper’s authors explain. “If we can learn locomotion gaits from scratch directly in the real world, we can in principle acquire controllers that are ideally adapted to each robot and even to individual terrains, potentially achieving better agility, energy efficiency, and robustness.”

The design challenge was twofold. Reinforcement learning — an AI training technique that uses rewards or punishments to drive agents toward goals — requires lots of data, in some cases tens of thousands of samples, to achieve good results. And fine-tuning a robotic system’s hyperparameters — i.e., the parameters that determine its structure — usually necessitates multiple training runs, which can damage legged robots over time.

“Deep reinforcement learning has been used extensively to learn locomotion policies in simulation, and even transfer them to real-world robots, but this inevitably incurs some loss of performance due to discrepancies in the simulation, and requires extensive manual modeling,” the paper’s authors point out. “Using such algorithms … in the real world has proven challenging.”

Above: The Minitaur robot taught to traverse unfamiliar terrain with the researchers’ AI system.

In pursuit of a method that would, in the researchers’ words, “[make it] feasible for a system to learn locomotion skills” without simulated training, they tapped a framework of reinforcement learning (RL) known as “maximum entropy RL.” Maximum entropy RL optimizes learning policies to maximize both the expected return and expected entropy, or the measure of randomness in the data being processed. In RL, AI agents continuously search for an optimal path of actions — that is to say, a trajectory of states and actions — by sampling actions from policies and receiving rewards. Maximum entropy RL incentivizes policies to explore more widely; a parameter — say, temperature — determines the relative importance of entropy against the reward, and therefore its randomness.

It wasn’t all sunshine and rainbows — at least not at first. Because the trade-off between entropy and the reward is directly affected by the scale of the reward function, which in turn affects the learning rate, the scaling factor normally has to be tuned per environment. The researchers’ solution was to automate the temperature and reward scale adjustment, in part by alternating between two phases: a data collection phase and an optimization phase.

The results spoke for themselves. In experiments in OpenAI’s Gym, an open source simulated environment for training and testing AI agents, the authors’ model achieved “practically identical” or better performance compared to the baseline across four continuous locomotion tasks (HalfCheetah, Ant, Walker, and Minitaur).

And in a second, real-world test, the researchers applied their model to a four-legged Minitaur, a robot with eight actuators, motor encoders that measure motor angles, and an inertial measurement unit (IMU) that measures orientation and angular velocity.

They developed a pipeline consisting of (1) a computer workstation that updated the neural networks, downloaded data from the Minitaur, and uploaded the latest policy; and (2) an Nvidia Jetson TX2 onboard the robot that executed said policy, collected data, and uploaded the data to the workstation via Ethernet. After 160,000 steps over two hours with an algorithm that rewarded forward velocity and penalized “large angular accelerations” and pitch angles, they successfully trained the Minitaur to walk on flat terrain, over obstacles like wooden blocks, and up slopes and steps — none of which were present at training time.

“To our knowledge, this experiment is the first example of a deep reinforcement learning algorithm learning underactuated quadrupedal locomotion directly in the real world without any simulation or pretraining,” the researchers wrote.