Carnegie Mellon researchers use Twitch to collect sounds for AI research

Carnegie Mellon researchers have designed a livestreaming video game that collects audio from players to populate a database for AI research. The team’s game — Rolling Rhapsody — is specifically designed to be played on Twitch, and it tasks streamers with rolling a ball across a map to collect “treasure” while viewers record sound from their homes via an app.

It’s researchers’ belief that recordings of domestic sounds like thudding from a bedroom door or a coughing fit could be used to create a range of useful technologies. For instance, Google drew on audio from thousands of its own meetings and YouTube videos to train the noise-canceling algorithm in Google Meet. Meanwhile, a team of Carnegie Mellon researchers created a “sound-action-vision” corpus to anticipate where objects will move when subjected to physical force.

Rolling Rhapsody recently completed a fifth playtest with popular Twitch streamers. In coordination with the researchers, the streamers prompted viewers to record and upload sounds from Android and iOS companion apps. Those sounds were played back during the stream each time the streamer collected a piece of treasure so that everyone watching could listen in unison.

“Imagine being able to collect 500 microwave sounds in 10 minutes by reaching just one person — Twitch makes this possible,” the team explained in a blog post on the project’s website. “It had more than 15 million daily active users and between 2.2 million and 3.2 million monthly broadcasters in 2018. For each streamer we reach, we get many viewers because the amplifier effect takes place … It enables the ability to collect thousands of sounds from the viewers.”

Above: The Rolling Rhapsody companion app.

The hope is to one day make the sounds publicly available without compromising contributors’ identities. All Rolling Rhapsody players and viewers must opt in and provide consent to upload sounds, and the game gives them opportunities to redact sounds that accidentally capture something personal. They can also delete submissions, choose to store sounds locally, and withdraw their consent at any time.

Rolling Rhapsody is by no means perfect — it requires players at home to label recorded sounds, and about 50% of them provide the wrong label. But work on it continues with a broader field test scheduled for later this summer.

“We can use this as a proof of concept for a new kind of game experience that can result in ethical data collection from the home. We can collect data in a way that’s fun and feels good for everybody involved,” lead researcher Jessica Hammer said.  “This research doesn’t have to be limited to gathering audio data for the home. A simple extension is gathering other kinds of audio data. Then you can use the same game, just change the kinds of challenges you give the players.”

Rolling Rhapsody, which is supported by sponsorships from Philips Healthcare and Bosch, is part of Polyphonic, a larger Carnegie Mellon initiative that includes an app for sound labeling and validation and an interface where researchers can view and download sounds.