Machine learning you can dance to

Rhythmic flashes from a computer screen illuminate a dark room as sounds fill the air. The snare drum sample comes out crisp and clean by itself, but turns muddy in the mix, no matter how the levels are set. Welcome to the world of modern music-making — and its discontents.

Today’s digital music producers face a common dilemma: how to mesh samples that may sound great on their own but do not necessarily fit into a song like they originally imagined. One solution is to find and audit dozens of different samples, a tedious process that can take time to finesse.

“There’s a lot of manual searching to get the right musical result, which can be distracting and time-consuming,” says Justin Swaney, a PhD student in the MIT Department of Chemical Engineering, a music producer, and co-creator of a new tool that uses machine learning to help producers find just the perfect sound.

Called Samply, Swaney’s visual sample-library explorer combines music and machine learning into a new technology for producers. The top winner at the MIT Stephen A. Schwarzman College of Computing Machine Learning Across Disciplines Challenge at the Hello World celebration last winter, the tool uses a convolutional neural network to analyze audio waveforms.

“Samply organizes samples based on their sonic characteristics,” explains Swaney. “The result is an interactive plot where similar sounds are closer together and different sounds are farther apart. Samply allows multiple sample libraries to be visualized simultaneously, shortening the lag between imagining a sound in your head and finding it.”

For Swaney, the development of Samply drew on both his research expertise and personal life. Before coming to MIT, he had produced albums with indie musicians including Eric Schirtzinger, a drummer and co-creator of the tool. The two recorded drums in a basement and tried to improvise with cheap hardware and hacks — like hanging rugs from the ceiling to dampen reverberation. “The constraints made us get creative,” says Schirtzinger, who is now a computer science major at the University of Wisconsin at Madison.

That creativity was further honed after Swaney completed 6.862 (Applied Machine Learning). He saw an opportunity to rekindle his music production hobby by applying what he had learned from the project-based course, devising a way to automate the search for the right samples when producing a new song.

“I figured the computer could listen to samples much faster than I could,” he says. Beyond the clever use of machine learning, the real magic of Samply is that conceptually, it is founded on a deep understanding of what it takes to make music. “We aren’t just AI enthusiasts applying machine learning to music,” says Schirtzinger. “We are musicians who want better tools for making music.”

It turns out that at MIT, they aren’t the only ones with a song in their hearts. While presenting Samply at the Schwarzman College of Computing exposition last winter, dozens of faculty, staff, and students gathered around Swaney’s poster and live demonstration to exchange ideas. Some had years of experience producing music with professional software, while others simply appreciated the visualizations and sounds in the demo.

Spurred by the interest in Samply at the exposition, Swaney and Shirtzinger are in the process of turning their project into a startup company. As a first step, the two reached out to the Technology Licensing Office (TLO) for advice, which referred them to the Venture Mentoring Service (VMS).

Samply joined VMS in April and was paired with two MIT-affiliated mentors and entrepreneurs, Stephen Bayle and John Stempeck. After pitching Samply to his mentors, Swaney received sage advice on a crafting a business plan and sales strategy, and then began making connections with others interested in music technology as a business.

Samply has since been accepted into the ELEVATE accelerator, sponsored by the local digital marketing firm HubSpot, and Swaney is applying for seed funding through the MIT Sandbox Innovation Fund.

“Starting a company as a student can be daunting, but the MIT community gives us confidence,” he says. “If we can’t do it at MIT, then where can we?”

In fact, the time and attention he has spent on Samply has had an “almost paradoxical” benefit to his academic life as a graduate student. “I was spending all of my time in the lab,” he says. “When I took a step back to make Samply, I could see the forest from the trees in my research.”

Swaney found that focusing on his love of music served as an “emotional outlet,” helping to mitigate intellectual burnout. Although Samply may have taken him away from the lab bench, it has also ended up informing his research. The original idea of visualizing samples, he says, stemmed from “my work on single-cell analysis.” Applying the method to the tool clarified his thinking in the biological realm, leading to a new method to produce better clustering, or a way to better sort, recognize, and visualize groups of cells. “It was a bit like a musical theme and variation, but with my research,” Swaney says.

As for Samply, there will be a free beta version of the app launching in September, and a Kickstarter campaign is due in the coming year to fuel future developments.

“We want to get Samply into the hands of more producers and content creators so that we can establish a feedback loop that guides our priorities,” he says. “Our technology may also have applications in live music performance, instrumentation, and in film and videography. We are excited to explore those possibilities.”