Jukebox of weirdness

Ever wish you could hear what it would have sounded like if jazz legend Ella Fitzgerald had done a cover of Baby Shark?

So classy, right? How did this happen? A couple of steps:

1. OpenAI trained a new neural net called Jukebox on over 1.2 million songs.
2. Unlike previous attempts at AI-generated music, Jukebox has the ability to sing lyrics.
3. I asked them very very nicely if they would please give some robo-artists the lyrics to Baby Shark.
4. They were very very nice to me.

Here’s The Beatles covering Baby Shark

This is seriously cool work. Till now, AI music generation has mostly focused on generating individual notes, kind of like a midi file. That’s because there’s so much going on in your average song - the timbre of the instruments, the squeak of strings, the sound of breath, notes bending, the room echo - that an audio file is much much more dense with information than simple music notation. The more a neural net has to keep track of, the harder its job is.

To do full audio files, OpenAI used some interesting tricks - like training a neural net on a super-compressed version of the audio file, so it could keep track of the overall song structure without worrying about getting the details right. Then other neural nets were trained to turn this super-compressed audio into something more realistic. Jukebox is multiple neural nets working together, all very specialized for music generation.

Here’s another cover of Baby Shark, this time by singer-songwriter Jack Johnson (for him, the ocean theme is fitting). Interestingly, toward the end, he seems to segue into seagull sounds, and then some kind of… bubbling? Gibbering? How lucky we are - none of this would have been possible with an AI that was restricted to music notation.

Now, the AI versions aren’t the greatest in the world - given the lyrics to “Yesterday” and “The Beatles” as the target style, it generates an utterly bland, meandering tune (it still has trouble with long-term coherence and will sometimes get stuck on endless instrumentals or repeated lyrics, not realizing how long it’s been in that particular groove). At least (in typical show-offy Jukebox style) it ends by applauding itself.

So is Jukebox the future of music? Well, again, its songs are pretty terrible. And possibly unusable for commercial things, depending on the legal implications of training on and parodying various artists. It also takes 9 hours to generate 1 minute of audio, and there’s no guarantee it will sing what you ask. Especially if you’ve given it weird lyrics for the style you’re asking for, it might just decide to do an instrumental instead. But it hints at the kinds of creative tools that COULD be built, as AIs get more efficient and as musicians get involved with building their own tools.

And for now, we have a rare opportunity to find out what it would be like if Frank Sinatra sang that one really intense AI-generated Christmas carol about Rudolph and human flesh for sacrifice yada yada:

Rudolph the Red Nosed Reindeer, with its red belly
The All-gracious King of all the earth,
Had a baby at Christmas time,
On Christmas Day,
A true and holy Deity,
Went down to earth,
With human flesh for sacrifice.

For sinful men such a Deity doth appear,
And wink and nod in reply,
As he winked and nod in reply,
As he winked and nod in reply

The wretched world is run by ox and ass
The wretched world is run by ox and ass,
And in vain build I.

Frank Sinatra, folks

There is also an Elvis version.

Shoutout in particular to Christine McLeavey Payne of OpenAI, who very kindly generated the above samples for me.

OpenAI published a bunch of examples, searchable by artist and genre.
You can generate your own too! OpenAI released a free colab notebook where you can specify your own lyrics, genre, and artist. (It takes 9 hours to do the full upscaling, but the basic-level track is still very listenable, and the mid-quality track emerges after a couple of hours and sounds fine to me).

I used the colab to see what a heavy metal version of Baby Shark would sound like. It goes… unexpected places. I was not disappointed.

AI Weirdness supporters get bonus material: I got the neural net GPT-2 to suggest some alternate lyrics to Baby Shark. Or become a free subscriber to get new AI Weirdness posts in your inbox!

Subscribe now

My book on AI is out, and, you can now get it any of these several ways! Amazon - Barnes & Noble - Indiebound - Tattered Cover - Powell’s - Boulder Bookstore