Disney songs, sung by a neural network with a very short memory
I train neural networks, which are a type of computer program that try to learn to copy human things by looking at examples. By giving them just a list of names and no further instructions, I can get neural networks to invent names for paint colors, guinea pigs, craft beers, and more.
The way I set up a neural network will control how well it does at its task. One important parameter I can control is the length of its memory. Let’s see what happens when I make its memory really, really short.
Dory Image spiced up by Neural net-generated deep dream
Here’s why memory is important for a neural network: As it’s generating the letters of new words, it’s looking at the letters it has already generated, trying to figure out what comes next. If it can look back over several words or even sentences, it will be more memory-intensive to run than a neural network that can only see a couple of words at a time, but it will also be better at learning long phrases.
Here’s how this plays out when I train a neural network to write lyrics to Disney songs.
I started with a list of 224 Disney songs from a lyrics dataset posted by GyanendraMishra on Kaggle, about 8200 lines in all.
50 character memory
At first, the neural network had a modest memory length of 50 characters - that is, 50 letters. Its view of the dataset, and of the text it generates, looks something like the this:
ow you the world
Shining, shimmering, splendid
Tel
Humans who know the “I can show you the world” song from Aladdin can probably recognize this bit and fill in what comes next. The neural network can work with this too. It memorizes whole lines of songs, although it sometimes jumps between different songs - I’ve built a small degree of randomness into its choice of which letters to use next so it doesn’t always go with its top choice.
A whole new world (A whole new world)
A new fantastic point of view
No one to tell us no
Or where we will live
Oh so surprise
Every time to spare
Let it snow, let it snow, let it snow, let it snow, let it snow, let it snow
But I couldn’t do that it’s all around the sun
With all the colors of the wind
Can you paint with all the colors of the wind
Can you paint with all the colors of the wind
Can you paint with all the colors of the wind
Can you paint with all the colors of the wind
Can you paint with all the colors of the wi
You’ll notice that it tends to get stuck on certain repetitive lines - that’s because those lines repeat in the original dataset, so the neural network learns that one “paint with all the colors of the wind” line is followed by a second. What it doesn’t know, however, is how many “colors of the wind” lines it’s already seen at a given point. Its view of these lines at any given time is something like:
paint with all the colors of the wind
Can you pain
And it has no idea whether to move on to the next line, or to keep singing the same one. Usually, like Dory, it just keeps singing.
20 character memory
With less than half the memory that it had before, the neural net is struggling a little harder. Its view of the training data looks something like:
ow you the world
Shi
It still can memorize phrases (it can spell supercalifragilisticexpialidocious like a champ), but now gets lost and jumps between songs more easily. It tends to string together songs with some similarity, so Heigh-Ho (we dig dig dig dig dig dig dig in our mine the whole day through) gets mixed up with drip drip drop little April shower, and Mama Odie’s song from Princess and the Frog (you gotta dig). And once it starts singing Heigh-ho, Heigh-ho, it basically doesn’t stop.
A new fantastic point of view
No one to tell us no
Or where to go
Or say in the wood
Is the savage one is mine!
We dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig up with your pretty little song
Greatest are seppation for you
You paint with your pretty little song
Gay little song right
The savage one is way to sing with your pretty little song
It’s gonna be a change
Shake you are the sky is cloudy
You gotta dig
Choir: (dit)
Mama Odie: Dig and dig and diggety-dig
Heigh-ho, heigh-ho
Heigh-ho, Heigh-ho
Heigh-ho hum
Heigh-ho, heigh-ho
Heigh-ho, heigh-ho…
10 characters
A memory this short is seriously unfair to give a neural network that’s trying to learn whole phrases. That line from Aladdin now looks something like:
ow you the
But it does its best.
[Troy]
If you say that must be true
When you leave the world and the same place in the sky
You can fly
You can fly
You can fly
You can fly
You can fly
You can fly
When you’re staring straight back at me?
Why a true!!
We’re breaking free.
[Troy]
Oh the most wonderful time of the year
Winnie the Pooh
Willy nilly silly old bear
Winnie the Pooh, Winnie the Pooh
Willy nilly wearing,
when you’re welcome, she’s going the bare necessities of life will come to you
I’ve been searching to find the bare necessities of the wind
I’ve seement her dont and bring you can fly
You can fly
You can fly
You can fly
You can fly
You can fly
You can fly
You can fly
5 characters
Evil laughter. It’s honestly a surprise that a neural network with a 5-character memory can manage to function at all.
ow yo
That’s what the line from Aladdin looks like now. This what the neural network has to work with. This is what songs are to it now.
It attempts “Under the Sea” or possibly “Part of Your World” from The Little Mermaid.
Wish the sky
What’s the street too
I wonder the sea
Under the sea
Under the sea
Under the sea she wants to dreamed to me
Un the shore here where the world in by me about the river bend
I wanna be so far
I could be to place
I’ll see it there to start
And the shore shore
Underer the sea
When the sea
Under the sea
Under the sea
Under the sea
It is going to try to be king. Or a wolf.
I’ve been in the sun
It’s longing with a s
This, dilly, dilly sofver who do it once a smiling
A whole new world to the told me so
If think with you
And I wonderful from the wolf cry to the blue corn mountain
This where who one souls
So be king
So good in a staring as the beginning with your cat
I’ll never thing thing the start
A whole new world with new the wolf cry to the tole on the otter that the same
That down tomb told me so
I’ll be king, dilly, dilly dain ine told me soy
And I’ll never paint with new world with all the colors of the cores the sun
I’ll be king
Oh, so chue the wolf cry to the colors of the beginning with your to
I wanna be king is the wolf cry to the storm to tell your corn who want the roll me soy
I’ll never seen the blue here, and the chain as old as the sun start
And I’m fine
I’ll try
I’ll try
It has a seriously metal-sounding version of Beauty and the Beast.
Something to the beast
And and the beast
The door
When the beast
The door
When will bare necessities
The beast
When the most wonderful time to spart of the house your learn and black as the sun
And the beast
Tale as old as time
Song as old as rhyme
Beauty and be soul
It’s the sun
We’re only friends and the door
When will sell you see her fleight and and bring about your heart don’t say a way the most wonderful time of the time of the shock
The beast
And a secret, winnie the Pooh
Willy all the same
I saw
There’ll be you feel the Hokey
the beast
And then it rears up, gathers all its neurons, and flounces out an impressive:
Birdriillagilisticexpialidocious
Become an AI Weirdness supporter to get bonus content! Or become a free subscriber to get new posts in your inbox.