Letting neural networks be weird

Metal Band or My Little Pony?

Neural networks are algorithms that learn by example, rather than by following a programmer’s set of rules. Although on this blog I’ve mostly been using them to generate new examples of things (like paint colors, halloween costumes, or craft beers), neural networks can do a lot more.

One thing neural networks can do is classify things. Give them a bunch of examples of one kind of thing, and a bunch of examples of another kind of thing, and it will (hopefully) learn to tell the two apart. This is really useful - for identifying obstacles for self-driving cars, for telling diseased tissue from healthy tissue, and even (with mixed success) for identifying spam or troll comments. I wanted to test this kind of algorithm out, so I devised the simplest task I could think of: telling metal bands from My Little Ponies.

I’ve previously trained text-generating algorithms to generate metal bands and My Little Ponies, so I had datasets ready to go. IBM Watson has a very easy-to-use tool for training classifiers (there’s a classroom-friendly version at machinelearningforkids.co.uk). I loaded in all 1,300 of the My Little Pony names I had, and filled the rest of the tool’s memory with metal bands (about 18,700).

Then I entered some new pony names - neural network-generated pony names so they weren’t in the original dataset - to see how it would classify them. The result:

The neural network labeled *everything* as metal. People who have worked with neural network classifiers before will have seen this coming: with a dataset that was 94% metal class and only 6% pony class, I had set myself up with a classic case of something called class imbalance. The neural network found it could achieve 94% accuracy on my training dataset by calling everything metal. Princess Pie? Metal band with 81% confidence. Sweetie Loo? 85% likely to be metal. Sparkle Cheer? 84% sure that’s a metal band. Flutter Buns? So, so metal. 97%. The only names it didn’t label as metal bands were ponies that were the original dataset. So, Twilight Sparkle? 100% pony. Twilight Sprinkle, though? 83% metal.

The fix was easy: I trained the classifier again, this time with equal numbers of ponies and metal bands. This time the results were a lot more believable. And, the classifier network mostly agreed with the generator neural network names. There were some surprises, though.

Generated metal bands:

Dragonred of Blood - 100% metal, 0% pony
Deathhouse - 97% metal, 3% pony
Vermit - 97% metal, 3% pony
Sespessstion Sanicilevus - 97% metal, 3% pony
Stormgarden - 97% metal, 3% pony
Vomberdean - 96% metal, 4% pony
Swiil - 96% metal, 4% pony
Dragorhast - 96% metal, 4% pony
Sun Damage Omen - 96% metal, 4% pony
Squeen - 96% metal, 4% pony
Inhuman Sand - 88% metal, 12% pony
Snapersten - 3% metal, 97% pony
Staggabash - 3% metal, 97% pony

Generated ponies:

Cherry Curls - 0% metal, 100% pony
Starly Star - 1% metal, 99% pony
Cheese Breeze - 0% metal, 100% pony
Agar Swirl - 1% metal, 99% pony
Sob Dancer - 1% metal, 99% pony
Derdy Star - 1% metal, 99% pony
Princess Sweat - 1% metal, 99% pony
Raspberry Turd - 1% metal, 99% pony
Arple Robbler - 3% metal, 97% pony
Pocky Mire - 6% metal, 94% pony
Cold Sting - 10% metal, 90% pony
Pearlicket - 48% metal, 52% pony
Blue Cuss - 79% metal, 21% pony
Sunsrot - 84% metal, 16% pony
Rade Slime - 84% metal, 16% pony
Flustershovel Aoetel Pakeecuand - 96% metal, 4% pony

When I fed the classifier names that were generated by a neural network trained on BOTH metal bands and ponies, it was not as confused as I had expected. Instead, it classified them with high confidence as one or the other.

Pinky Doom - 99% metal, 1% pony
Strike Berry - 0% metal, 100% pony
Cryptic Mane - 1% metal, 99% pony
Bloody Star - 4% metal, 96% pony
Killy Power - 96% metal, 4% pony
Spectral Apple - 1% metal, 99% pony

Of course, this classifier will also work on any text I give it.

Benedict Cumberbatch - 96% metal, 4% pony
Jane Austen - 17% metal, 83% pony
Dora the Explorer - 55% metal, 45% pony
Aluminum - 17% metal, 83% pony
Aluminium - 96% metal, 4% pony
The Earth’s Core - 99% metal, 1% pony
Lobsters - 18% metal, 82% pony
Opossums - 97% metal, 3% pony
Yogurt - 17% metal, 83% pony
Kumquats - 96% metal, 4% pony

According to this neural network, we may need to rethink Star Wars canon.
Leia Organa - 96% metal, 4% pony
Luke Skywalker - 31% metal, 69% pony
Darth Vader - 19% metal, 81% pony
Kylo Ren - 18% metal, 82% pony

I had some fun with the metal/pony mashup algorithm and generated more names than will fit here. If you’d like them, enter your email here and I’ll send them to you.

Pony pictures created using General Zoi’s Pony Creator