How to haunt and unhaunt a house

What do you get if you instruct an AI to turn a house into the most haunted house in the world? What if you ask it for the LEAST haunted house? How does an AI know what "haunted" looks like, anyways?

I did some experiments with CLIP+VQGAN (link and tutorial here). This makes use of CLIP, which learned what kinds of images and text are associated with one another on the internet. Since CLIP has probably seen lots of things labeled "haunted", it can probably guide VQGAN into making some seriously spooky alterations to this lovely 1894 Victorian house:

For my first try, I had it start with this image and alter it to look more like "A haunted Victorian house".

The wood has turned to stone, the yard has proliferated more iron fences (and become greener), and one of the turrets has changed to a chimney pot. Other than that, it looks pretty ordinary.

I decided to kick it up a notch, and told the model to head toward "An extremely haunted Victorian house | dramatic atmospheric night photography."

The walls became wobblier, another of the turrets changed to a chimney, and curiously, there now seems to be a smaller more modern house inside the larger house.

It turns out a lot of this is the "dramatic atmospheric night photography".

But the interface I'm using lets me assign negative weights to things, so I can actually subtract "haunted" from the house I'm trying to make. It sort of works! Once I subtract enough "haunted."

Apparently CLIP has learned that if you want to make things less haunted, add flowers, street lights, and display counters full of snacks.

This, in case you're interested, is the least haunted place in the world. It's what you get when you give CLIP+VQGAN no instructions except for a negative penalty to "haunted".

I'm not sure what most of this stuff is supposed to be, but if you can find it and bring it to the middle of a baseball diamond in an office park, you will be safe.

I also tried haunting a Waffle House.

Bonus content: a haunted bookstore, a haunted bakery, and a haunted grocery store.