Unexpected Treasures

Training neural networks is hard.

Yeah, I'm lost

The setup

This is the attention network - you feed it an image, and it spits back out an "attention mask" - an image in greyscale where every pixel tells you to how much attention you should pay to it. So even if I love you lots, it would still be a number between 0 and 1 :))

The view in Tensorboard

The attention network (inside the box there) is just a convolutional stack. So it should be able to output anything we want. In this case, I want to see if it actually does that... by making it do some simple task.

The simple task is this: reconstruct an image, channel by channel. The first pass, it should focus on whatever is red. The second pass, it should focus on whatever is green. The third pass, it should focus on whatever is blue. At the end, we take these three outputs, stack them together, and call it our "inferred image".

Slight technicality: The masks, in real life, should sum to 1 (ie, account for everything in the image). However, if we want to do this image reconstruction task, that wouldn't be so good. Why?

Imagine if you had an image which is all black. The pixel values would be 0, so there's no way that the masks (which sum to 1) would be able to make 0. Is that why the loss is fluctuating?

One solution would be to add a fourth mask, which doesn't contribute to the image, in order to capture all the extra attention. If the image is black, this fourth mask would be 1, leaving the other three free to be 0. Thankfully, I didn't think of this before.

By trying to get this to work, I accidentally made cool picture! It's weird and surprising how simple things behave sometimes.

What we really did

A collection of the (really really interesting) mishaps that happened during training, good enough to turn into a standalone.... Presenting:

We Who Trained Monsters (2019)

Tribute to human beings, past, present, and mostly future - To all the unborn, who shall live in total kindness and know not of to-day. Can we be wiser?

Brazen Planets (2019).

(Oteration 2361 of the First Run)

This one looks a lot like Earth and three satellites going off. Are they spaceships searching for a new home? Is Earth a desolate place in the future?

Split 1 (4131).

Wow, bacteria. The planet and the satellite are cloning themselves.

Split 2 (5931).

At some point, the neural network decides to put red at the bottom of everything. Not just any red, too, but a very nice red. It's starting to learn about green as well. It stays like this for many, many iterations.

Star and Sky I (2019).

The pyramids of Stargate were built by ancient aliens. This picture also conveys a sense of the ancient - it's the desert brown, the sunset red, the triangles turned into pyramids by the (im)perfections of the neural network.

Star and Sky II (2019).

I bet no one has ever imagined the triangles as red. Sun-stones! A glowing orb sits in the center. We are unsure. Works of the fae are oft confused with advanced technology.

Reign of Fire, Trial 2 (2019)

Is it a metal album cover?

Vortex Man (2019)

We would have called St. Paul the Apostle but This emerges out of the mind of the machine.

So yeahhh this morning has been fun