This appeals to my sense of the absurd. Instead of taking an AI and training it to turn photographs into converted images “in the style of” some training set, what if we plug the pipeline in backwards?

A team of researchers at UC Berkeley have revealed an “Unpaired Image-to-Image Translation” technique that can do something really interesting: it can turn a painting by Monet into a ‘photograph’… also, it can transform horses into zebras and summer into winter.

It sounds a lot like Adobe’s “smart layers” except the layers are regions identified by the AI, plus some basic AI repaint transformations. [petapixel]

It is billed as “turning an impressionist painting into a photograph” but that’s not really possible – a photograph would show information that never existed in the painting. The AI may add new information, in which I would call it a “rendering” not a photograph. One might call it an attempt to produce a “photo realistic” image.

The AI doesn’t do a very good job at the horse->zebra->horse conversions. Look at the ears. And, look at the right buttock of the right zebra the “horse” conversion: the stripes are still there.

I don’t want to fault the AI, it’s pretty cool, but it’s not a good enough artist to fool a human. It’s impressive progress, though. I assume the AIs will only get better, and their training sets will only get deeper – their knowledge will get better with practice. The zebra->horse conversion would fool someone who doesn’t know horses and as the AI gets more knowledgeable it’ll inevitably learn about the ears.

The winter->summer conversion didn’t get all the snow off the rocks, either. And the summer->winter conversion was cleverly selected to be  scene with evergreen trees and no trees that drop their leaves.

In his book Implied Spaces [amazn] Walter Jon Williams introduces us to a person who is existing in a procedurally-generated artificial world. He uses the term “implied spaces” to describe the fractally-detailed regions of the map that are ’empty’ until you look at them – at which point the game fills them with an appropriate level of detail.

Let’s look at an implied space:

Star Citizen does the same sort of thing that Elite: Dangerous does – it rolls dice and looks on a procedural table, so that the closer you look the more you see – and everything is consistent with the rest of the picture because each thing inherits some of its parameters from the things around it. You can see where this is going: AIs are really good at that. And, once they infer the basic rules for the procedural generation, they can churn out all the detail you can possibly consume. The remaining question is how convincing it is – and that is going to depend on how knowledgeable the AI is. Star Citizen‘s procedural generation works at a planetary scale, with cities that are fractally detailed down to the level of individual rooms in individual buildings with (supposedly) reasonable-seeming floor plans. Elite: Dangerous treats star systems as implied spaces – except for systems that are configured to match known systems, it procedurally generates downward from the main star type. [see geeky note below]

Adobe constantly experiments with new imaging manipulations; a good new transform is worth a lot of money (e.g.: that popped-out HDR stuff everyone was using for a couple years) Adobe has developed what they “Scene Stitch” [petapixel] which is a content-aware fill (call it an AI that trains itself quickly regarding the contents of an image) These images are from Adobe:

Let’s say you have a desert scene that is marred by the presence of a road. Normally in photoshop you’d start blotting over it with the clone brush and it might look like this:

There are repeats, because we had to duplicate the information – we couldn’t create it. Note that the demo forgot the other piece of road (lots of experience with doing demos makes me wonder if there’s a reason that it wouldn’t handle the other road correctly!) – but with the new Adobe content sensitive stitching, it apparently generates implied spaces.

Immediately I wish I had a copy of that to play with. Why? I’d take one of my photographs and enlarge it hugely. Then, I’d let the stitching routine generate implied spaces in the enlarged image. There are some interesting and warped possibilities.

Here, an image is shown with the region to be filled highlighted (Adobe’s image) – presumably there is a source for the additional information.

That’s not out of this world, difficulty-wise, for just plain photoshop pixel-pushing. But, if you can do that with a couple clicks, then you could automate sampling in implied spaces over a much larger image – of you could give it a weird source and see what it does. I would like to see what would happen to that skyline picture if my sample dataset was a plate of spaghetti and meatballs. Or if the skyline was stitched with city-scapes from Star Citizen.

One last point about all this: these features are powered by Adobe’s cloud service, Adobe Sensei. Like with Prisma and Deep Dreams you shove your data up into the cloud, it gets hammered and pixel-pounded, then you get it back. There are going to be interesting security problems with that. I hope Adobe thinks this through (they have a horrible history of security mistakes) because I can think of a couple ways that this could go wrong.

Geeky note on procedural generation: A procedural generation system has great big tables of conditionals and probabilities following those conditions just like a Markov Chain. So, when you enter an implied space – let’s say it’s a stellar system in Elite: Dangerous – you roll some dice and see what the typical distribution of planets is in that type of star system. Then for each planet, you roll dice for probability distribution of moons for that type of planets in that type of star system. Then you roll dice for the orbits, etc. You get down to a broad voxel-space for each moon, and then within the voxels, you subdivide and subdivide as necessary. Zoom in far enough, you see individual rocks.

Of course there is one problem with that: if you roll dice you’d need to remember the die rolls, or the universe will look different for each player – remembering the die rolls for every detail of 400 billion stellar systems is going to take a lot of storage. So, instead, what you do is pick a magic string as a seed, and plug that into a cryptographic function. Let’s say we’re using the Data Encryption Standard (DES). So, we take 1K of zeroes and encrypt them with our string.

int   dicerolls[1000 * sizeof(int)]; // 1000 dice rolls
des_keyschedule key; // cipher machine state
bzero(star_system_id,dicerolls); // pre-fill the dicerolls with the star system number
des_setkey(key, “the secret to life is low expectations”);
des_cbc_encrypt(key,dicerolls,); // cipher block chaining mode (cbc) folds the data from one block to the next

Now, you have 1000 dicerolls that are, for all intents and purposes, random-looking – but everyone who goes to that system will have the same dice rolls. As long as the dice rolls are specific to the same decision-points on the table, you’ll get the same answer. So if I fly to SKAUDE AA-AA-1H14 planet B’s first moon, and go to a particular region on the map, and look closely, I’ll see the same dice-rolled rocks in the same positions as you would.

My DES function calls are from 20-year-old memory, don’t try to write a crypto application using them, they’re just an illustration. The point is, once you bootstrap your dice rolls, then all you need to keep is what you rolled on which ‘table’ when you rolled it. Next time you go to the same system, it’ll generate the same ‘rolls’ and the system appears persistent but it’s not.

I first encountered procedural generation playing rogue on a PDP-11 running BRL V7 UNIX in 1981. rogue was probably not the first game to use it – it’s a very old technique; in the case of rogue it generates a new dungeon for each player every time they start a game. But, if you used the same random number seed for every player (let’s say the game used the day of the year as the seed) everyone who played that day would get the same dungeon.