Data driven photo editor turns rainy pictures into sunny days

Researchers input 8500 images from over 100 outdoor webcams into a massive database, and used crowdsourcing and machine learning to teach the tool to recognize more than 40 attributes of outdoor scenes. Is it sunny? Raining? Dry? Gloomy?

An algorithm identifies the differences between the same scene when viewed under various conditions by separating out the elements, such as the sky, buildings, and the ground. Each element is affected differently by an attribute change.

In real life, snow is more than a simple white layer. It might make the ground white, a building darker, and the sky greyer. So the program must apply multiple changes to the user's image on a pixel level to make it look realistic. In effect it can automatically change a photo to a new weather scheme.

There are limitations, of course (as discussed in the video), but this data driven approach sounds interesting.

 

Nice. I like the "crowd-sourcing" aspect of this in particular.

 

While it sounds limited to color and value changes, it is very impressive. Sounds like there may be a way to apply texture and shape data in the future. One of the biggest challenges might be to transform a Summer photo with green trees, into a Winter scene with leafless trees. If you attempt to remove the leaves, what image is added behind each leaf, and how does that affect the light below the trees on the ground? Another would be to add the reflective light, from snow on the ground, up into other objects (although maybe he mentioned something about that?). Very cool stuff.

 

I don't think they can do that - the information simply isn't there, ie the branching structure. But, perhaps they could identify the type of tree and match it with instances of the same tree from a database, and than apply a fractal branching algorithm to 'grow' the branches in accordance to the constraints of the tree in the input picture.

That would be a lot more of a synthetic approach, but I don't know that they'd be willing to go to those lengths. Still, if they did, then they could easily account for the lighting under the tree as the branching pattern would exist as a 3D model.

 

Or Kodak could simply have kept Kodachrome in production...

 

We can finally have the vacation we really wanted in pictures.

 

The irony of all of this is that quite often, photos taken in very cloudy or even rainy conditions can be exceptional----balanced lighting quality, lack of shadows and glare, and none of the flattening effect of mid-day sunlight on static subjects.

Fall color landscape shots, IMO, look better on cloudy days, because the color pops out, whereas in bright sunlight, the oranges and reds are dulled in comparison.