Can AI Truly Give Us a Glimpse of Lost Masterpieces?

In 1945, fire claimed three of Gustav Klimt’s most controversial paintings. Commissioned in 1894 for the University of Vienna, “the Faculty Paintings”—as they became known—were unlike any of the Austrian symbolist’s previous work. As soon as he presented them, critics were in an uproar over their dramatic departure from the aesthetics of the time. Professors at the university rejected them immediately, and Klimt withdrew from the project. Soon thereafter, the works found their way into other collections. During World War II, they were placed in a castle north of Vienna for safekeeping, but the castle burned down, and the paintings presumably went with it. All that remains today are some black-and-white photographs and writings from the time. Yet I am staring right at them.

Well, not the paintings themselves. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to revive Klimt’s lost work. It’s been a laborious process, one that started with those black-and-white photos and then incorporated artificial intelligence and scores of intel about the painter’s art, in an attempt to recreate what those lost paintings might have looked like. The results are what Smola and Wallner are showing me—and even they are taken aback by the captivating technicolor images the AI produced.

Let’s make one thing clear: No one is saying this AI is bringing back Klimt’s original works. “It’s not a process of recreating the actual colors, it is re-colorizing the photographs,” Smola is quick to note. “The medium of photography is already an abstraction from the real works.” What machine learning is doing is providing a glimpse of something that was believed to be lost for decades.

Smola and Wallner find this delightful, but not everyone supports AI filling in these voids. The idea of machine learning recreating lost or destroyed works is, like the Faculty Paintings themselves, controversial. “My chief concern is about the ethical dimension of using machine learning within the context of conservation,” says art conservator Ben Fino-Radin, “because of just the sheer volume of ethical and moral issues that have plagued the machine learning field.”

To be sure, the use of technology to revitalize works of human artistry comes riddled with thorny questions. Even if there was a perfect AI that could figure out what colors or brushstrokes Klimt might have used, no algorithm can generate authorial intent. Debates about this have been raging for centuries. Back in 1936, before Klimt’s paintings were destroyed, essayist Walter Benjamin argued against mechanical replication, even in photographs, saying that “even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be.” This, Benjamin wrote in The Work of Art in the Age of Mechanical Reproduction, is what he called a work’s “aura.” For many art lovers, the notion of a computer reproducing that intangible element is preposterous, if not downright impossible.

And yet, there is still a lot to be learned from what AI can do. The Faculty Paintings were pivotal in Klimt’s development as an artist, a crucial bridge between his more traditional earlier paintings and later, more radical works. But what they looked like in full color has remained shrouded in mystery. That’s the puzzle Smola and Wallner were trying to solve. Their project, organized through Google Arts and Culture, wasn’t about perfect reproductions; it was about providing a glimpse of what’s missing.

To do this, Wallner developed and trained a three-part algorithm. First, the algorithm was fed some hundred thousand images of art from the Google Arts and Culture database. This helped it understand objects, artwork, and composition. Next, it was schooled specifically in Klimt’s paintings. “This creates a bias toward his colors and his motifs during the time period,” Wallner explains. And finally, the AI was fed color clues to specific parts of the paintings. But with no color references to the paintings, where did these clues come from? Even Klimt expert Smola was surprised by how much detail the writings of the time revealed. Because the paintings had been considered so sordid and weird, critics tended to describe them at length, right down to the artist’s color choices, he says. “You can call it an irony of history,” says Simon Rein, the project’s program manager. “The fact that the paintings caused a scandal and were rejected puts us in a better position to restore them because there was so much documentation. And those kinds of data points, if fed into the algorithm, create a more accurate version of how these paintings probably looked at the time.”

The key to that accuracy lies in pairing the algorithm with Smola’s expertise. His research revealed that Klimt’s work during this period tends to have strong patterns and consistency. Studying existing paintings from before and after the Faculty Paintings provided clues to the colors and motifs recurring in his work at that time. Even the surprises Smola and Wallner encountered are corroborated with historical evidence. When Klimt first showed his paintings, critics noted his use of a red that was, at the time, rare in the artist’s palette. But The Three Ages of Woman, painted soon after the Faculty Paintings, boldly uses a red, one Smola believes to be the same color that caused an uproar when first seen in the Faculty Paintings. Writings from the time also raise a hue and cry about the shockingly green sky in another Faculty Painting. Pairing these writings with Smola’s knowledge of Klimt’s particular palette of greens, when fed into the algorithm, is what produced one of the first surprising images out of the AI.

“As soon as you see a black-and-white picture, the first thing you do is imagine what it would look like: You assume things about a painting; you see the sky as blue,” Wallner says. As he watched the image generate, a swirling, mysterious green-tinted sky appeared in the rendering on his screen. “That was the shocking part because you see your bias,” he says. “For me, the first moment when I saw these paintings in color was like wow, this is what it looks like!”

Klimt’s isn’t the only work getting an AI resurrection. As part of an ongoing research and conservation program called Operation Night Watch, Robert Erdmann, senior scientist at the Rijksmuseum in Amsterdam, is using machine learning to solve a mystery surrounding Rembrandt van Rijn’s 1642 masterpiece The Night Watch. Currently, the painting is about 15 feet wide and 12 feet tall, but that’s much smaller than the artist’s original. It was trimmed on all four sides in 1715 to fit in a new location (the deepest cut was a whopping two feet, taken from the left side). The cut pieces were never found, but Erdmann hoped machine learning could decode Rembrandt’s original vision for the painting.

When Erdmann began developing his plan, his strongest data point was a 17th-century scaled-down copy by Gerrit Lundens—a painter known for his faithful reproductions of old masters—that included parts of the Rembrandt currently missing. Erdmann’s design used a series of three neural networks. With the first one, he mapped out visually matching points across both paintings. Seen side by side, scaled to the same size, it was apparent that the Lundens is faithful to the Rembrandt. However, as Erdmann toggled between a digital overlay of the two paintings, it was clear how much distortion and stretching there was in the copy. That’s where the second network came in. It warped the Lundens image, stretching it in some places and compressing it in others until most of the spatial distortion disappeared.