Can AI Truly Give Us a Glimpse of Lost Masterpieces?

Recent projects used machine learning to resurrect paintings by Klimt and Rembrandt. They raise questions about what computers can understand about art.


In 1945, fire claimed three of Gustav Klimt’s most controversial paintings. Commissioned in 1894 for the University of Vienna, “the Faculty Paintings”—as they became known—were unlike any of the Austrian symbolist’s previous work. As soon as he presented them, critics were in an uproar over their dramatic departure from the aesthetics of the time. Professors at the university rejected them immediately, and Klimt withdrew from the project. Soon thereafter, the works found their way into other collections. During World War II, they were placed in a castle north of Vienna for safekeeping, but the castle burned down, and the paintings presumably went with it. All that remains today are some black-and-white photographs and writings from the time. Yet I am staring right at them.

Well, not the paintings themselves. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to revive Klimt’s lost work. It’s been a laborious process, one that started with those black-and-white photos and then incorporated artificial intelligence and scores of intel about the painter’s art, in an attempt to recreate what those lost paintings might have looked like. The results are what Smola and Wallner are showing me—and even they are taken aback by the captivating technicolor images the AI produced.

Let’s make one thing clear: No one is saying this AI is bringing back Klimt’s original works. “It’s not a process of recreating the actual colors, it is re-colorizing the photographs,” Smola is quick to note. “The medium of photography is already an abstraction from the real works.” What machine learning is doing is providing a glimpse of something that was believed to be lost for decades.

Smola and Wallner find this delightful, but not everyone supports AI filling in these voids. The idea of machine learning recreating lost or destroyed works is, like the Faculty Paintings themselves, controversial. “My chief concern is about the ethical dimension of using machine learning within the context of conservation,” says art conservator Ben Fino-Radin, “because of just the sheer volume of ethical and moral issues that have plagued the machine learning field.”

To be sure, the use of technology to revitalize works of human artistry comes riddled with thorny questions. Even if there was a perfect AI that could figure out what colors or brushstrokes Klimt might have used, no algorithm can generate authorial intent. Debates about this have been raging for centuries. Back in 1936, before Klimt’s paintings were destroyed, essayist Walter Benjamin argued against mechanical replication, even in photographs, saying that “even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be.” This, Benjamin wrote in The Work of Art in the Age of Mechanical Reproduction, is what he called a work’s “aura.” For many art lovers, the notion of a computer reproducing that intangible element is preposterous, if not downright impossible.

And yet, there is still a lot to be learned from what AI can do. The Faculty Paintings were pivotal in Klimt’s development as an artist, a crucial bridge between his more traditional earlier paintings and later, more radical works. But what they looked like in full color has remained shrouded in mystery. That’s the puzzle Smola and Wallner were trying to solve. Their project, organized through Google Arts and Culture, wasn’t about perfect reproductions; it was about providing a glimpse of what’s missing.

To do this, Wallner developed and trained a three-part algorithm. First, the algorithm was fed some hundred thousand images of art from the Google Arts and Culture database. This helped it understand objects, artwork, and composition. Next, it was schooled specifically in Klimt’s paintings. “This creates a bias toward his colors and his motifs during the time period,” Wallner explains. And finally, the AI was fed color clues to specific parts of the paintings. But with no color references to the paintings, where did these clues come from? Even Klimt expert Smola was surprised by how much detail the writings of the time revealed. Because the paintings had been considered so sordid and weird, critics tended to describe them at length, right down to the artist’s color choices, he says. “You can call it an irony of history,” says Simon Rein, the project’s program manager. “The fact that the paintings caused a scandal and were rejected puts us in a better position to restore them because there was so much documentation. And those kinds of data points, if fed into the algorithm, create a more accurate version of how these paintings probably looked at the time.

The key to that accuracy lies in pairing the algorithm with Smola’s expertise. His research revealed that Klimt’s work during this period tends to have strong patterns and consistency. Studying existing paintings from before and after the Faculty Paintings provided clues to the colors and motifs recurring in his work at that time. Even the surprises Smola and Wallner encountered are corroborated with historical evidence. When Klimt first showed his paintings, critics noted his use of a red that was, at the time, rare in the artist’s palette. But The Three Ages of Woman, painted soon after the Faculty Paintings, boldly uses a red, one Smola believes to be the same color that caused an uproar when first seen in the Faculty Paintings. Writings from the time also raise a hue and cry about the shockingly green sky in another Faculty Painting. Pairing these writings with Smola’s knowledge of Klimt’s particular palette of greens, when fed into the algorithm, is what produced one of the first surprising images out of the AI.

“As soon as you see a black-and-white picture, the first thing you do is imagine what it would look like: You assume things about a painting; you see the sky as blue,” Wallner says. As he watched the image generate, a swirling, mysterious green-tinted sky appeared in the rendering on his screen. “That was the shocking part because you see your bias,” he says. “For me, the first moment when I saw these paintings in color was like wow, this is what it looks like!”

Klimt’s isn’t the only work getting an AI resurrection. As part of an ongoing research and conservation program called Operation Night Watch, Robert Erdmann, senior scientist at the Rijksmuseum in Amsterdam, is using machine learning to solve a mystery surrounding Rembrandt van Rijn’s 1642 masterpiece The Night Watch. Currently, the painting is about 15 feet wide and 12 feet tall, but that’s much smaller than the artist’s original. It was trimmed on all four sides in 1715 to fit in a new location (the deepest cut was a whopping two feet, taken from the left side). The cut pieces were never found, but Erdmann hoped machine learning could decode Rembrandt’s original vision for the painting.

When Erdmann began developing his plan, his strongest data point was a 17th-century scaled-down copy by Gerrit Lundens—a painter known for his faithful reproductions of old masters—that included parts of the Rembrandt currently missing. Erdmann’s design used a series of three neural networks. With the first one, he mapped out visually matching points across both paintings. Seen side by side, scaled to the same size, it was apparent that the Lundens is faithful to the Rembrandt. However, as Erdmann toggled between a digital overlay of the two paintings, it was clear how much distortion and stretching there was in the copy. That’s where the second network came in. It warped the Lundens image, stretching it in some places and compressing it in others until most of the spatial distortion disappeared.

With that, the Lundens and the Rembrandt were very closely aligned. But these are still two works created by artists with their own styles. Rectifying that necessitated a third step, the one Erdmann refers to as “sending the neural network to art school.” Through a process called backpropagation, the network learned to render the Lundens in the style of Rembrandt. It created iteration after iteration, getting closer and closer until it plateaued out. Was it a perfect match? No, there’s always a loss, a limit to how close it can get.

Like all new technology, AI and machine learning raise questions around usage and ethics, including when it comes to decades-old artworks. Richard Rinehart, director of the Samek Art Museum at Bucknell University, points out that working with technology has always been about determining our social contracts with it, but AI might be unique in one aspect. “Techno-social contracts have so far been decided unilaterally, but AI may be able to negotiate on its own behalf,” he says. Yet technology has always been at the heart of conservation, across material sciences, chemistry, and color science. “Bringing AI into the mix may signal a potential sea change,” Rinehart adds, “but the concept of applying technology to art is a historically accepted part of the practice, with self-critique as a healthy part of those practices.”

Self-critique within the industry is what art conservator Fino-Radin would like to see more of, but their concerns run deeper. They are excited for the creative avenues this technology opens up but are wary of it being confused with restoration and conservation. “Calling AI ‘restoration,’ calling it anything that implies that it’s like bringing the artwork back to life, is a misnomer, it’s overly simplistic,” Fino-Radin says. “This kind of work belongs in the field of what’s being called Digital Art History.”

Smola and Wallner are aware of the criticisms and take pains to explain the scope and limitations of the Klimt project. “We used the photos as they were to make sure that we don’t deviate too much from the original paintings,” says Wallner. Erdmann notes that the goal behind his reconstruction was to let the public see what Rembrandt’s original composition looked like. “When I translate from the Lundens copy into the style of Rembrandt, the AI doesn’t have the ability to put the life and the genius that is Rembrandt back into the painting,” he emphasizes. “I’m not trying to do that. I don’t want to do it.” What you see at the Rijksmuseum today is the cropped painting, all that remains of the original Rembrandt. The printouts of the extended composition were on display only temporarily, from June to October 2021, and were mounted in front of the painting, not flush with it, so there was no mistaking them for the original.

Rinehart sees both projects as valuable case studies in how AI can be used effectively in the art world. Instead of shying away from what this technology holds for the future, he hopes for more involvement from everyone—curators, conservators, museums, and the public. “What is important is to invite the public to follow museums along that continuum so that we use these instances to learn to see more clearly the shades of nuance and utility among ‘real’ and ‘simulacrum,’” he says.

When technology produces plausible answers to age-old mysteries, does it diminish the aura of the art or artist? Ask the team at Google Arts and Culture and their answer is a straightforward and pragmatic “no.” If anything, they believe their work highlights the Faculty Paintings and heightens the mystery around Klimt, a revolutionary painter known to most only through works from his less rebellious Golden Phase. With Erdmann’s AI reconstruction, people can see Rembrandt’s original, dynamic vision for The Night Watch. Surely, this ability to visualize what is lost is a net gain.

Perhaps it all comes back to aura. AI can fill in a lot of art history’s blanks, but it can’t recreate masterworks. Nothing can. “Aura does not offer a binary choice of ‘true authentic original’ versus ‘fake artificiality,’” Rinehart says. It’s possible to enjoy being right in front of a painting or looking at it on a computer screen, but they’re different, layered experiences. What matters is what we feel when we see them.