From the Louvre in Paris to Casa Battló in Barcelona, the museum-going experience is getting a technology makeover. Instead of just physical maps and exhibit labels, visitors to museums and galleries can explore exhibitions with the help of a smorgasbord of handheld apps, iPad games and VR/AR features.
But the Smithsonian American Art Museum (SAAM) is exploring a virtual reality (VR) experience that sets a new high bar in technological sophistication and artistry. The project—for now an internal pilot—would allow people anywhere to virtually visit the second-floor east wing of the Washington, D.C. museum. The VR experience would allow them to interact in unique and immersive ways with three exhibits—a painting, a sculpture and a three-panel video installation.
"We're bringing the museum into your living room," says V.A.L.I.S. studios CEO Peter Martin, who produced and directed the project in collaboration with the museum, Intel, Framestore VR, 8i and xRez Studio. "I wanted to fuse together and expand the idea of art in the VR space. It's also a way of future-proofing the museum."
The project pulls together three technologies not previously used in this way—Lidar scanning, which scans things like rooms to produce 3D models; photogrammetry, which uses photography for surveying and mapping; and volumetric capture, which trains several cameras on a figure and stitches the feeds together to create a three-dimensional model that can be placed in any environment. Intel wanted to combine the technologies in a proof-of-concept VR experience.
"We wanted to create a wow feeling—to demonstrate the beauty of VR and what it could do," says Intel's Raj Puran, a business development manager focused on VR/AR and mixed reality (MR) who was the project's executive producer. "It was really about how technology, the museum and creative people can come together to elevate the VR experience and help people understand what immersive VR looks like."
Teleporting into Art
Whether they're in Los Angeles or Timbuktu, users wearing a VR headset will be able to visit the section of the museum that has been meticulously recreated in VR. In addition to being able to view "inactive" pieces up close—such as pictures hanging on walls—users will be able to "teleport" into the three interactive exhibits.
Seen through a headset, the interactive exhibits are marked by a spinning vortex on the floor. As viewers approach, they can read about the piece on an AR holographic display that appears floating in space—no need to stoop to read a printed label. Then, as they step forward and move through the graphic, they are teleported into a VR experience.
For example, on approaching the 19th century oil painting Aurora Borealis by Frederic Edwin Church, viewers are immersed in a 360-degree video at 6K resolution of the aurora borealis in Iceland, courtesy of designer and photographer Olafur Haraldsson.
"It transports people," Martin says. "You can see it on their faces. They're a little bit shocked."
Another interactive exhibit is the Adams Memorial, a famous bronze sculpture by Augustus Saint-Gaudens. The AR text overlay tells the story of photographer "Clover" Adams, who committed suicide in 1885 by drinking chemicals used to develop photographs. Her grieving husband, writer Henry Adams, commissioned the sculptor to create a memorial—the original of which sits in a D.C. cemetery, a 20-minute drive from the museum.
At first glance, the sculpture gives no hint of this evocative back-story. But when viewers step closer, they are teleported into a digital recreation of the memorial in the cemetery, complete with birds and trees surrounding them.
"A lot of artwork lose their energy when taken out of their location," Martin notes. "When you first look at the statue, you don't get the feeling attached to the piece. But the minute you hit that graveyard, you get that emotional impact."
The third exhibit is a three-screened video installation titled Face in the Crowd by artist Alex Prager. Martin says the VR experience of the video is "like being inside a movie," but what really sets it apart from the physical experience is that viewers can turn to Prager herself—actually a volumetric 3D version of her—for a director's commentary on the piece. Captured at 6K resolution, the display is gaze activated, so Prager speaks only when viewers look at her and stops speaking when they look away.
Each of the VR exhibits allows users to explore space, interact with it and have a degree of agency, versus the watered down VR experience of a 360-degree video. Each is also one virtual asset that can be used on any platform—from an IMAX screen to a headset with PC to a mobile device—without being re-versioned.
"The key is to create content that is more flexible in terms of its distribution," Martin says.
Challenges and Potential
The technological challenges of the Intel-funded project were matched only by the difficulties of meeting stringent deadlines. SAAM greenlit the project just before Thanksgiving 2016, which left just five weeks to meet Intel's goal of creating a demo for CES 2017. Martin turned to 8i for the volumetric capture, xRez Studio for the photogrammetry and Framestore VR for the Unity build (a game engine used by VR/AR developers) and the main interaction as well as some CG assets and help with navigation and sound design.
A chief benefit of such VR experiences would be increased access to museums for the public.
"What about those folks—students, art lovers, creative and intellectually curious people—who never get the chance to buy that once-in-a-lifetime plane ticket to Washington, D.C.? What of those who live in far-flung places across North America or the globe?" says Sara Snyder, head of media and technology for SAAM and the Renwick Gallery who collaborated with the team on the pilot. "Virtual reality will offer a chance for anybody to experience SAAM and the Renwick Gallery as a full experience, wherever they are."
Puran says museums could also house VR pop-ups that would allow visitors to put on headsets and interact with exhibits virtually—part of the location-based VR trend that's gaining traction in, for example, China. And then there's the fact that many museums have vast quantities of art the public never sees because it's stored away in archives due to limited gallery space. Puran sees an opportunity for museums to digitize their archived art and make it available to all in virtual galleries.
"VR as a tool is going to unlock a lot of possibilities," Puran says. "We're just scratching the surface."