Cortina Productions (McLean, Va.) is an oft-awarded, full-service creative media design and production company that specializes in museum exhibitions.
We recently caught up with Jim Cortina, principal and development manager at Cortina Productions, to talk about the firm’s experimentation with mixed reality and museums during SXSW 2017 and what that might mean for their future.
Let’s start at the beginning.
JC: When we first learned about the Microsoft HoloLens, we immediately realized that there was a potential for using the technology in museums. One of the things we were drawn to is that you could still see and safely navigate your surroundings while using the device. Even though it’s first-gen, we bought a couple of them for the office.
Then we got the idea to test it and we had the perfect opportunity to do so: We were going to be at SXSW in Austin. Cortina Productions had completed film and multimedia projects for the LBJ Presidential Library as well as produced a film for the Bullock Texas State History Museum. Knowing that we had already produced a 3-D CGI model of the ship “La Belle,” my brother recognized the unique chance to take this 3D model and marry it with the actual hull of the ship in the museum space, which we felt would be a great test for the technology.
On the other hand, the LBJ Presidential Library has wonderful architectural features like the central archive that creates a large wall. Visitors can explore the archives through kiosks but don’t have direct access to documents. So we thought, wouldn’t it be cool to actually use the space? We contacted both of the museums, whose staff thought it was a wonderful idea and so we set it up.
Our interest wasn’t whether we would be able to activate small artifacts, but rather how we could use the technology to activate the larger spaces within museums, and how visitors would react to the technology. The reaction was over-the-top positive; people loved it.
Why Augmented Reality and not Virtual Reality?
JC: As a media production company that specializes in custom location-based experiences, we find this technology very exciting because we are challenged to think in new ways about the experiences and the spaces we are designing for.
For us, the greatest benefit to utilizing the HoloLens technology was the opportunity for wearers to engage with the physical museum space and its archival collection while escaping the confinements of a traditional VR headset’s virtual world. It significantly reduces sensory issues, such as motion sickness, that can occur when a user is placed in a completely virtual environment.
HoloLens wearers are able to interact both with the new digital content as well as the friends, family and structures around them in the physical environment because it uses holographic computing technology to overlay digital content within a physical space.
What do you see as the possibilities that Augmented or Mixed Reality presents for museums?
JC: Mixed Reality technology allows us to take media experiences out of their interactive kiosks or mobile devices and actually place our storytellers, historical characters and activities within a physical environment.
One problem we were trying to solve using the Microsoft HoloLens was the negative space created in museums by larger artifacts such as the hull of the “La Belle” and spaces that have restricted accessibility for archival and preservation reasons. The potential for overlaying digital content within these sorts of spaces greatly increases a museum’s or historical site's storytelling options.
Right now, the technology is still clunky. It is heavy and cumbersome and requires some finesse but the technology will become smaller and more user friendly and, ultimately, we think that it is going to be the museum tour device.
It’s very exciting because the opportunities are limitless—the glasses can provide your audio tour experience as well as enhanced visuals. You could put a virtual docent in the space, or an educator, even historical characters. You could enhance artifacts by showing scale or layering—for example, showing a full dinosaur over its skeleton—all without having to redo the exhibition or change the layout of the space. A museum could offer multiple types of tours for different topics or languages, or offer a text-overlaid version for hearing impaired guests.
The ability to place holographic images in the space and lock them to a physical location, which allows visitors to walk around for a 360-degree view, is really the standout feature and is unique only to the Microsoft product.
How does that location feature work?
JC: The HoloLens has the feature built-in. It uses cameras on the device to scan and create a 3-D wire-grid map of its real world surroundings. During use, it continues to map the space, updating as it goes. Developer tools allow users to see the mapping happen in real time.
In terms of the 3-D content, we were already using Unity3D, which integrates perfectly. But it’s not the only option.
More information on how to develop for HoloLens.
How long did it take to develop these two experiences and what were some of the challenges?
JC: We went through an internal research and development phase with the HoloLens to understand its capabilities and limitations. We also did a test run at both museums before eventually doing the more extensive demonstrations during SXSW 2017. We also brought the HoloLens team into the conversation about the demos. They were extremely interested in it and one of their team members came out and met with us in the office.
The development time was really quick because we already had the content. We just had to bring down the polygon counts and do some programming to import the experience. It was six weeks of one senior developer’s time.
One of the challenges that the design team faced was how to achieve a realistic look of the recreated “La Belle” ship while not overtaxing the processing power of the HoloLens. A lot of time was spent on the 3D model of the ship to make sure it ran smoothly, which meant sacrificing some realism.
In general, the mapping of different vantage points in the physical space to optimize the content within the HoloLens is a unique challenge to using this technology within museums. Attention needs to be spent on understanding what the visitor has access to and adjusting the experience to accommodate the visitor’s access.
For example, since visitors are only able to physically walk around about two-thirds of the ship, design choices were made to add more details to specific parts of the model where visitors would be able to get closer to it, versus other areas of the model that are less accessible.
Your willingness to commit that much time and money to trying something new is inspiring.
JC: It’s not something we do often, but in this particular case we decided to just go ahead. We feel it’s an extension of what we already do and we wanted to learn about it and experiment with it early on.
Have the projects spurred any new work?
JC: The Bullock Texas State History Museum has asked us to come back to do another demo, but I can’t say that these projects have created new work for us. HoloLens is not yet a product that we can produce work for. Right now we are using it in the office as an architectural and spatial design tool, but it’s just not yet ready for public visitor use in museums.
However, we have been getting increasing requests for augmented reality experiences using tablets and other mobile devices that further activate existing museum content and artifacts. Mixed reality headsets are a logical next step.
Could you describe the two projects you mentioned earlier?
JC: Our main purpose for this project was to understand the potential for the use of mixed reality technology in museums by observing visitors’ responses. From the moment visitors put the headset on and began to experience the content, their reaction was extremely positive. Immediately it was clear that this is an ideal technology for the museum space.
At the LBJ Presidential Library, there is the dramatic multi-story document wall within the main gallery space that has numerous windows that look into the Library’s archival collection. This is where all of the documents accrued during LBJ’s presidency are stored. It’s dramatic to look at because the numerous documents bound in red are not available to the general public. We designed a solution that turns six of the windows into menu items, one for each day of the Six Day War of June 1967. Since the windows are higher than most visitors’ natural sight lines, we then had the content “fall” from the selected window onto the floor in front of them. This allowed visitors to walk up to the archival images and documents, walk around them and get as physically close to the story as they wanted. Archival audio was also included in portions of the experience.
“Sometimes it can be difficult for a visitor to really understand the breadth of what it means to have a 45-million-page archive even as they’re looking straight at it in our museum. What Cortina did with the HoloLens technology brilliantly combined our physical space with our archive and brought those pages and even photos to life in a way that had never been done before.” —Kassandra Navarro, Director of Digital Strategy, LBJ Presidential Library
In contrast, the “La Belle” Gallery houses the once sunken hull of the expedition ship which serves as a focal point in the exhibit and acted as our physical base for the experience at the Bullock Texas State History Museum. Visitors walk up to the hull of the ship, put the HoloLens on and are able to see a recreated version of the ship overlaid on the actual hull. Visitors walk around the space to experience the immense scale of it and see historic details of the ship before it sank. This approach really works within the space because the artifact hull is in an area that has a multi-story ceiling, allowing visitors to look up and grasp the amazing height of the mast and rigging. Finally, to enhance the experience and really activate the space, the ship model we created was enriched with subtle animations of the ship at sea, such as flags waving in the wind or cannons firing at enemy ships.
How did you manage the experience for visitors, given the devices aren’t designed for mass use?
JC: People would queue up, and two to three staff members would fit the device to the wearer, give them a few tips for use and then let them go, retrieving the device after several minutes.
If you had to guess, when do you think this technology could become common in the museum world?
JC: I would say within the next two years. It’s hard to say for sure, but that’s what we think. With the museums we worked with, they could see the potential and found it to be very exciting way to offer new and enhanced content without changing the physical exhibit environment.
VR is becoming more and more common in museums, especially in Europe. We’re seeing more and more projects that come out via an RFP or RFQ that include a VR experience. We’re actually installing a shark cage VR experience in Norfolk using the HTC Vive. Again, the challenge there is that you have a visitor that can’t see their surroundings, so you have to manage that.
Do you think Mixed Reality technology will win over Virtual Reality, like Blu-Ray and HD-DVD?
JC: (laughs) I think they’re different. The VR experience takes you out of your reality and completely immerses you, and that has a place. The layering of content onto the real world also has a place. The MR glasses are really less obtrusive and will allow visitors to fully experience their surroundings, but they won't replace the unique and completely immersive VR experience. It's very possible that eventually one device will be capable of delivering both experiences.
And what would you say to those who would argue that AR and VR in museums are gimmicks?
JC: There will always be traditionalists, and that's great. Technology like this will be there for the people who want it and it won’t impact the museum for the people who don’t want it. That's the best of both worlds. Everyone wants to enjoy the experience in his or her own way. Personally, when I go into a museum I typically don’t pull out my phone—there are boundaries I set for myself in terms of technology and valuing the shared experiences I want to have with my family as we go through a museum together.
The last word?
JC: By leaning in to this technology, we can design amazing and engaging experiences without needing to alter the historical artifacts or physical location. Alternatively, eventually museums spaces could be designed in conjunction with mixed reality technology, making it easier to design and produce experiences that let visitors dive deeper into museum content without needing additional space in the museum for graphic panels and interactive kiosks.
Who knows? In the future everyone may have wearable headsets that they carry around like we carry smartphones today.
Project Name: Holograms in Museums
Client: Bullock Texas State History Museum, LBJ Presidential Library
Location: Austin, TX
Open Date: March 13-15, 2017
Interactive Experience Design: Cortina Productions, Inc.
Photos: Cortina Productions, Inc.
