Out of the Box

With ambient interactivity, user engagement (and design) is escaping the screen.

Not so long ago, ambient interactivity was largely the province of the art world, considered a fringe phenomenon. But digital technology is radically changing the concept of interactivity. No longer restricted to the artificial, fixed “interactivity in a box” exemplified by kiosks and display screens, it is becoming more fluid and organic, embedded in a wide range of physical forms—from furniture, objects, and floors to architectural elements and building facades.

This new generation of interactivity provides designers a wealth of new creative platforms and offers brands, retailers, and cultural venues almost unlimited opportunities to engage their audiences. The ubiquitousness of sensors in the environment and the rise of natural user interfaces like gesture-based technology have dramatically changed the prospects for ambient interactivity. Although it’s still in its infancy, more extensive experimentation with ambient interactive forms is shaping its development and yielding key insights into the nature of users’ interaction with their environment, information, and knowledge.

The Internet of Things

The evolution of ambient interactivity  is closely intertwined with the evolution of digital technology and the emergence of the Internet of Things, says Ana Monte, creative director for the interactive technology developer YDreams Brazil. “Touchable surfaces of all kinds of objects will be the next step in ambient interactivity. These surfaces will be combined with biometric recognition and devices controllable by apps to afford highly intelligent and customized environments.”

As the Internet of Things evolves, Monte adds, “The whole environment is connected and controllable by many input devices and the environment itself must communicate with the user.”

Eli Kuslansky, partner and chief strategist for New York-based Unified Field, agrees. He predicts that in the next three to five years, ambient interactive forms such as responsive environments, embedded technologies, and interactive surfaces will be part of networked ecosystems that integrate multimodal sensors, media, big data, and social media networks into the built environment.

“Conversations about these ecosystems won’t be as focused on technology and networks as they’ll be on people networks and resource and knowledge networks. It’s a new paradigm where multiple information and communication channels will be linked intelligently and can be accessed from multiple points of access.”

Transforming user interaction

Ambient interactivity completely changes the dynamics of user interaction, facilitating organic, multidimensional interactivity that is more natural to the way people use and manipulate their physical environment. It creates a completely new interaction model because it dispenses with the experience of “staring at a flat computer screen,” says Mark Foster Gage, founder of Mark Foster Gage + Associates (New York) and assistant dean in the Yale University School of Architecture. “It can add a deeper and more responsive element to architecture and content.”

It can also keep physical spaces fresh, responsive, and dynamic. For the innovative and “rebellious” brand Diesel, Gage created an architectural strategy that uses robotic screens, reflective surfaces, and social media feeds to create an immersive, techno-industrial space. In this yet-to-be-implemented store concept, the physical space is canvas for both Diesel-produced and customer-generated content. Visitors interact with the space and change the content, sound, hue, and color effects in real time via infrared sensors that track their hand movements in front of the screens. Illuminated handrails increase in brightness dramatically where they were touched, using proximity-sensitive film dots.  

Gage says ambient interactive design allows architects to fine-tune the environments they create, especially by allowing spaces to be programmed to vary by time of day and audience. “It also offers the element of surprise,” he asserts. “Spaces become more comfortable, active, and engaging.” As a result, interactivity becomes easier to use and more transparent, allowing visitors to retrieve timely information more casually.

Architectural scale

Marcela Sardi, owner of Sardi Design (Baltimore), sees ambient interactivity in public spaces increasing rapidly in response to the need to make spaces more intelligent and engaging. In some instances, interactivity is embedded at an architectural scale.

Working with brand and placemaking consultancy MRA International and a team of content developers, Sardi was responsible for integrating a series of monumental media components into the new Tom Bradley International Terminal at LAX. The $40 million Immersive Environmental Media System (IEMS), launched in September 2013, includes seven architecturally scaled digital displays sited to coincide with key points in the passengers’ journey through the terminal. The project uses multimedia content developed by Montreal-based Moment Factory, leveraging its X-Agora content management and software system.

Sardi emphasizes the need to place interactive media “judiciously, in direct relation to a point in the visitor’s experience.” At the Tom Bradley terminal, immersive media components are synchronized with specific points in the passenger journey through the terminal, and often keyed to their destinations. As passengers navigate through the terminal’s North and South concourses on their way to their departure gates, they encounter a sequence of 10 LCD pylons displaying content focused on the art traditions of destination cities accessed from that concourse. As they pass the 28-foot-tall monoliths, their movement triggers changes in the displays.

The 80-foot-tall Welcome Wall greets arriving passengers as they enter the terminal, displaying dramatic high-definition video interlaced with audio that greets them in their native languages, keyed to the origin of the flights. Video displays at the base of a 72-foot-tall “Time Tower” generate customized, real-time visual effects in response to visitor gestures.

For ambient interactivity at this scale, as with other forms of digital technology, says Sardi, “the experience must be natural, invisible, and simply part of the environment.” 

Windows on interactivity

Interactive store windows are one of the most vital and popular forms of ambient interactivity, and offer one of the most transparent user interfaces.

Marcus Wallander, a digital creative official for Great Works (Stockholm), says interactive windows are potentially powerful tools for retailers because they attract passing customers and are relatively cost efficient compared to other media.

Wallander was part of a team that developed a prototype interactive window for streetwear brand WeSC. Content was created by online users who chose WeSC “activists” as characters in their videos and selected visual effects as their backdrops. The videos were projected inside the store window in stop-action mode until the movements of passersby triggered the “Activists” into action, performing daring skateboarding moves, for example, against kaleidoscopic backgrounds.

In collaboration with NIKE Brand Design, Staat Creative Agency (Amsterdam) conceived and designed an interactive window for the NIKE House of Innovation at Selfridges in London in June 2013. The window, which Staat developed with Random Studio, Jurlights, and The Set Company, was designed not so much to market certain products, but “to showcase the diversity of NIKE’s products in a non-intrusive way, amplify Nike’s brand image, and immerse people in the NIKE brand experience,” says Martijn Lamabada, Staat partner and creative director.

The window invited visitors to experience NIKE products in new ways. For example, 74 flashing strobe lights in the window and video art were employed to showcase a distinctive feature of a new metal jacket. Staat created a simple game connected with Nike’s lunar shoes, inviting passersby to “jump the highest.” Kinect-based motion sensors calculated the height and velocity of each visitor’s jump.

Lambada notes the nature of content for interactive windows and other forms of ambient interactivity depends to a great extent on their location. Content for outdoor interactive windows, designed to catch the eyes of passersby, should be “short and hard hitting.” Content for interior interactive windows can be longer, he concedes, as customers typically spend more time in stores than outside.

Interactive surfaces

As ambient interactivity evolves, it is spreading to a wide variety of surfaces: walls, ceilings, floors, and multi-touch interactive tables used in museum and retail settings.

Companies like Ubi Interactive (Seattle) are marketing off-the-shelf products that turn virtually any surface into an interactive screen. Ubi uses Microsoft Kinect, proprietary software, and projectors to create multi-touch and gesture-based interactive surfaces.

Interactive tables have become a digital technology staple that museum-goers are now used to seeing, but these tables are also showing up in retail settings, particularly restaurants and bars. Kuslansky said they deliver multiple benefits. “They are still a novelty, and importantly, they break the one-screen, one-person limitations of touchscreens.” They also create social hubs and in space-constrained exhibition spaces, allow museums to cover a lot more content than traditional interpretive displays.  

Kuslansky says new developments such as transparent monitors with touch surfaces and flexible materials that respond to pressure are just coming on the market. Embedded and tagged objects like fiducial markers that can be recognized by sensors are becoming more common. In retail settings, he adds, connecting these surfaces to tagging technologies like RFID, near-field communication, and iBeacon will allow businesses to link them to their databases.

Kuslansky is also tracking is Microsoft PixelSense, a multi-user, object-recognition technology that allows direct interaction and multi-touch contact and can detect common objects placed on the screen. Older technology used infrared cameras in the base of a table that stopped functioning when the ambient light was too bright. Pixelsense instead has sensors embedded in every pixel of the display so the table can be much shallower.

Design and technology studio Potion (New York) creates interactive experiences for a wide range of applications, from museums to corporations. For NOVIY, a hip restaurant and lounge in the heart of Moscow, Potion designed a suite of ambient interactive elements including interactive tables, countertops, and murals.

In the restaurant entry, Potion created “Digital Floorboards,” a 20-foot-long reactive floor projection. The floorboards reflect the restaurant's unique wooden ceiling pattern designed by Russian architect Alexander Brodsky. When patrons step on the floorboards, they change from their natural soft gray to NOVIY's iconic orange, then fade back to neutral tones when not being stepped on. The result is a digital burst of orange that follows guests as they enter and exit the restaurant or cross from side to side. The visual experience happens behind the active participant, so it is best seen by their fellow guests—making the installation as much an event for spectators as it is for the individuals in the spotlight.

Lighting it up

Ambient interactivity can be applied to something as unexpected and mundane as lighting fixtures. That was the approach taken by Second Story (Portland) when it collaborated with Intel to develop Lyt, a collaborative lighting fixture and drawing canvas. The Lyt prototype allows individuals and groups to use their mobile devices to paint their space with color and movement. By connecting to the fixture via WiFi, they can draw on it, control its intensity, and adjust the character of the light, says Daniel Meyers, creative director, environments. Debuted at Maker Faire Rome in 2013, Lyt “tapped into an innate citizen desire for crafting their own objects and controlling their environment.” Second Story used Intel’s Galileo microcontroller board for the project.

Lyt’s aim was simple fun, he explains. It was “a DIY electronic dream” and essentially “an experiment designed to test user reactions to drawing on a large canvas in big groups.”

He envisions great potential for what he describes as “responsive” environments. “The idea here is that objects at the scale of furniture can become digitally enabled through light. In the future we’ll see the proliferation of technology into objects all around us. Smart technology will increasingly be embedded in every facet of physical environments.”

Ambient…and everywhere

Unified Field’s Kuslansky believes ambient interactivity will soon begin to surface everywhere, from stores to museums and city streets. In stores, it will take the form of “seamless, participatory shopping experiences connecting all touchpoints, extending from online to in-store, and linking personalization with customer service and sales with branding.”

He believes ambient interactivity is revolutionizing the museum experience by offering greater context for exhibits and extending the bricks-and-mortar experience to encompass the digital realm. For art museums with extensive collections that can’t be displayed due to space and resource constraints, multi-touch tables, gesture-based interactive experiences, and mobile apps are great ways to draw from a museum’s broader collection, engage audiences, and generate new revenue streams.

In public spaces, the emergence of Big Data will dramatically impact civic and urban planning and ultimately change the way public environments look and work, Kuslansky says. “Ambient sensors are everywhere, and are collecting an avalanche of data every millisecond of the day.” Unified Field has developed a concept called Legible City that provides a means for people to “read the city” in an actionable way. “By providing greater access to big data from cities, sensors, and social networks, for personal use, and to mediate its flow, citizens will have a greater awareness of their environment. Ultimately, the Legible City concept allows cities to leverage these vast storehouses of data to enhance services and improve residents’ quality of life.”

-By Michael Mascioni, eg magazine No. 08, 2014

Editor's note: Michael Mascioni is a writer, market research consultant, and conference planner in digital media. He has written about digital technology for such publications as Internet Evolution, Digital Signage Today, Sign Media, and Technology Review, and is co-author of The Out-of-Home Immersive Entertainment Frontier (Gower Publishing, June 2014).