It’s 95 degrees Fahrenheit — 35 Celsius — or at least it feels like it. I’m standing in the middle of Times Square, looking at my phone at an augmented-reality fleet of boats floating above my head. My phone is hot. I can’t hear through the earphones well. A crowd of family tourists nearly collides with me as I try to stand near the marker painted on the ground. Nearby, the Statue of Liberty dances on stilts: A real person, I mean, in a costume.
Next, I’m wearing aon my head, seeing the floating boats semivisible in a haze of bright sunlight and my own sweat. I try to crane my neck at the holographic effect, as representatives from Microsoft try to keep me in the shade. There’s a reason why: my demo suddenly ends midway, going dark. The HoloLens has overheated.
Times Square in the summer is an awful place to experiment with augmented reality.
Still, the reason for all of this is good-hearted: Mel Chin, an acclaimed Houston artist who’s had a year-long series of works at the Queens Museum and throughout New York City called All Over The Place, is debuting two immersive works in the middle of Manhattan’s most crowded tourist spot, near the TKTS booth.
“Unmoored” and “Wake” are a two-part installation: Wake is an animatronic ship-hull-meets-whale-skeleton, a sort of physical wreck erected in a pedestrian plaza. Unmoored is an AR app designed to layer over the sculpture that is Wake, and create a six-minute audiovisual immersion to shake visitors out of their everyday lives and become more aware of how climate change could change places like Manhattan in the future.
How things got too hot
It’s a noble effort, and as a free installation running through September 5, it’s a great thing to check out if you have time. As a demonstration of AR in public spaces, however, it revealed a lot of the problems the technology still needs to work through.
It was soooo hot. I really wanted to focus on the illusion of decaying ships hovering over Times Square, the mysterious boat skeleton sculpture, and how the HoloLens handled broad daylight. But I couldn’t, because I was dying from the heat. That’s the funny part. Mel Chin’s work is made to raise awareness of ideas around global warming and flooding spaces. But mission accomplished: New York’s scorching ambient temperatures were already doing the job.
Augmented reality and bright sunlight don’t mix. Mostly, it had to do with heat. Both my phone and the HoloLens didn’t do well at all with high-performance stuff like AR in the super-intense summer conditions. ARKit-enabled apps cause iPhones to get a bit warm on any day, but in direct sun it became a big challenge. My phone didn’t conk out, but the HoloLens did. (Microsoft reps said that, once the sun cleared the buildings a bit and we were in shade, it would far better).
Ironically, virtual reality and bright sunlight don’t mix, either. I accidentally damaged anearlier this year by leaving it face up in the sun — the lenses fried part of the display like an evil kid burning an ant with a magnifying glass.
Crowds aren’t great for AR, either. Staging AR in outdoor places is, frankly, a bit of a mess. There were too many people in the crowded pen at Times Square during the premiere. It’s not a bad model of the average day in Times Square, though. There were other issues: AR effects make it seem like ghostly things are floating in the regular universe. But consumer AR can’t do occlusion well yet, which is the special effect of making virtual objects “hide” behind the real ones. If someone walks in front of AR, the optical illusion is ruined. In Unmoored, the boats are cleverly positioned high up, away from interference with pedestrians. But still, I found people breaking the effect a bit. Also, I nearly collided with lots of tourists.
AR can’t really sync the real and virtual worlds as well as I’d like. To make Unmoored work, I had to stand in one of a handful of labeled spots marked by letters, and the orientation of the letter had to match between my phone’s on-screen guided overlay and the real thing. I learned why: I started my experience imperfectly synced, and the real-world ship-sculpture didn’t overlap properly with the virtual one that appeared and slowly floated into the sky. Sync the markers wrong, and ships will look like they’re flying into buildings. AR can recognize walls and objects in close range and position objects properly, but long-distance effects don’t always work as well.
I could barely see anything! Augmented reality’s ghostly types of projections can look washed out on a HoloLens in super-bright daylight. On a phone, really bright light makes it hard to see the phone screen properly. Either way, I wished I was in a more sheltered space.
A promising exhibit you should try on a cooler day
I don’t want to ruin the experience for you, and odds are that on a cooler, more well-shaded day, this could all be a lot more fun. I love the potential of immersive art experiences that blend technology with the real world. But Mel Chin’s Microsoft collaboration ended up showing that the challenges of making this work well are still pretty daunting. Future AR headsets have a ways to go before they’re really ready for serious outdoor use.
: It’s also partnering with Adobe on a new AR file format
: The next-gen HoloLens is reportedly code-named Sydney.