• Breaking the Fifth Wall with Mixed Reality
 Trivia question: What do the following actors have in common — Kevin Spacey, Woody Allen, Jeff Daniels, Michael Caine & Matthew Broderick?

Well, yeah, the title of this article does kind of give it away — they’ve all played characters who famously broke the fourth wall, of course.

“Breaking the fourth wall is when a character talks directly to the audience or viewer and acknowledges their presence”

Storytellers employ this device for different reasons but often the purpose is to make you, the viewer, feel less like the passive observer of a fictional world and more like a participant. Or more involved. The idea is, figuratively speaking, to make the fictional realm spill out into your own world. Which is why my favorite example is the first meeting of the characters played by Jeff Daniels and Mia Farrow in The Purple Rose of Cairo

It’s a movie within a movie of course so perhaps this is more correctly referred to as a Third and a Half Wall Break but, pedantry aside, it beautifully illustrates the purpose of a Fourth Wall Break when Daniels’ character literally walks through the screen into the audience.

From a charmingly old-fashioned example let’s jump to a thoroughly modern take on TV ‘wall breaking’. At the recent NAB trade show Vizrt demonstrated its technology for virtual TV studio sets and augmented reality effects. By transitioning animated and photo-realistic graphics of football players, from the video-wall behind him to the physical set in front of him, the presenter creates added drama by using the technique to bring the avatars to life, so to speak. They don’t quite spill into the viewers’ home but they do spill into the studio –

As it happens Microsoft imagined what it would look like to take the next step, that is to pull the virtual set all the way into the living room, by using Hololens of course –

With virtual set technology like Vizrt the Augmented Reality stays in front of the camera, in the studio. But Hololens effectively pulls the augmentation through the camera. If you think of the camera and the TV as being back-to-back — one pointing into the studio, the other pointing into the living room — then Hololens figuratively opens the portal to pull some of the TV studio right through.

TV Thereness

Let’s think for a moment about the levels of thereness that you, the viewer, can appear to have to other viewers in a TV show (I use the word thereness instead of ‘presence’ or ‘involvement’ so as not to confuse with VR presence or emotional involvement).

  • Text-in: At the most basic level you can participate in a poll or voting. But there’s very little ‘of you’ apparent to other viewers.
  • Phone-in: This is a big step-up in thereness — now other viewers can hear your voice and its inherent emotional qualities.
  • Skype-in: A much bigger jump — other viewers can now see your body language and facial expressions. You’re much more ‘there’. In fact, sometimes you’re too much there! —

I doubt any reader of this blog hasn’t yet seen that video but I’ve included it here because the qualities that made it a viral hit point to something that is almost the opposite of a Fourth Wall Break. Talking heads Skype-ins are old-hat by now but what makes this special is how Mr. Kelly’s living room spills into the TV studio, so to speak.

So what’s the next level? After Text-in, Phone-in and Skype-in how can you, the TV show viewer, be even more there? Well, you can Beam-in —

Okay, with 35 HD cameras required you’re not going to be doing that anytime soon. And the usage here was a little bit ‘gee-whiz look what we can do even if we don’t know why we’re doing it’. But the idea itself is a natural progression.

And the good news is that we’re on the cusp of being able Beam-in with consumer grade VR and AR technologies. How? With Mixed Reality. The video below shows a Mixed Reality experience we set up for Eir at the Dublin Tech Summit where VR headset wearers stood against a green-screen to be live composited into the Tilt Bush sketch they were drawing such that onlookers could fully grasp the three-dimensional spatial context on a regular TV screen —

The thing is of course, that same VR set-up (HTV Vive) could be in the player’s home, and that TV screen could be anywhere (we can live stream to it, just like we can live stream VR video on Facebook now). So using regular consumer VR tech the ‘viewer’ now has the means of beaming in from her living room into ‘the studio’. The Hololens allowed us to pull the virtual studio set into the living room, to Break the Fourth Wall. And now Mixed Reality allows us to push the living room into the TV studio (the exact opposite) to Break the Fifth Wall.

But the scenario above is rudimentary. This basic technique will be enhanced by more ambitious software and camera movements. Cameras can be programmed to respond robotically to the context of what’s happening in the VR (e.g. a low angle shot to bring an enormous monster into frame looming above the player). And we’ll see graphically dynamic multi-player scenarios where eSports meets game-show –

Great, but nobody is going to paint their living room walls green in order to beam-in to a game show! True, and they won’t have to. Because the same effect of extracting a person from their background can be achieved using depth cameras alone. And within a few years the majority of smartphones will have depth cameras.

Google’s depth camera technology — Tango — is already appearing in a few high-end Android phones and the next iPhone is strongly rumored to be introducing depth-sensing.

In case you think this is crazy talk, FremantleMedia, who produces and distribute some of the best-known TV content in the world, has started broadcasting Lost in Time in Norway — a “newly-developed, first-of-its-kind entertainment format using Interactive Mixed Reality on both TV and mobile devices.”

Using a game engine means its cross platform and bridges the gap between traditional TV entertainment and gaming activity. Advertisers can blend commercial messaging into the content itself making it a seamless part of the content.

Product placement is a given with logos designed to mimic the virtual environment (e/g a Pepsi logo styled to fit a saloon in the Wild West).[Source]

Nolan Bushnell, the founder of games developer Atari Corp. and a consultant to the project, claims that the fusion of gaming with TV can “bring a standard construct for new kinds of entertainment.”

And that’s before they even start Breaking the Fifth Wall.

Leave a Reply