Breakthrough Examples

a complete chapter excerpt from:







You can’t see the future in your rear-view mirror.



Robots will need holographic assists as much as we do.


The whole point of this book is to help you approach your work in a way that will result in real breakthrough experiences. In practice, leaping forward is a bit harder than you’d think. First, considering yourself visionary and predicting exactly how things will unfold is fun, but far from reliable. Next, when looking around to see what you can copy from current competitors, remember that it isn’t going to get you anywhere in the marketplace. And finally, contrary to the myth, deeply analyzing the past is not the best way to predict leaps into the future.


Envisioning is your way to tap into the future.


Turns out the future is hiding in plain sight. By combining all three of those perspectives (past, present, and future) we can synthesize experiences that feel very new but are still familiar. We don’t rely too heavily on any one of the three viewpoints, but rather blend them and follow our instincts to emphasize particular aspects of each that best lend themselves to the scenario we are working through.



Leaping forward means combining the past, present, and future into an insightful perspective


For example, if we were looking at developing innovative sports coverage using mixed reality, a natural place to start would be with futuristic sci-fi movies that overlay information onto quick moving objects. That would translate well to augment the players. We can also see similar digital callout technology already being used in today’s broadcast sports. Looking back into the archives, we’ll notice a huge shift in how the action was televised with the advent of remotely driven camera rigs that could follow the action without being on the field and in the way of the players or officials.

Taken together as one combined viewpoint, you would suggest that our mixed reality leap forward for sports coverage would be to put us right into the middle of the action through elaborate camera tricks. This would be superior to completely immersive virtual reality broadcasts due in large part to being able to see and anchor yourself to familiar surroundings while watching. The real world acting as an anchor would potentially keep incidence of motion sickness down while enjoying fast-paced action. Not a perfect vision, but a good start.

Let’s go through some other examples where we’ve truly leapt ahead by combining different viewpoints and examine why they work.


Data Visualization

One the areas that will benefit the most from the rise of mixed reality is the visualization of data and complex concepts. We have made significant progress over the last decade in depicting data sets, but we have made little progress in being able to interpret them quickly. We keep drawing the same difficult to understand charts over and over again.

There are countless examples of data being poorly rendered in 3D or shown as awkward digital appendages of real world objects, so 3D and AR are not the answer here. It’s also not the tech that’s holding us back, it’s more that we haven’t been able to leverage people’s abilities to quickly comprehend and compare based on our natural ability to deal with our three-dimensional world. That situation is about to change due to the much richer palette available to all of us using holographic visualization.

Room-scale Data

Remember when we talked about the first time you ever saw a 3D movie? It was mind-bendingly different than anything you had seen before. Not because it was surreal, but rather because it felt ultra-real in a strange and new way. Seeing holographic data visualization done at room-scale is exactly like that. It’s truly transformational from an experience standpoint. It’s not that you are inside the data itself – that’s perhaps best done with virtual reality. The thing that gets you the most is how real the data now seems after it has escaped from behind the screen into your world.



Imagine being inside a room-scale datacenter visualization showing storage problems


Room-scale data feels so right that you start to question whether it can actually be that easy to comprehend, or if you’re missing something important about all this. You’re not. It’s just that we can finally deal with things in more natural way to make comparisons and judgements. We have become so used to seeing data visualized and charted in a very particular manner over the last decade, that when faced with any new technique our kneejerk reaction is that it must somehow be wrong. Trust me on this one, when you see data you are familiar with at room-scale, you want it to be right even if it isn’t. That’s the power of mixed reality in action.



Working with data that is building-sized is another extraordinary aspect of mixed reality



The real leap room-scale data has more to do with the experience than the visualization itself. There will always be skeptics who point to the inefficiencies and distortion of data when its presented in three dimensions. What you don’t realize until you are standing knee deep in data (literally) is the visceral power of that experience. Our ability to make sense of the physical world and its contents kick in immediately. Coupled with even a passing familiarity of the data itself, the act of viewing room-scale visualizations is a bolt of lightning breakthrough for those of us who look for insights.

Consider the following aspects:

Understanding. One of the hardest parts of designing data visualizations is being clear enough upon first inspection to create understanding in the mind of the viewer. Many data charts are not designed to be easily read and analyzed. Holographic visualization has the built-in advantage of producing a feeling of “getting it” due to our ability to compare and contrast three-dimensional objects quickly. Used deftly, this aspect can reduce the time-to-understanding measures in this field.

Initial Impression. Holographic visualization also has another built-in advantage of producing a first impression of shock (in a good way). It’s hard to describe how it makes you feel when data envelopes you, or is spread out across a table. This qualifies as a real breakthrough moment. Suffices to say it’s nothing you’ll soon forget.

Visual Fidelity. We know that in most cases there’s little reason to use 3D charts over 2D to convey meaning. A well-constructed flat chart is plenty clear most of the time. But, there are those times when using 3D adds some other dimension of information that results in better clarity or insight. Holographic visualization has the property of allowing you to view it from new and more natural angles which creates a situation where insight surfaces more readily.

Direct Manipulation. You will immediately reach out to touch and try to directly interact with the data objects when looking at holographic visualization for the first time. It’s an instinctual reaction. They are right there. Why wouldn’t we want to see what happens? Well, now we can. It’s incredibly satisfying to use physical gestures to work with digital data objects.


Broadcast Media

Many of the early examples of pushing the envelope of mixed reality experiences focused on entertainment and education as many of their common scenarios lend themselves well to holographic content. Not hard to imagine being inside a game or an atom for that matter using this new technology.

As you’d expect, the seemingly more mundane area of broadcast news has been left mostly untreated. There were some interesting experiments during the 2016 United States presidential election coverage where pundits and reporters were beamed into the main studio set from remote locations, appearing as full-sized holograms. Interesting use of telepresence concepts and mixed reality tech, but not completely surprising or implausible.

That all changed recently with the debut of a broadcast-quality mixed reality video compositing solutions to seamlessly integrate live action TV reporters with digital content to astonishing effect.



Mixing broadcasters with sports players in the studio is becoming common (source: Vizrt)


Breaking the Fifth Wall

In his insightful post Breaking the Fifth Wall, James Corbett notes we have recently been able to move past the charming cinematic technique known as “breaking the fourth wall” (where a film character surprises the audience by talking directly to them mid-scene) to an entirely new level of sophistication.

Using mixed reality techniques, digital actors can now not only break out of character to address viewers, but also inhabit the same physical space as the audience. People and things can now join you wherever you happen to be viewing from.

This is somewhat profound. We can now shift the audience from mere observer to active participant in close proximity to live action. This new capability called “breaking the fifth wall” by Corbett extends the original film metaphor to have characters literally leap out of the screen and into your physical space. There are nearly endless possibilities for this new mixed reality technique in broadcast news and sports.



The emphasis here is on cutting-edge video processing technologies for mixed reality. These new technologies enable the following techniques:

Real-Time Live Action Compositing. It’s one thing to be able to experience live action news reports or sporting events and feel like you’re really there on the field. Virtual reality has been able to create that effect for awhile through 360 degree video. But, we haven’t until now been able to pull live action content into our own physical space – the complete opposite arrangement of what’s come before. Careful to not interfere with that fly ball or get swept up by that storm.

Come To Me. We are comfortable asking for the ability to travel great distances without leaving home. Again, that’s old news in the VR realm. What is not even thought about usually is asking for people and things to come into my own living room or office as digital objects that are perfectly placed, sized, and aligned with my world.

Dynamic AR callouts. Notice in the previous images the 2D textual information and 3D graphics that are deeply integrated with the live action content on screen. This level of alignment and tracking is possible today within mixed reality, but the difference here is that the capability now scales up to allow for many of these augmented reality style callouts to accurately track critical aspects of the broadcast.



There is perhaps no area of creativity more interested in leveraging new technologies to educate the curious than the fine art world. Museums had traditionally been able to offer guided tours of their collections to patrons by using staffers and volunteers. They learn all about pieces of high interest so that they can answer questions on the artist, period, and contemporary works. As low-cost technologies came to market that would enable unattended tours, museums started investing in them. The state-of-the-art until recently was a multimedia affair using tablets to see video clips and hear narrator voiceovers. Now with mixed reality, it’s possible to explore completely different dimensions of these artworks.



One of the most important aspects of artwork is of course the artist. There are sometimes plaques and wall displays containing some background information on them if you look. Yet, far too many artists go unrecognized and unheralded because they are not featured as people along with their works. Somehow, we detached the person from the art, never to be reunited again except in art history books. Within galleries, the artist is predominately invisible. Until now.

With a bit of foresight and forethought about what aspects of their work the general public will want to know about, artists can maintain a round-the-clock presence with their work to answer questions, chat with art lovers, and just have some fun with the gallery patrons.



Finally, artists can explain their motivation and methods to us in a personal way



Much of the innovation here comes from adding fun to a traditionally subdued experience. Imagine the following experiences next time you visit a museum:

Meet the Artist. Just as we never expected to be able to have a video phone in our pockets, the idea of having a conversation with an artist about their work didn’t ever seem to come up. Goes without saying that possibility approached zero after their death. Now, using easy 3D capture techniques, we can either beam in live artists to moderate discussions in real-time with students, or have artists posthumously explain their thoughts and process to gallery visitors on demand. It’s also easy to imagine vast amounts of digital content on the artist themselves being available right there along with the pieces.

Get Closer. By digitizing the artwork itself, it can now be viewed from any angle, at any size, from any location on the planet or off. The primary benefit of hi-fi reproductions is not the ability to zoom in to great detail, but rather to have the piece hold its quality constant as it is viewed and placed at any size that seems to fit the location its being viewed in.

View Related Work. A common question when an art patron comes across a piece they admire is “Are there any other pieces like this one in the collection?”. By using voice input to ask these types of things casually, the overall experience and discovery of new artwork is greatly magnified. Serendipitous exposure to new ideas and styles is increased.



Music and dancing never go out of style, but it seems to be getting harder and harder to see the popular acts we like. If we do manage to get a few astronomically expensive tickets, it’s often so crowded or restricted at the venue that cutting loose and dancing up a storm is either impossible or frowned upon by security. Thanks to magic of technology, we could all don VR headsets and watch acts perform from the comfort of our living rooms, but what’s the fun in not being able to dance without smashing into your furniture? Immersive experiences are great, but much better to be able to see and interact with friends and loved ones while you enjoy the moments together in real life.


DJ Holo

With the ability to see everything happening around you in mixed reality situations, this scenario of dancing wildly around while listening to your favorite performers live with friends is entirely possible today. Using the beam in approach mentioned earlier in the Breaking The Fifth Wall example, we can literally bring the acts we’re dying to see to wherever we want to gather and rave.

There’s two parts to this of course – the ability to see and hear the holographic versions of the artists, and the ability of others to see the same thing. Done and done. Thanks to the shared experience code freely available for holographic platforms, we can synch the show for more than one person at a time. We can even add some special guests to the ambient environment in the form of holographic sidekicks or stage props.

The only thing we didn’t talk about yet within this breakthrough experience is how ridiculous you and your friends will look singing and dancing around with special optics on to an invisible and silent band wherever you happen to be. Whatever, right? This is fun. And in the years to come, others will be able to see and hear the act just fine without any special optics or headsets thanks to holographic projection technology.



Inviting the hottest DJ around into your lobby for live beats is just around the corner



Shared real-time experiences are still in their infancy. This is an area of rapid developments. Expect to see experiments in the following areas in the years to come:


Living Room Shows. Who hasn’t dreamed of having their favorite artists play a show in the privacy of their own home or backyard? Using the evolving volumetric video broadcasting technologies, performers can push out shows over broadband that can be subscribed to wherever you want on multiple types of devices. Whether on the go, or at home, you’ll be able to catch the show in life-size or scaled down to fit on a tabletop.

Virtual Tickets. Being able to broadcast and receive holographic performers on multiple types of devices opens up an entirely new market for live and pre-recorded content on-demand and via subscriptions.

Holographic Friends. You don’t really have to invite your friends or coworkers over to throw a party while catching the show. The same presence techniques that we use to broadcast and monitor the show can be used to pull other people into the space (or mute them if they get a little out of hand).


Emerging Technologies

One of the most fun things about envisioning is projecting your thoughts forward enough to catch a glimpse of things that won’t exist for a while. It’s usually not the case those things are impossible to create, it’s more likely the technology they are based on doesn’t exist in a form that can be put to that use immediately. Even though we try to never lead with technology as a driver rather than human needs or curiosity, it’s exhilarating to see what those combinations of emerging tech and novel approaches can create. It’s tempting to try and predict which will become true breakthroughs (even if they’re still on our mental drawing board).

Let’s go through that exercise of designing breakthrough holographic experiences based on the latest crop of emerging technologies listed below. The real question of course is how do holograms help turn these technology-influenced ideas into true innovations for people?

Some examples of breakthroughs powered by emerging technologies:


Artificial Intelligence


In a few simplest ways, holograms already feel like they embody some sort of AI because of their digital presence. What if the power of Alexa, Cortana, Siri, and Watson was somehow omnipresent and active when dealing with holograms. We’re kind of used to talking with a few of these already in public. Do we need to infuse this type of conversation with an AI into every hologram?


Machine Learning


There’s nothing worse than a dumb hologram. We have come to expect that any kind of digital object or program will understand our intent and the environment it exists in, plus offer solid suggestions based on even minimal information or interactions. This is more a function of modern apps and services rather than strictly a mixed reality play, but we need to design this into our scenarios.




Will robots of all shapes and sizes need their own digital companions to keep from going crazy? Are those companions going to appear to us as holograms? I think so. Also, think about a world where robots are more common in the workplace and the home. How do holograms blend into those types of interactions? Will there be a balance during interactions where each player knows the intent and responds accordingly, or will robots and holograms collide in awkward ways? We’ll see shortly.


Autonomous Vehicles


You would think that the near future of transportation will come with its own holographic additions and advances. What better way to spend your commute time or long train ride than conversing with a holographic entity. Certainly, the media we consume will be punctuated by holographic experiences during this travel time. But, will we want to be talking with holograms that other people may not be able to see?


Sensors and The Internet of Things


What a fantastic opportunity exists to reinvent how we deal with dashboards of information reporting the exabytes of information pouring in each second from our billions of sensors around the world. Holographic displays bring the promise of using effective dimensional data visualization to help people better understand the state and trends within these incredibly complex systems.


What’s Your Breakthrough?

With all these features of other experiences bouncing around in your head, it’s time to dive into creating our own breakthroughs. Let’s get started by running through an easy to learn approach to constructing some working prototypes of our holographic innovations.



End of Chapter 15 – “Datascrapers”
from the book “Envisioning Holograms” by M. Pell






Learn more about the author M. Pell at







Copyright (c) 2017-2018 Mike Pell. All rights reserved.