How to Solve the VR model problem

The VR model problem

VR has a problem. Much current content is simply a 360-degree video. For those producing 360-video, I know the production of these experiences is anything but simple. That said, there is little complexity in 360-video, which means the participant doesn’t play much more than observer. Perhaps a participant can guide a path through the experience, but they can’t touch or manipulate anything in a 360 video.

360 video forms the base of the model problem, a problem that consists of three layers.

Problem 1: 360 video is flat and non-interactive.

Problem 2: 3D video is more representative of reality, but no more interactive than a flat 360 video. 3D video also increases data requirements without necessarily creating a better overall experience.

Problem 3: Photorealistic models offer the best, most programmable experiences, but they are hard to create. The inability to create these models also creates data throughput issues because they are the most efficient from the data standpoint, but represent only a small portion of available content.

This clearly oversimplifies the current issues with VR content. We currently live among tradeoffs. Relatively easy to create content is bandwidth-intensive but experience poor. Experience rich model-based VR offers the best approach to data compression, but also the longest lead-time for development.

I see two solutions to consider:

First is the development of tool kits for building 3D experience. This will be crucial to the B2B market, especially training, so that curriculum development professionals can leverage VR for content without be VR developers. This same approach is likely to also apply to product-oriented VR, such as retail and e-commerce.

The second is subtler, and much more powerful: the use of AI to transform video into models.

Video through the looking glass: AI to Models

We already know that Google has succeeded with language, and many images, that matches bitmaps to metadata (offering up tags or translations). Facebook users who upload photos know well the recognition of metadata based on facial pattern recognition.

Take pattern recognition and imagine a multi-layered AI approach that can identify objects in a video, match the images to a repository of models, and then reconstruct a scene from available models. This solves the model problem because the video becomes an input to the model. (It also creates a market for model makers that could be monetized).

This solution, however, does nothing to speak to narrative issues with VR, only a mechanism for transforming video into models.

Interestingly, this approach could be applied to any video stream, including realtime video. Early on it is likely to require extensive post-processing, but as the layers become more efficient, they should be able to create models in near realtime.

How big an issue is the VR model problem?

VR will not hit its stride until the majority of content accepts realtime interaction: the ability to walk through a scene, to interact with it – to experience it much as one would a real-world experience. The real value of VR may not be creating new experiences that people “haven’t imagined” as Oculus Head of Content, Jason Rubin, suggested at today’s VRX conference in San Francisco. I would argue that extending the reach of real-world experiences may be the most important and pervasive use of VR in the future.

Humans have spread around the globe and reached into space, but we still travel very narrow pathways. The ability for people to experience what other people already can has always been the driver for the tourist industry. In VR, everything becomes open to tourism, from surgical theaters to space stations. But for this to happen, we need to solve the VR model problem. Documentaries shot from a director’s point-of-view shouldn’t drive VR.  VR needs real experiences where people can interact with the objects, the people, and the environment as seamlessly in a virtual world as we do in the real one.