The Challenge:

We were invited by Augmented World Expo, the world’s number one XR conference and expo,  to showcase a never-before-seen XR experience at the Playground, a 20,000 sqft experience center. We were one of the 20 companies selected for this opportunity to demonstrate the most innovative tech in XR to  6000+ attendees.

The Solution:

It had only been a month since we unveiled the world’s most spatially accurate mixed reality experience at TED 2018 and we wanted to push the boundaries of our mixed reality framework and prove why spatial data is necessary for multiplayer mixed reality experiences.

First, we built a 6-foot tall physical replica of the empire state building and we programmed a virtual cat to interact with it. The cat is spatially aware so it understands the geometry of the building and can jump from ledge to ledge.

The players have to work together to build a virtual barrier around Kitty Kong before it goes on and destroys the world. The players are synchronized in real-time and can see what each other is doing. They are operating in the same frame of reference and this is possible because they are on a shared coordinate system.

By capturing the spatial data of this environment, we can also achieve things like occlusion, which is hiding virtual objects behind physical ones, and collision, which is colliding virtual objects with physical objects and casting of virtual shadows onto the real-world environment. All these things make the experience so much more immersive and believable.

The Results:

Kitty Kong turned out to be the first true multiplayer mixed reality experience the world has seen. Industry professionals were amazed by the seamless tracking, level of immersion and believability of the experience. Many recognized Kitty Kong as one of the first demonstrations of the AR Cloud, a machine readable, 1:1 scale model of the world. The AR Cloud is a proposed operating system for the spatial era and it will enable people to have shared AR experience.

The AR Cloud will add digital context to our physical world

As Augmented Reality devices become more ubiquitous, we will have the means to access information at its origin, whether it’s learning the history of a place or immersing ourselves in a multiplayer role-playing-game (RPG) in the real world.

In order for mass adoption of AR to occur, AR content must persist in the real world across space, time and devices. Persistence means that someone can create an AR experience in a physical space today, come back tomorrow and interact with it. Another user can also collaborate with it on a different device.

For example, an artist can place her 3D virtual art in a plaza for the community to interact with it, regardless of which device they use. The 3D virtual art will “live” in that space as if it’s really there and will not disappear between different app sessions. To enable such a streamlined experience, the AR Cloud is required.

Just like how the Internet was able to connect people through the web, we need a software infrastructure that can enable everyone to have a shared AR experience. This software infrastructure, commonly known as “The AR Cloud”, will become the single most important software infrastructure in the history of computing, far bigger than Google’s search index or Facebook’s social graph.

The AR Cloud will enable everyone to have a shared experience and become the operating system for applications in the spatial era.

Interested to learn more? Check out our Managing Partner, Alex Chuang’s talk on The AR Cloud below.