A new way of mapping that captures real space is the next big digital revolution, and futurist Rich Green is impressed.

In May 2019 the 10th annual Augmented World Expo took place in Santa Clara, California, in the heart of Silicon Valley. Founded by AugmentedReality.org with the mission of ‘advancing augmented reality to advance humanity’, the events are held annually in the US and Europe, and have previously been held in Asia.

On the 12 July 2019 episode of The CEDIA Podcast – Augmented World Expo – iconic integrator and futurist Rich Green reported on his first experience at the show.

“I have been receiving emails and blurbs about the expo for years but I never really took it that seriously. I thought ‘Who cares? It’s just a game’. I couldn’t have been more incorrect.”

Most people in the residential technology sector know Rich as somewhat of a boffin when it comes to augmented technologies, and he has spoken on the subject in several forums. However, even he found some of the presentations ‘life changing’.

“Imagine CEDIA Expo about 20 years ago; it had so much passion and intelligence and energy. That’s where the Augmented World Expo (AWE) is today.

“What I found was a passionate group of people who had PhDs. I was surrounded by thought leaders from all over the world. It was astonishing.”

AWE was founded by Ori Inbar, who also founded the modern push towards spatial computing.

“Ten years ago, spatial computing involved clunky industrial things that you wore on your head, as you would see in Terminator.

“Ori saw that as a potential trend. Despite it being crude, he thought it was going to be very big.”

Ori is one of the leading voices in this market and is seeking to change the world in a collaborative and open way.

“There’s a lot of ways to describe what’s going on. Spatial computing is the general term but it includes the spatial web and Web 3.0 as well as augmented reality, virtual reality, mixed reality and ‘XR’, which is the general umbrella term ‘extended reality’.”

Ori wants to make computing human again.

“With the augmented world, you’ve got a digital twin of reality. It’s a 1:1 overlay of digital content on top of reality. The methods for making that happen are extremely complicated. It will take thousands of people to pull this together in a collaborative way.

“When the spatial computing world really starts to blossom, it will be because of user-generated content. We’re going to create this world.

“We’re going to map our interior spaces and exterior spaces with wearables that go with our mobile devices. We’ll be moving quickly from flatware to spatialware.”

Rich says this is a new way of mapping that captures real space, determining the geometric relationships between objects, modelling the state of the objects and the semantic relationships between them, and gathering it all into a digital collection.

That digital collection will be stored against an index so it can be easily queried.

“How we get there is ‘six degrees of freedom’, or 6DoF,” he says.

“If you’re going to put on augmented reality glasses, they had better support 6DoF. A real object in space can have three components; the Z-axis, X-axis and Y-axis. Then you go into pitch, roll and yaw. Those are your six degrees of freedom.”

The combination of an object’s position and orientation is referred to as the pose. When a pose is defined relative to a geographical frame, frame of reference or co-ordinate system, it is geographically anchored (geo-pose).

“All of the dynamic objects like people, cars, dogs, planes and insects in the real world have a pose. Spatial computing is a way of grabbing that, then making it searchable and indexable.”

During AWE, Rich attended a session sponsored by the chip manufacturer Qualcomm.

“Hiren Bhinde and Prince Gupta presented a session called ‘5G and XR’.

“Their view is that real-time 6DoF needs high bandwidth, low latency, high resolution and low power. To put all of that into a pair of glasses will burn the skin off your face. It will melt.”

The solution could be to have devices on your head as well as a device in your pocket and 5G communications with edge or fog capabilities so that you’ve got a low latency roundtrip for a data request from something that does real heavy-duty processing.

Augmented audio

Rich has long been talking about another aspect of the augmented world – audio augmented reality.

“Audio augmented reality is here. Bose, for example, was at AWE. The company has been putting audio augmented reality algorithms into its products without telling us.

“The QC35 II noise-cancelling headphones that everybody wears in planes actually have AR features built into them already. It was a secret release in November 2018.

“The new Bose 700 model, which replaces the QC35 II, has four beam-forming mic arrays as well as gesture recognition. Bose calls it Aural AR.”

Imagine, Rich says, people walking around airports or in the main street of a city. They are all using headphones or earbuds, and you think they are listening to music or a podcast.

“With Aural AR, they could be in an immersive audio field. They’re walking down the street quite normally but the audio augmented world is a 360º bubble of audio that surrounds them.

“It responds too, because it knows where they are. It does geo-positioning so it knows where they’re walking. It knows where they’re looking and it knows the orientation of their head.

“A little chirp off may sound to their upper right. They look over and sure enough, there’s a ‘sale’ sign.”

Rich says the opportunities extend beyond marketing. A blind person using the Bose headphones could walk down the street freely.

“We just have to wait for 5G to deploy. Cities may be getting it today, and phones with the technology are being manufactured right now, but the 5G plus edge computing package is years away.”

It will probably be 25 years before a fully implemented spatial computing world is possible, but there are many exciting steps, not the least of which will be the release of important products over the next 18 months.

Here and now

Several augmented technologies are available to integrators, some of whom may not be aware of them.

“I attended a session by a company called Spatial, which will be of great interest to anybody who is installing conferencing systems and things like that,” Rich says.

“During a video conference, if you were wearing a HoloLens 2 headset, for example, private messages would appear and we could throw virtual notes up in front of each other.

“We could create a virtual whiteboard to the side and scribble notes on it that we all had access to in real-time.

“Those notes have ‘location persistence’, so they will still be there the next time you meet in that virtual room.”

Another interesting application is Vuforia from PTC.

“PTC has a feature called Waypoint, which enables institutional knowledge transfer.

“This is making use of spatial anchors, waypoint beacons, QR codes and image markers – all kinds of things that you can pepper a facility or home with.

“With Vuforia, you can create a virtual home control keypad, put it on the wall and then when you put on your AR glasses, you can point to its keypad and say ‘lower the volume’. Or you can point to the ceiling fan and say, ‘turn on that fan’.”

Rich says augmented reality is growing in public awareness.

“Tim Merel from Digi-Capital, a statistics firm, explained that some people are familiar with Microsoft HoloLens 2. They also have some familiarity with the Magic Leap One head-mounted glasses.

“But 43% of the market said the most significant head-mounted augmented reality glasses are from Apple. What’s really funny is that Apple hasn’t even released them yet.”

The prediction is that Apple smart glasses will arrive in Q4 2020.