While the metaverse, in all of its various depictions, may not be matured just yet, there are opportunities in adjacent spaces for integrators. Nick Ross adjusted his headset to take a deep dive into the metaverse.

Of all the buzzwords circumnavigating the globe, there are few that are as well-known, but little understood, as the metaverse. So, what is it, what opportunities does it provide the AV industry and where is it heading?

I don’t know what the metaverse is and at this point I’m too afraid to ask

Metaverse is an all-encompassing term for immersive and interactive virtual worlds that are found on the internet.

Early examples of The Metaverse (many of which are still thriving today) include Second Life, Minecraft, Roblox and numerous other gaming-related worlds like the Massive Multiplayer Role-Playing Games (MMRPGs) Eve Online, World of Warcraft and, more recently, Fortnite.

The Metaverse also includes Augmented Reality (AR) games like Pokémon Go. Meta’s recently launched virtual world is called Horizon.

AV opportunities

For some years, Melbourne’s Eureka Tower has utilised VR to complement its visceral-but-dated, real-life The Edge ‘experience’ which shifts visitors out of the side of the building in a glass-floored booth. Its VR Plank experience is installed in the middle of the room and consists of a plank lying on screens displaying a cityscape far below.

However, to active participants wearing a VR headset, they race up 88 floors in a virtual lift and exit the building on the virtual plank (which they can actually feel with their feet) avoiding the shear drop on either side. They then grab a hang-glider rail and fly around the city. The core technology simply involves two HTC Vive headsets and an Acer Predator gaming laptop.

But that’s not all. When Eureka Tower relaunched its (rebranded) Melbourne Skydeck last December it added its Skydeck Voyager theatre which is a ground-floor room that features 24, cutting-edge, immersive, pods from Positron. These 1960s-supervillain-style chairs rotate, tilt, provide haptic feedback and even administer scent. Combined with VR, they’re the basis of an experience that takes paying customers on a journey through Melbourne’s greatest sights, smells and sounds (from local artists) throughout history.

These are the sort of rides that one would typically associate with Disneyland and such ilk, but now any provincial tourist attraction can reach out to the AV industry (or vice versa) for a world-class metaverse-oriented installation for profitable or promotional reasons.

Nonetheless, at the recent Integrate conference in Sydney, few delegates responded positively when asked about metaverse opportunities. The only enthusiasm came from digital signage companies (with internal creative departments) which were looking at virtual billboards within video games and virtual worlds to boost inventory.

The closest thing we saw to a metaverse installation was Melbourne-based Igloo’s ‘igloo’ tent which used five ceiling-mounted projectors to create an almost-360-degree, seamless, immersive environment for multiple use-cases. Igloo stresses that its technology can be installed into any shape and size of room in addition to utilising off-the-shelf tents like the one on display. Its primary aim is to create an immersive and/or interactive space for groups of people.

This can include medics practicing high-intensity procedures within a virtual hospital; tradies learning about dangerous OH&S issues in a safe, look-a-like environment; academic groups discussing complex imagery on huge, interactive, virtual whiteboards; TV presenters discussing Winter Olympics replays on the walls of a virtual studio that appeared atop a snowy mountain; real-estate salespeople showing what life in an un-built apartment might look like; and walking through virtual worlds that have been created with Unreal Engine.

Will the metaverse rely on VR?

Personally, I have long suspected that VR will only thrive when wearing a headset becomes like wearing sunnies. That said, VR headsets like the Oculus Meta Quest 2 are becoming increasingly quick and simple to put on.

Could this be the technology kickstart adoption of services like Fox Sports VR which enable the watching of multiple-PoV streams of sporting events, movies and concerts – on a virtual couch with mates? It’s doubtful: leaving aside discussions of the demographic differences between Foxtel’s audience and new-technology adopters, wearing a headset for that long (maybe while enjoying a few tinnies) will be sweaty and uncomfortable for most people. Furthermore, it didn’t catch on despite lockdown’s lockout of crowds.

There are success stories though. Epic Games’ Fortnite and Meta’s Horizon have both staged massive in-game, music concerts. But these events set the technology-barrier bar low to achieve maximum audience size and support every screen size and consumer headset in lieu of professional installations. It’s not yet clear what value the professional AV industry can add to such consumer content.

Volumetric video

At the forefront of blending the physical world with the virtual world is volumetric video which has been perfecting itself for more than a decade. Currently, the peak use of the technology can be seen in London at the Abba Voyage live-yet-virtual concert. The show took six years to develop and included the four original stars spending five weeks performing in mo-cap suits, surrounded by 160 cameras, which scanned their body movements and facial expressions.

The resulting virtual avatars (affectionately called ABBAtars) were then digitally adjusted to resemble their 1977 selves. At the concert, they’re displayed ‘on-stage’ on a UHD, 63-megapixel screen, alongside 10 live musicians, in front of a live audience, at a specially built arena in London’s Olympic Park.

The ABBA Voyage live-yet-virtual concert in London re-created band members as virtual avatars, or ABBAtars, of their 1977 selves.

It’s notable that, even with the show’s huge budget, holograms still aren’t being used. Producer, Svana Gisla told The Daily Mirror last year: “We’re not making three-dimensional holograms… I don’t think any hologram shows have been successful. After five minutes, I don’t think they’re that interesting.”

Abba’s concert represents a journey that’s just one fork of a road that arguably started with pioneering virtual (anime) celebrity Hatsune Miku in Japan, includes Snoop Dogg performing with the late Tupac Shakur (at Coachella 2012), plus mind-bending music videos from Japanese female group Chelmico. The latter should be of particular interest to the AV industry.

Chelmico’s Eesy Breezy video used Japan’s Crescent’s 4DViews Holosys studio which captures: “realistic expressions such as the unevenness of the face and the wrinkles of clothes [using] 32 dedicated cameras… and can be output as mesh structures with optimised textures both in real time and offline.”

The result is that a performance can be captured from every angle, watched from any location using a virtual camera and repurposed to suit any media. Nonetheless, when it came to a live performance, the group used Canon’s Volumetric Video Studio in Kawasaki which has broadcast numerous events ranging from traditional Kabuki to baseball.

In Australia, facilities like this are more limited but do exist and include a collaboration between Imagine Room and Swinburne university that’s been designed for, “research into virtual world content production, digital human double [creation], CX, UI/UX and postproduction workflows, as well as human biometrics, performance and assisted technology design in the disability sector.”

While these use-cases might attract interest from studio builders, they offer less opportunities to AV professionals in the entertainment consumption space. At the recent Tech Leaders conference, Paul Devlin of AWS Sports Enterprise explained that the sports industry recognised how young people have shorter attention spans and are often more interested in video games than traditional sports.

Plus, this is what’s driving the rapid development of metaverse-related technologies that will help to engage them while simultaneously creating new revenue streams and opportunities for modern fans.

One of these is ‘optical tracking’ which can now record precise movement without requiring wearables. Devlin told us that, within five-to-ten years, you’ll be able to watch live, life-like, ‘virtual’ matches from anywhere in virtual stands or on the pitch itself. The key technology driver will be 5G-based streaming from the cameras, via the cloud to the couch.

Devlin also threw out some additional mouth-watering developments. Firstly, original TV broadcasts of historical matches are being transferred into virtual environments (with Unreal-engine generated stadia) meaning sports fans could watch their favourite team lift the cup… back in 1972!

He added that old tobacco ads could be removed and replaced with personalised alternatives. He also said that moves were afoot for AR Marvel Superheroes to guide kids to their seats at Marvel Stadium. Furthermore, AR and ABBA Voyage-like technologies will soon enable live, virtual sport to be displayed upon one’s coffee table. The latter may not be too far away with Unity Metacast recently working with UFC to present actual fights, live in a virtual environment.

Education potentially paving the way

Educational gaming has arguably pushed the metaverse to where it is today (and is where the current AV-related potential lies). Minecraft is a prime example. The popular ‘game’ started out as digital Lego but it’s now a fully-fledged metaverse that has evolved in numerous directions including Minecraft Education Edition whose built-in lesson plans are used in many schools to teach STEM subjects. It enables students to combine maths, engineering, science and computing skills to perform tasks like building a dam or removing pollution from a town.

Meanwhile, Roblox has 300 million users, many of whom are very young children. It bears a resemblance to Minecraft, but it’s more of a platform that encourages third-party coders to make games within it (often which require the purchasing of in-game currency, Robux). Roblox puts a major focus on older students learning to build virtual environments and experiences and provides educators with numerous, captivating and immersive lessons that very young students immediately know how to interact with.

ViewSonic Education has invested in the Metaverse, with its education platform universe offering an immersive digital learning experience.

Elsewhere, ViewSonic recently launched its Universe virtual classroom and campus for educators. While it might resemble a dystopian future that combines virtual students and video-based participants who can interact through emojis, it’s arguably made more of a splash than HTC’s Vive Events virtual meeting equivalent that’s failed to catch on despite lockdown.

Beyond STEM, the latest iteration of Bilby Labs’ Virtual Songlines First Nations virtual heritage project is illustrative. Built using Unity3D technology, it depicts Australia in its pre-colonised state and features an accurate, immersive 3D environment including the interactions of the people, flora, fauna and climate of the time.

It can be accessed on any medium, including AR and VR, and has attracted interest from almost 2000 teachers seeking to leverage it as a gamified cultural and historical lesson plan. There are many other, topics that use similar technology available via Australia’s EDUmetaverse website.

What will it take to make the metaverse mainstream?

It feels like VR and AR mass adoption have been just around the corner for a decade. Perhaps smartphones with depth-scanning cameras will boost adoption or virtual presenters in TikTok?

While this might sound trivial and irrelevant to AV professionals, is it not fair to say there’s a link between increased amateur adoption and professional demand?

In the meantime, there are plenty of AV-related metaverse opportunities in markets like tourism, real estate, health, art and education plus, the exciting roadmap that’s already manifesting in the world of sports and entertainment. It’s not all quite aligned just yet, but we’re finally getting there.