Agora Surfs the Metaverse Wave with 3D Video and Audio SDKs – The New Stack
Although there is no standard definition for “the metaverse”, a series of startups have emerged to offer specialized services for the emerging 3D Internet. Such a company is Agora, which offers “real-time live streaming and communication features” for interactive applications. Its functionality covers a variety of platforms, but is mainly used currently for 3D and Unity web applications.
To find out how Agora fits into the metaverse trend, and what it offers from a development perspective, I spoke to Ben WeekesAgora’s main software engineer.
In one of the demos Weekes showed me on our Zoom call, he and four other people sat down at a virtual table. A photorealistic avatar represented each participant (top half only) and the user could take a 360 degree tour of the virtual room. Developers can integrate such functionality into their application, such as a conferencing application or a game, using Agora’s SDKs and APIs.
However, it’s not necessarily Agora’s SDKs that set it apart from similar development platforms – or even developers building the same functionality using native web coding or Unity. What excites Weekes most is Agora’s global network, SD-RTN™ – which stands for “software-defined real-time network”. It’s similar to a CDN (content delivery network) from Fastly or Cloudflare.
“Low latency video is what we’re looking for,” he said. “We have servers in every Tier 2 ISP in the world.” Currently, Agora does “something like 16 billion minutes per month” of online video.
Metaverse Use Cases
Agora doesn’t exaggerate in promoting itself as a metaverse company – the word is only used once on its homepage. However, the company’s PR representative presented this interview to me as a discussion of interoperability in the metaverse, so I asked Weekes how Agora defines “metaverse” and what the company offers developers in this domain ?
He replied that a metaverse is “a 3D environment that can be rendered either with something like Unity, Unreal, or on the web.” He added that Agora has SDKs “for all three of these platforms, allowing you to put real-time voice and video into 3D worlds and games.”
One of the popular use cases for the metaverse among Agora customers is real-time video streaming in a 3D world (see images above). The company is also exploring 360-degree videos shown on the Agora network, such as 360-degree coverage of a boxing match, so you can switch views while watching it. Weekes admitted that the latter is “a little futuristic at the moment, given that you usually need more than 10 megabits per second”, but he thinks the advent of 5G will soon make it a reality.
Another popular use case is spatial audio, which Weekes described as “when you’re in a group and you can talk to each other, and as you get closer, it gets louder. […] and if your headset supports it, you can get a sense of 3D if people are behind you, in front of you, etc.
Agora also offers SDKs for creating augmented reality (AR) experiences, such as an example using Apple reality kit framework for iOS. It’s a collaborative AR app “where you can create a scene with anyone in the world, seeing each other’s live updates.”
Unity vs. Web
As for the 3D platforms that Agora’s customers rely on, according to Weekes, it’s mostly Unity or the web at the moment. There is a lot of discussion among their customers about whether to build on Unity or on the web, but he prefers the latter.
“I’m a big web advocate, to be honest, but I’m aware that there are certain operations where Unity is more optimal than WebGL at the moment,” he said. “I always argue that Google and Meta etc. push the browser, [so] there is no reason for him to stay behind. And I’ve spoken to a number of our web-first customers, because it’s much easier to share a link to a prototype than to put a Unity app in the app store and manage all the iterations around that.
He added that the web also makes it easier for users.
“You don’t need to install an app and then sync. So I can share a link to a session, and you in your Oculus [headset] it would suffice to click on it in your messages, and you would be directly in it.
So what about interoperability?
Since the interview was supposed to be about the (still unresolved) issue of interoperability in the emerging 3D Internet, I asked Weekes for his take on it. Does he think the metaverse will be web-based or will it evolve into a separate internet platform?
“I think much like on our iPhones today, you have a mix of web and apps – it will be the same thing. If you’re using an Oculus headset, you’ll see there’s a browser there- bottom, making it easy to click a link and join different WebVR worlds.
Weekes refers to the Oculus browser, which Meta describes as “the built-in browser for Oculus Quest devices”. He also pointed out that it is based on Google’s open-source Chromium rendering engine.
“I think they [Meta] benefit from all the hardware acceleration work done by Google in video encoding and decoding,” said Weekes. “It certainly works well with the demo apps I’ve created, and I’ve mostly tested on it.”
From a developer perspective, Weekes said Agora is completely cross-platform. “It’s just integrating the SDK into your app, which isn’t a big job,” he said.
Although the code is different from platform to platform, according to Weekes “they all follow the same conventions of listening to users joining a channel and then once you get that notification you can subscribe and post to the channel” .
The Big Question of the Metaverse Platform
We’re at a point in the metaverse hype cycle where we’re waiting for big companies to define their architecture going forward. Meta, which started the Metaverse trend last year, said it would build an interoperable platform, but has yet to release technical details. Tim Sweeney of Epic Games, the company behind Unreal Engine, said in a recent interview that we [the industry] need to move our proprietary technology to open standards, file formats, and network protocols so that all of our systems can interoperate and we can all participate in the metaverse. But again, there are no specific details on how this might play out.
In the meantime, we have companies like Croquet and Decentralized creation of web-based metaverse platforms. On the augmented reality side, Niantic (the company behind Pokémon Go) recently announced increased web support. So there’s a lot of experimentation going on in the “open metaverse” space.
As for Agora, it currently covers all platform bases, with a particular focus on Unity and web applications. You could almost call it “Interoperable-as-a-Service”. While it makes sense to support all current metaverse platforms, a lot seems to depend on Agora’s CDN-like network, which is a very competitive field.