When Sony’s Lead System Architect Mark Cerny first revealed what the PlayStation 5 can do in a presentation released back in March, he dedicated a surprising amount of time to a feature he called “3D Audio”. Powered by a new hardware engine named Tempest, Cerny spoke – in rather complex terms – about ear shapes, positional audio, sound sources, and HRTF. It was clear that Sony is trying to do something impressive with video game audio. Exactly what that is, though, and how it will actually improve games, was far less obvious.With the PlayStation 5’s launch into the world, I decided to talk to people designing audio for the next generation of games in order to find out exactly how 3D audio could change the way we play. We began, though, with the fact that you’ve probably already played games that use it.
“I don’t know if you have played Resident Evil 2 remake over headphones, because [Capcom] have their own implementation of 3D audio,” Daan Hendriks, Lead Sound Designer at The Chinese Room, developer of Amnesia: A Machine for Pigs, tells me.
If you’ve played Resident Evil 2 using just a pair of regular stereo headphones, you’ll know that if Mr. X is stomping around in his massive boots on the floor above, you can tell he’s one storey higher thanks to the way the noise sounds. And that’s not because of the thuds being muffled or disguised to simulate distance; the noise quite literally ‘feels’ like it is coming from above. That’s 3D audio in action.
“The way I think about it is, without 3D audio, you don’t have a sense of height at all,” explains Jey Kazi, Senior Sound Designer at Sumo Digital, developer of PS5 exclusive Sackboy: A Big Adventure. “So, when you [hear] sounds in a [standard] game, you’re not going to be able to hear it above you or below you. Whereas with 3D audio, you are going to get that sense of space and directionality, which you wouldn’t get from traditional panning that we usually use in the games industry.”
“It’s like we have a sphere of sound that we can place sounds in,” adds Hendriks. “And obviously you as a listener, you’d be right in the middle of that sphere, basically.”
Currently, most games use an audio system that places sounds in a ‘circle’ around you. You can tell if the sound is coming from your left or right, or if they’re in front or behind you. Surround sound systems, such as 7.1 speaker arrangements, can help enhance that sensation. But 3D audio is about positioning individual sound-emitting objects – multiple hundreds of them, potentially – within a ‘sphere’, which provides an accurate sensation of direction, depth, and distance.
3D audio is vital for creating a sense of immersion in virtual reality games. As such, Sony’s PSVR headset comes with its own dedicated audio processor; a device that is essentially the precursor to Tempest Engine. But the PS5 can bring that sense of audio immersion to regular games, not just VR experiences. And – arguably most importantly – it can do this without need for any fancy gadgets in your console set-up. “It’s basically stereo, but it uses specialized filters to mimic the sensation of sound coming behind you,” Hendricks explains. Because 3D audio is stereo-based instead of channel-based (the system used in surround sound that uses an array of speakers to deliver varying audio tracks) all you need is a pair of standard stereo headphones to experience it.
“
But hold on – if Resident Evil 2 already has 3D audio, then why isn’t it already standard in all games? Well, according to Cerny, 3D audio is “computationally expensive”. Capcom’s solution is software-based, and so all the sound objects must be calculated by a console’s CPU – the chip that’s also dealing with thousands of other processes. Unless sound is particularly valuable to the game experience – like it is with Resident Evil – then that CPU bandwidth is better off being used for something else. That’s why PSVR has its own audio processor, so your PS4 doesn’t need to deal with these new, complicated audio demands on top of all of its regular tasks. The idea behind the Tempest Engine is exactly the same: with PS5, audio developers have a dedicated hardware component that doesn’t tax the CPU. Suddenly, there’s free bandwidth and lots of it.
“Having a dedicated resource means you’re not fighting and competing with [CPU processes],” explains Joe White, Senior Programmer at Sumo’s Nottingham studio. “And having a dedicated audio compressor means that the actual hardware is optimized for particular types of processing. It suits digital signal processing, like what you’d do to generate 3D audio. So the whole pipeline, in terms of simulating that certain aspect, is really a lot faster.”
“It means that we can be more creative,” says Kazi. “It makes things sound cooler. That’s the crux of it, really. It means that we can go into more detail, more layers, more plug-ins. So, making things sound like they’re more in the space.”
Accurate positional audio means that game designers will be able to truly enhance existing audio cues and increase a game’s sense of atmosphere. Cerny has mentioned the idea of rendering the sound of individual raindrops to enhance a scene, which would make for a more authentic storm, but it’s easy to imagine that applied to the clatter of individual brass cartridges ejecting from a minigun mounted to a helicopter flying above you in Call of Duty, or to the many voices of a crowd in Hitman.
But beyond such atmospheric examples, evolved audio opens the door for both new gameplay mechanics and – more importantly – new accessibility innovations within games, too. “Blind people play games purely on the audio cues,” says White. “[3D audio is] improving the ability for them to do that. That’s pretty cool.”
“I feel, especially after games like The Last of Us 2, from an industry perspective, we’re all trying to make sure everyone can play our games,” White says. He also recognizes that while 3D audio does make purely sound-based mechanics possible, games may never have them because it excludes players who are deaf or hard of hearing. AAA game development’s long-overdue advancements in accessibility measures mean that sound designers must now think as much about those who can’t hear their games as those who can.
It’s important to note that while Sony is making a big deal of its Tempest Engine – even to the point of producing an official set of PS5 3D headphones – PlayStation 5 is not the only console capable of hardware-rendered 3D audio. Xbox Series X|S also has a dedicated audio processing chip that powers Microsoft’s obtusely-named Windows Sonic for Headphones system. Xboxes also provide the option of purchasing DTS Headphone:X or Dolby Atmos systems to use instead of Windows Sonic, but all three are ‘spatial audio’ solutions akin to Sony’s Tempest technology. Simply put: no matter what your next-gen console of choice is, location-based audio is the future of how you’ll hear your games.
“
“Over the next few years we will see some amazing things happen,” predicts Kazi, musing on how a dedicated processor is a dream come true for audio designers. “I think just more detail, more fidelity, because we are definitely [currently] constrained by things like sample rate and bit rate and just the size of sounds and how they’re compressed.”
For White, this is the next step in providing audio design the kind of attention and advancement that the games industry has consistently given to graphical horsepower. “It’s interesting to compare graphics and audio, because you think of audio being the splintered stepchild of the games industry, right?” he notes. “It’s about 20 years backwards from the latest ray tracing simulations that are coming out now. 3D audio’s just this step towards improving the simulation.”
It’s difficult to talk about audio, partially because the language of games has so heavily skewed towards graphics over the industry’s lifetimes. We can show you improved visuals in a video or screenshot, but how exactly do you describe the position or sensation of sound? Because of that, the leaps this generation is making in sound may seem difficult to comprehend, or haven’t appeared as important as all the talk of teraflops or ray tracing. Hopefully, though, when you put a headset on and boot up a next-gen game for the first time, you will find that 3D audio makes a difference. And if that advancement really is obvious to players, here’s hoping that the “20 year” gap between audio and graphics is about to close at a speedier pace.
For more PlayStation next-gen coverage, make sure to check out our PS5 review as well as our thoughts on the Sony Pulse 3D Wireless Headset and PS5 DualSense controller.
Matt Purslow is IGN’s UK News and Entertainment Writer. News Source
About Author
