As the dragnet is cast for accessories that consolidate virtual reality into the de facto standard entertainment medium, one old but largely unfamiliar phrase has been doing the rounds: Binaural Audio. Its focus is on re-creating the exact experience your ears wold have if they were hearing the sounds for real. Given this approach and the way it differs from a fixed-speaker placements, it has the potential to revolutionise VR gaming (and even films viewed with headphones).
Surround sound has been *cough* around for a while and it’s clear that it’s an effective means of conveying spatial awareness to seated listeners. However, when it comes to VR, there is still a need for a more elegant solution. There’s nothing wrong with aggressively shoving air molecules around a room, but if the visuals are going on inside a screen strapped to your face, then all that sound leakage is clearly a waste of energy, as well as your neighbour’s good-will.
5.1 surround headsets were a (minor) revelation over stereo, but I’m sure most people would agree that they could be improved upon. While they do convey some sense of direction as to the sound’s source, it’s really just a bunch of tiny surround speaks really, really close to your head – the source direction is a bit ‘samey’. The bull-headed approach to improving the scope of the sound field seems be moving towards a set of cans that makes you look like Princess Leia. What this is missing is an appreciation of the incredible sophistication of the human ear and the brain’s relationship with it.
Re-introducing binaural audio: the natural way to hear. Two microphones + Two ears + Two earphones = Binaural Audio. That’s one whole microphone for each ear. “Ridonkulous!” You say, “Stereo headphones have been around for donkey years! They’re past it, you’re making an ass of me!”. This is a mistake. Don’t assume you know everything there is to know about the potential of stereo headphones… That is, don’t throw out your existing hardware just yet. Getting two separate recordings for each ear can be all the information your head needs to build a view of the world around it. If you are ‘faking it’ with game-generated sounds then impersonating the way those sounds arrive at each ear of your virtar individually is, if you think about it, exactly how things work in real life. Pointing 7-and-a-bit speakers at your head is not.
Binaural Sounds vs Binaural Recording
Recording binaural audio is really as simple as setting up two microphones either side of your head as you hear something and then playing them back over stereo earphones. The very subtly different sounds picked up in each microphone actually convey an enormous amount of information to your brain, which has spent ages (literally) practicing the art of interpreting acoustic nuances to build a spatial model of your environment. It can even read the minute delays (phase-shifting) of a particular set of frequencies in one ear as a reflection of the same source sound heard by the other, even going on to use that information to triangulate the position of the source object! WTF? For real?! Yep. Woah.. Go humans. (Who needs Daredevil?)
Generating binaural sound is like ray-tracing compared to the ‘rasterisation’ of surround sound. In 3D scene rendering ray-tracing is the computationally-expensive plotting of light-ray paths within a virtual environment as they hit 3D objects, including reflections and absorption. Rasterisation is the computational equivalent of slapping on a coat of digital paint, by comparison (this is a harsh analogy, but it helps to illustrate my point).
Now, all that phasing and delaying and stuff is easy when you have, y’know, real life to generate it for you. Generating it programmatically? Not so much. Not only is the sound that reaches each ear dependent on a lot of environmental factors, it also needs to be filtered, phased and delayed as if it were hitting the sides of a head that actually exists in the virtual world. Now, your brain does a lot of the heavy-lifting and as someone whose hearing isn’t all that, I know that the subconscious mind is capable of filling in a few blanks. Even so, faking that kind of thing isn’t easy and not many people feel up to the challenge. Here’s a couple who are:
One aptly-named game who is keen for this challenge is the aforementioned Technolust, from Iris VR (co-founded by a rock-star ex-GTA developer). It’s inclusion is largely down to the reaching of a stretch goal on Kickstarter (*VR-Gaming pats self on back*), so we’re really interested to see/hear how well this works – but setting hype-meter to cautious at this early stage. Look out for our (p)review in June/July.
Acoustic-modelling company TwoBigears have created a plugin for game devs that manages to reduce the CPU-hogging demands of binaural calculations to run in real-time with unlimited sources. It’s brand-new tech so we’re waiting to see who picks this up, but according to their blog it’s seen a massive amount of interest. Given the promise of Binaural, I really hope it can deliver – a mobile-friendly version would indicate that processing power isn’t too much of an issue. If only ray-tracing were this easy!