3d spatialization

[Tetsujinfr]

hi, do you have any plan to support 3d spatialization at some point?

[mgeier]

The latest development version (in the master branch) already has some 3D features, but not all parts of the SSR support 3D yet.

For now, ssr-binaural is the only renderer that supports 3D, and only when using SOFA files (see #165).

The Qt-based GUI, however, does only show a 2D top view (and there are currently no plans to extend that).

There is a new (and very experimental) 3D browser-based GUI, see https://ssr.readthedocs.io/en/latest/network-websocket.html.

The new network interfaces (WebSocket and FUDI) fully support 3D.

There are concrete plans to extend the scene description format to 3D, see #155.

Other than that, I don't know of any concrete plans, but it would of course be great to have full 3D support!

Here are some slides I made recently about the latest SSR developments: https://github.com/mgeier/fk2019

[Tetsujinfr]

Thanks for your great reply and all the information. I will look at the code and see if I can hook it up with some 3D renderer to visualize the 3D spatialized sound source.
Do you have some benchmarks regarding performance of your binaural model w.r.t compute load? e.g. can it run on a small device lie a smartphone or does it require some powerful desktop cpu?

[mgeier]

I will look at the code and see if I can hook it up with some 3D renderer to visualize the 3D spatialized sound source.

You should check out the above-mentioned browser GUI. Even though it's very bare-bones, it can already visualize 3D sources.

Feel free to make improvements to the browser GUI, I'm looking forward to your contributions!

Do you have some benchmarks regarding performance of your binaural model w.r.t compute load? e.g. can it run on a small device lie a smartphone or does it require some powerful desktop cpu?

There are no "official" benchmarks, but I'd say it's quite fast.
There isn't really a "binaural model", there are only impulse responses which are convolved with the input signal(s).
Therefore, the performance mostly depends on the length of the impulse responses and the number of sources.

IOW, if the impulse responses are not exceedingly long and you only have a few sources, it definitely would run on a smartphone (if you could somehow manage to compile the SSR on it, which I guess won't work).

On a desktop CPU you can typically render hundreds of sources at the same time.

Anyway, you should just try it with your impulse responses on your target CPU, that's the only thing that counts.
And please let us know about the results!

[janhenhan]

Hey Matthias, just double-checking my understanding: SSR's binaural renderer is now supporting 3D positioned sources that can be visualized/modified with the 3D GUI? You're not supporting Yaw/Pitch/Roll 3D-Headtracking yet, right? I got Ben's headtracker ("Supperware") yesterday and for the first time feel confident enough in the tracking capabilities of a low-cost tracker, such that this feature would actually become interesting. Let me know in case I'm wrong and I will update my SSR right away ;)

[mgeier]

SSR's binaural renderer is now supporting 3D positioned sources that can be visualized/modified with the 3D GUI?

Correct. The master version does. And there will be a new release quite soon, I hope.

The 3D visualization and interaction is quite bare-bones, but it works.

If you want to improve it, that would be great!

You're not supporting Yaw/Pitch/Roll 3D-Headtracking yet, right?

Correct.

The general infrastructure is in place, but the individual tracker classes haven't yet been updated to 3D.

Would you like to fix this?

Related: #140, #146.

[mgeier]

PS: if you manage to create network messages from your tracker that are compatible with one of the network interfaces (https://ssr.readthedocs.io/en/latest/network.html), you could use that for 3D tracking, because the network interfaces already support 3D orientations.

[janhenhan]

Awesome - your P.S. would have been my approach anyway! That's great news then, I'll look into it.