Real-Time 3D-Audio for Digital Cinema - 2020 3D Media

Real-Time 3D-Audio for Digital CinemaPau ArumíDavid GarcíaToni MateosAdan GarrigaJaume DuranyAcoustics 08 Paris, 3 rd July

S c e n e s h o o t i n g w i t h p r e - h e a r i n g●●Video: camera zoom to listener positionVideo: the final result

T h e r e a l - t i m e s y s t e m●Offline part:– IR database using ray-tracing. See J. Durany's poster– Video: ray-tracing tool●Real-time part:– IR database → partitioned convolution → crossfade of changingIR→decoding (5.1, binaural,...). Total latency 512 samples– Recently: we calculate IR on-the-fly (still some limitations)http://clam.iua.upf.edu

C o n v o l u t i o n a n d X f a d e

P a r t i t i o n e d C o n v o l u t i o n ( 1 )Actually, it is not so simple

●Torger and Farina, “ Real-Time Partitioned Convolution for Ambiophonics Surround Sound”

D e c o d i n g o p t i o n sBinaural(HRTF's)

O u r I m p u l s e R e s p o n s e D a t a b a s e●●●●B-Format (X,Y,Z,W) IR for each coupleemitter/listener pointGrid with 50 cm points distance => 2 GBBut only 4 static sources! And without ZFor a general use case: compute IR's on-the-fly

T h e d e v i l i s i n t h e d e t a i l s !●Switching IR's + IFFT overlapp and addBIG PROBLEM!! (click)

S o l u t i o n : S e l e c t - o r - X F a d e

S o l u t i o n d e t a i l : S e l e c t - o r - X F a d e●●●Two pipes: Current IR and Previous IRWhen IR changes the convolved frame is invalidBut only in one pipe => select or XFade

H i g h - r e s o l u t i o n v e r s i o n●●In-situ pre-hearing is all very nice. But we also needmore control in post-productionOffline application with GUI, orchestrating the CLAMnetworks– Allows many sources, B-Format editing of sources– Complex and changing 3D models, source directionality, useof Direct Sound●Video: virtual party

C o n c l u s i o n s●●●The technology already proved useful for digitalcinema productionsWe are working on high-quality IR's in real-time.Almost there.Towards new post-production tools for the mediaindustry, with (3D positioning)– Positions and room metadata as a central feature.

Thanks!