Musical, dramatic or installation performances artistically and aesthetically exploit the perception of space at the performance location. The spatial dimension of perception is usually lost in a reproduction. By creating a digital twin of the performance, this dimension can be preserved in detail using the methods of virtual and augmented reality. Our goal is to create digital twins of existing performance venues as a virtual infrastructure and to make them open access and publicly usable. The optimal integration of audio-visual components, their simplification and modelling for an efficient, interactive presentation is itself a research task. The virtual acoustics methods established at the research site are combined with machine learning and supplemented by the visual component.

Project goals

This project aims to create digital twins of the highest possible quality as virtual infrastructure from existing performance venues, to make them available open access and thus to make them publicly usable. Examples of applications for this include research into innovative stage design concepts, performance practices of early and new music, as well as contemporary sound installations, which can be recorded, manipulated and played back through the research infrastructure in a way that can be walked through and relived. Another application is preparing for concerts or audition situations in virtual reality without having to physically enter the future performance location. Another goal is to increase the reproducibility of psychoacoustic experiments that require special performance locations and infrastructure, as the digital twins enable repetition at any location.

Specifically, at the end of the term, the workflow for creating the digital twins should be established and prototyped, practically implemented using use cases, and published open access.

Work packages

Work package 1

Audio-visual measurement and recording

Work package 2

Computing/Modelling

Work package 3

Augmented playback for large audiences