Researchers develop mixed reality platform for performing artists

Researchers from the School of Engineering and Digital Arts, School of Computing, and the School of Arts have developed an online platform to bring performing artists together remotely to rehearse and create mixed reality improvisational performance.

The platform, called 'Virtual Director', provides a shared virtual 3D environment to connect performers who cannot rehearse face-to-face, as experienced by many during Covid-19 pandemic restrictions.

Through the development of the platform, the researchers explored how immersive online systems can benefit performance artists remotely. They found that performers seeing themselves as if performing in the same location had a restored sense of connection between one another, which was typically lost when performing via standard video conferencing tools. Furthermore, study participants reported that performing with Virtual Director turned online 'improv' into a distinct kind of artwork with new creative possibilities that traditional stages could not offer.

This research, led by Boyd Branch, a PhD student at the School of Engineering and Digital Arts (EDA), alongside Dr Christos Efstratiou (Kent's School of Computing), Professor Paul Allain (Kent's School of Arts) and Dr Rocio von Jungenfeld (EDA), is the first of its kind to examine how tele-immersion (visually co-locating remote performers) affects creativity, feelings of social connection, and the ability to enter the cognitive 'flow' states required for improvisational theatre.

Boyd Branch said: 'Virtual Director was developed with a modular design that allows performing arts groups to adapt the system to meet the unique demands of different artists. Testing it amongst professional improvisers from around Europe and the U.S. was valuable to understanding the role tele-immersion can play in the future of performing arts, beyond the Covid-19 pandemic. We are now working to make the platform available to the public where we plan to further explore how tele-immersion might affect other collaborative experiences with remote participants.'

Dr Efstratiou said: 'The Virtual Director platform allows a user to turn any video conference call into a mixed reality cinematic experience where members of the conference call can be arranged into 3D scenes as if they were performing on the same stage. The software removes the backgrounds of participants and arranges each performer into a selectable grid where they can be placed inside virtual cars, boats, and planes, or as if sitting in a cafe, inside a medieval castle or any kind of 3D or 2D scene created with popular CAD programs.'

Professor Allain, a world-leading expert on theatre and performance, said: 'Boyd's platform and this research offer something that goes well beyond video communications like Zoom or Teams, enabling creative, imaginative possibilities in virtual playgrounds that will make you want to stay on screen. Virtual Director is fun but will also help develop new modes of interaction and professional arts practice as we emerge from the pandemic.'

Virtual Director has already garnered the attention of the performing arts community. Using the technology, improvisational theatre troupe Improbotics won the Paris Fringe Festival 'Ballsy Award', a four star review by Binge Fringe, and the 'Most Innovative Show Award' in 2020 for their novel use of the platform. Piotr Mirowski and Kory Mathewson, co-founders of Improbotics, have continued their partnership with the Kent research team to develop new AI driven features for Virtual Director including real time language translation for tele-immersed improvisers.

Tele-Immersive Improv: Effects of Immersive Visualisations on Rehearsing and Performing Theatre Online' is published in the CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems and was presented at the Conference. Doi: 10.1145/3411764.3445310