Immersive tech enhance research on hearing in complex settings
Carolina Signorelli
Technological innovation is revolutionizing the study of auditory perception, allowing researchers to examine how the brain adapts to sounds in complex and dynamic settings. The SAV Project uses the potential of Virtual reality and Gamification to recreate realistic sound environments, offering new perspectives for neuroscience research and improving auditory skills in individuals with hearing difficulties. In crowded places, such as restaurants or conference halls, the brain quickly adapts to surrounding acoustic characteristics to distinguish relevant sounds from background noise. This phenomenon, known as reverberation adaptation, is crucial for daily interactions and occurs automatically and often unconsciously. Despite environmental interferences, it enables individuals to isolate and understand a specific auditory signal, such as speech. However, this process remains insufficiently studied, particularly in cognition and interpreting auditory stimuli.
The SAV Project is funded under the Marie Skłodowska-Curie program and is coordinated by Pavol Jozef Šafárik University in Košice, Slovakia. It began on October 1, 2023, and will run until September 30, 2027, with an EU contribution of €496,800. The goal of the project is to explore the mechanisms underlying this phenomenon. In fact, it is already well established that high background noise levels impair speech comprehension, making cognitive processing more challenging, especially for individuals with mental or auditory impairments. However, current uncertainties persist regarding other aspects of the auditory experience – both emotional and subjective – such as the listener’s perceptual integrity, emotions, engagement, and practical elements, like the spatial distinction of sound sources.
The complexity of the auditory experience
This new approach allows researchers to assess the cognitive component of auditory perception and the emotional aspect, providing a more comprehensive understanding of the auditory experience.
Auditory neuroscience has historically struggled to fully capture the complexity of listening experiences, as real-world settings constrained traditional experimental methods and did not account for the listener’s active engagement. The SAV Project aims to overcome these limitations by developing innovative tools for studying auditory perception and its associated cognitive processes.
To achieve this, Virtual reality and Gamification technologies are integrated to create advanced models capable of accurately assessing auditory abilities and their underlying neural mechanisms in individuals with normal hearing and those with hearing impairments. Gamification has been a part of scientific research for several years and involves applying game mechanics to non-game contexts. This approach enhances experimental engagement, increasing participants’ motivation and improving learning effectiveness and perceptual adaptation. In auditory perception, traditional tests – often repetitive and unstimulating – are transformed into interactive experiences, allowing subjects to train their listening skills more naturally and realistically. Virtual reality (VR), on the other hand, enables the creation of immersive and controlled auditory environments where researchers can simulate various acoustic conditions, replicate standardized real-life situations, and analyze how the brain adapts to changes in reverberation, background noise, and the distance of sound sources. Specifically, the project involves a series of experimental studies aimed at understanding how to achieve a realistic perception of sound origin in virtual environments. Additionally, it analyzes the role of virtualization in identifying cognitive factors that influence auditory perception. Lastly, it explores the potential of gamification to enhance spatial perception and speech comprehension.
Out of the lab
While past research was limited to externally studying how attention shifted between sound sources in laboratory conditions, today, these innovations allow for the recreation of highly immersive and interactive scenarios, improving analysis methods and increasing research potential. The project results will be used to develop new modules for existing tools. They may also positively influence spatial and speech perception and auditory mechanisms, leading to discoveries applicable to clinical diagnoses, the design of advanced auditory devices, and the optimization of listening experiences in real, virtual, and augmented reality environments. Furthermore, training new researchers through dedicated workshops will ensure widespread adoption of the developed methodologies, contributing to the evolution of the field and the creation of increasingly effective tools for enhancing human auditory experiences.
Integrating virtual reality and gamification in auditory neuroscience represents a transformative approach to understanding and enhancing auditory perception. By moving beyond traditional methods, these technologies allow researchers to explore the cognitive mechanisms of sound adaptation and the emotional and perceptual factors influencing real world listening experiences.
The SAV Project’s findings could lead to groundbreaking applications in clinical audiology, assistive technology and immersive sound design, ultimately benefiting individuals with hearing impairments and those seeking optimized auditory environments. As these methods continue to evolve, they will shape the future of auditory research, offering innovative solutions for real-world challenges in hearing and communication.
