VR in Audio Visualization

Tongyu Zhou, March 2022

Introduction

The immersion of VR spaces allows audio to be experienced in rich ways that were previously difficult to understand. These new modalities for interaction and embodiment can introduce unique experiences that span data visualization, telepresence, collaboration, and entertainment. Depending on each visualizer's purpose, different affordances were introduced to produce natural combinations of what is audible and visible. Here, I describe several categories of VR audio visualization software and research that suit these different purposes.

Experiencing Chromesthesia


Chromesthesia is a specific type of synesthesia where color is perceived in response to a particular stimulus, in this case, sound. As it is difficult for those who do not experience this phenomenon to understand its ramifications, many VR visualizers focus on creating discernible visuals to replicate its effect.

  • (depicted on the left) Benjamin Outram: colors are mapped to the frequency spectrum (specifically, according to a log scale since humans perceive equal pitch differences roughly based on log differences), users can use an "orbital technique" to have hands-free user observation and control

  • Chromesthesia Music Visualizer: features over 30 different visual effects that sync colors with the imported music

  • Tepljakov et. al's: introduces methods to detect source of sound in a 3D space around the user and transform the audio into visual shapes determined by spectral components

  • Synesthesia VR: simulates the experiences of 4 synesthetes in different audio experiences

Soundscape Research

A soundscape describes the acoustic environment as perceived by humans. Understanding soundscapes can reveal important information about the organisms and events that occur within that environment.

  • (depicted on the right) Eduard et. al: visualizes sound propagation in closed rooms using Phonon Tracing

While not strictly visualizing audio, other studies have also looked into how sound should be embedded in virtual spaces:

  • Sanchez et. al: introduces a methodology to assess the role of noise in sound planning in the audio-visual design of an urban space.

  • Jeon et. al: investigates different audio-visual effects in various urban soundscapes to understand how soundscape perception influences human satisfaction of urban environments

  • Kern et. al: effects of soundscape on understanding of presence

Entertainment

Unsurprisingly, the majority of these audio visualizers were actually created for entertainment -- to visualize the mood of a piece of music in a scene, to sync with a brush, or to serve as a game. Visualizers in this category boast bold visuals and flashy effects to attract users.

  • Beat Saber: VR music game where the user slashes incoming "beats" synced to the music to earn points

  • (depicted on the right) Vision: animates colors, lights, and shapes based on the audio, allows for user customization for the audio

  • Soundscape Universe: simulates a music festival in audio-reactive fantasy worlds

  • and many more!