Innovative Sound Generation for 3D Human Models

While creating realistic 3D human models for visual media like video games and movies has improved a lot, creating the sounds these models make has been mostly overlooked. This work introduces a new way to generate high-quality sounds that come from a human body, capturing everything from speech to footsteps.

We use 3D body positions along with Audio recorded from a head-mounted microphone to create a full sound environment. Our method allows for the accurate Rendering of sounds in any 3D Space, making it possible to hear sound as if a person is truly present.

To make this happen efficiently and quickly, we borrow ideas from graphic rendering techniques that use simple shapes, which we call "acoustic primitives." These primitives help us create sound representations that are much smaller and can produce sounds that feel closer to the body than previous methods.

#The Importance of Sound in 3D Models

Creating lifelike 3D humans is important, especially for applications in gaming and virtual reality (VR). Many modern tools, like MetaHumans and Codec Avatars, allow for stunning visual models. However, accompanying the visuals with matching sounds has not received nearly as much attention.

Accurate sound representation is vital for a believable 3D experience. When people see a virtual human, they expect to hear sounds that correspond with their movements or actions. Currently, the research in creating spatial sounds for these virtual humans is lacking.

In this work, we focus on two key requirements:

1. We need to be able to render sounds at any point in a 3D space produced by a virtual human.
2. The sound environment must be controllable, meaning it can be adjusted in real-time based on body movements and sounds emitted.

#Challenges in Rendering Sound

Previous methods typically used a single, complex representation of sound around a human body, making it difficult to capture sounds close to the body accurately. The past approaches also required a lot of computing power and were not able to provide real-time results.

To address these issues, we propose a new method using smaller sound components, or acoustic primitives. Each primitive is a small sphere attached to points on the human body. Instead of relying on one complicated model, we sum the sound produced by each primitive to generate an accurate sound environment. This method allows for easy modeling of sounds very close to the body.

#Advantages of Acoustic Primitives

1. Better Near-Field Rendering: Traditional methods would struggle to accurately create sounds close to the body. Our approach accommodates this by using many small sound primitives, allowing for realistic sound representation even at close distances.

2. Compact Sound Representation: Rather than using one complex model, we create simpler, smaller sound representations, which make the overall sound modeling much faster.

3. Efficient Sound Rendering: Our method can predict sound coefficients directly, avoiding traditional complex processes that slow down sound rendering. This means we can create sounds in real-time based on body movements and sounds picked up by the microphone.

#System Overview

We designed a system that uses both audio and body position information to create sound environments. This system consists of several parts working together to capture, process, and render sounds.

#Input Data

The system receives input from:

• Audio signals captured with a head-mounted microphone.
• 3D body poses that describe the position of joints on the human body.

The goal is to create sound representations in a specific 3D location based on this input.

#Processing Stages

1. Learning Acoustic Primitives: The first step is to capture the sound fields generated by the body using input data.
2. Rendering Audio with Primitives: Once the acoustic primitives are learned, we use them to generate sound waves at the desired locations.

#Feature Encoding

#Pose Encoding

The movements of the body provide crucial information about how sounds are produced in space. We encode these movements into a format that captures their temporal aspects. This helps us understand how the sound changes over time as the body moves.

#Audio Encoding

Since audio can come from various places on the body, while being recorded at the head, we consider this slight time delay when processing sounds. This allows us to create audio features that reflect the actual sound coming from the body.

#Feature Fusion

We merge the encoded audio and pose features into a single representation. This fusion allows our model to utilize both types of data effectively, improving the accuracy of the generated sounds.

#Sound Rendering Process

The sound rendering process involves calculating how each acoustic primitive contributes to the overall sound environment. Each primitive's location changes as the body moves, so we need to account for these changes accurately.

#Predicted Locations and Weights

We calculate the new locations of each primitive by adjusting for any learned offsets. Additionally, different primitives will have varying impacts on the final sound based on their importance at specific moments.

#Rendering the Sound Field

To create the sound field that a listener hears, we transform each primitive's position into a format suitable for rendering. Summing all the rendered sounds from each primitive allows us to produce the final sound field.

#Loss Function and Training

To train our model, we use a loss function that compares the generated audio signals against the actual ground truth audio. By optimizing this loss, we improve the model's performance in rendering accurate sounds.

#Evaluation Metrics

We measure the success of our sound rendering using:

• Signal-to-Distortion Ratio (SDR): This metric indicates the overall quality of the sound produced.
• Amplitude Error: This shows how closely the rendered sound matches the original in terms of energy distribution.
• Phase Error: This evaluates how accurately the timing of the sound waves aligns with the original sound.

#Experimental Results

Our model has shown comparable results to state-of-the-art models in terms of sound quality while being significantly faster. It is also capable of rendering sounds close to the body, which previous methods struggled to accomplish.

#Dataset Used

To validate our approach, we used a publicly available dataset capturing synchronized audio and visual data in controlled settings. This dataset is specifically designed for sound and body modeling research.

#Implementation Details

In our experimental setup, we utilized a specific sampling rate for audio and frame rate for body data. The model was trained using contemporary GPUs, allowing for efficient processing.

#Performance Comparison

When comparing our method with existing approaches, we found that our system performed similarly in sound quality but with a much faster processing speed. This means that our method is not only effective but also practical for real-time applications.

#Visualizing Sounds

We created visualizations to represent how different sounds were produced by the virtual body. These visualizations revealed that the system correctly matched sounds to their source locations.

#Future Directions

While our system shows promise, there is still room for improvement. Potential future developments might include:

• Reducing reliance on complex microphone setups to make it easier to collect sound data.
• Generalizing this approach to work with a wider variety of audio sources beyond just humans.

#Conclusion

Our work presents a method for creating sound environments directly from body movements and audio signals. By using acoustic primitives, we maintain sound quality while significantly improving speed, allowing for realistic audio experiences in 3D settings like virtual reality and video games.

This new approach offers a foundation that can pave the way for future advancements in sound rendering technology, making virtual environments richer and more immersive for users.