In this project, I created a 3D world using Max 7. User can wear oculus rift and connect it to the 3D world. This project aims to spatialize audio so that the user's position and head orientations are reflected in what user hears through headphones.
Welcome to the Minions' world
Using Max's jitter and Graham Wakefield's oculus rift, I created a 3D world with 4 yellow objects ("Minions") where a user can navigate through using arrow keys (and head orientation when connected to oculus rift).
The baby step to designing sound in 3D space is to measure the distance between the user and the sound source. The closer the minions are, the louder they would be. The audio signal is multiplied by the reciprocal of the distance.
The next step would be panning. Users within the same distance from a sound source will hear different amount of sounds in each ear depending on their position.
Users positioned at same distance from a sound source will hear differently from each ear. Assuming same head orientation of all, the amount of sound signal they will get from each ear depends on the differences of the x-coordinates between the sound source and the user. (x-z plane being the world's "ground").
Multiply by (1+x/d) to the left ear (channel 1) and by (1-x/d) to the right ear (channel 2). (x = (my x coordinate) - (source's x coordinate), d = distance between the user and the source's positions)
The messiest part of this project (as of now). User at a fixed position will experience change in audio signals as they turn their head (yaw). Change in positions of the two ears matters hear, as the distance from the source to the ears will change as the head turns.
The new position of the ears are calculated using quaternion. Any head orientation produces a set of quaternion values, and the new ears' positions can be acquired by performing quaternion multiplication.
How we move in the "World"
Using keys I can navigate the world. The upper left window represents the lens in the oculus rift. This creates the 3D effect in oculus
More into Quaternions
I imported quaternion values from my Max patch to excel to figure out the relationship between quaternions and rotation angle. Through extensive excel-ing, I figured out the relationship between the two, and thus was able to calculate the rotated position of two ears.
This video shows how it sounds in the 3D world I created. While distance and panning is reflected quite accurately, orientation is not. This is because I only adjusted amplitude of the signals when I dealt with orientation. The next goal is to take delay into account, because the slight delay between our ears is how our ears actually locate sound.
Special thanks to professor Bradford Garton for helping me start off with this project and providing incredibly intricate max patches!
Music Spatialization: DJing Minions
This project added the 'delay' factor from the previous project. The four minions in the virtual world play different instruments. Yellow minion is the vocal, pink one the bass, blue one the kick, and green one the synth.
'Minions Deep House', F minor, 120 BPM
Minions Deep House
Users can navigate through four different sound sources that play deep house. (FYI: 2-bar loop on repeat)
DJing Minions: Navigation
On September 22nd, I presented my demo 'DJing in Virtual Reality' at NYCML16 (NYC Media Lab). Thanks for everyone who tried my demo and gave me feedbacks!