in development | status: WORKING PROTOTYPE
An extended reality interactive art piece in which the user plays on a electronic drum kit while watching a real time sound visualization on virtual reality. designed to explore a possible augmentation of how we play musical instruments, the experience promotes a feeling of synesthesia as the user can literally see the influence of the sound being played. Unlike any experience possible in the physical world, the user change its environment with the power of his or her music.
The electronic drum kit is tracked in virtual reality and the midi signal is processed in real time generating a responsive sound visualization, allowing the user to freely play the drums while immersed on this virtual experience.
Music, sensations and Interactivity
Even with unknown origins, it is a consensus that Playing music has become a fundamental constituent to humankind. we can use music to express feelings, tell stories, interact with each other, and many more. Besides, playing music is INTRINSICALLY linked to our bodies, with a few examples being the use of our voice, hand and arm movements and the overall rhythm sensations naturally processed in our brains.
in a journey for achieving immersion, we could argue that not only aspects such as photorealistic graphics and low latency visual FEEDBACK are SUFFICIENT, but a combination of several factors that promote a sense of presence and embodiment, such as immersive audio, full body tracking, naturalistic responses, etc. An extremely natural human activity such as playing a musical instrument may pose as an interesting way to expand virtual reality’s frontiers on achieving this immersion.
You put on the virtual reality glasses to find yourself on a completely different - but believable - reality. As you naturally begin to interact with the drum kit, and see on the environment the influence of your music, what exactly happens to our consciousness? We think These NATURALISTIC interactions JUXTAPOSED with virtual environments are a great source for TRULY immersive, fun and memorable experiences. We hope our findings will help us understand and always improve future virtual reality equipment and experiences.
Play the drums in a new way with virtual reality. While you are playing, a real time sound visualization is exhibited on virtual reality. Several parameters of your play style are processed to provide a responsive feedback, You see the sound of when you are playing.
A custom experience every time you plug in. all raw Midi input is processed in its full spectrum, considering aspects such as hit intensity, variety, tempo, and other values. Being an interactive art installation, Most feedback is inherently determined by each unique experience.
Try a whole new feeling as you change your environment with the power of your music. While impossible in the physical world, in virtual reality you can really dive your consciousness into your own songs on an truly immersive and bodily experience.
THE PROJECT is being deVELOPED FOR the htc vive USING UNREAL ENGINE 4.20. We started the project when searching for a way to explore how the future of extended realities may be helped by sensorization. We hypothesized that Playing a musical instrument would be an effective way to truly embody a virtual experience, while keeping the user immersed with this synesthesic sound visualization - also present on one of our previous project, LS4D. We decided to quickly test this hypothesis by developing a quick prototype on which a midi input would be translated to a simple visual feedback. After SUCCESSFULLY testing it for the first time, we bought a electronic drum kit to start the production.
The Drum kit was connected to the computer via an usb 2.0 type-a cable and MIDI input from the electronic drum kit was processed in unreal engine USING THE blueprints based MIDI manager, as demonstrated by dan.reynolds IN THE UNREAL ENGINE FORUMS. The user’s midi input is processed and some patterns are detected based on a recent past (e.g.: all notes played in the last 5 seconds). To ensure RESPONSIVENESS Both raw midi input and these processed patterns are somehow expressed to the use.
Next steps of development
With the results gathered in our internal testing, we are optimistic on the main features that will be available on the main experience. Even with a simple visual feedback, playing our digital drums has already proven to be a fun experience.
Currently, the electronic drum kit is manually mapped into unreal engine and responsive hand tracking using htc vive trackers has not been tested yet. However, these are basic features planned to be tackled in the very next steps of development. Later progress will be directed to ensuring an interesting visual feedback to the sound visualization and overall polishing of the user experience.
Besides serving as an interesting technology experiment on mixed reality, Digital Drums will be initially released as an interactive art installation. AS the market for virtual reality arcades is rapidly growing in countries such as china (forecast to grow 13-fold according to iResearch consulting group) , united states and brazil, we can also prospect this market as it is not limited by the need for physical installations like the overall consumer market for virtual reality.