Designing Virtual Instruments by Real-Time Finger Tracking System

Abstract

Kinect, 3D sensing device from Microsoft invoke the Human Computer Interaction research evolution. This device which mainly aiming used for games, now implemented in many scope area including music. One is use to be a MIDI controller device named Virtual Instrument, which using natural gestures to produce synthetic sound similar with real musical instrument. From related works, found that the use of big joint such as hand, arm and leg are inconvenient and limit the way of playing virtual music instruments. Thus this study proposed fast and reliable finger tracking algorithm yet suitable for musical instrument playing. In addition, Virtual guitar application, which is one kind of Virtual instrument which represent real music instrument named guitar, developed as an implementation of proposed algorithm. Experimental result show that proposed method able to be used to play many variety of songs with the restriction of the speed of song is 120bpm. Furthermore proposed application is easy to use even for beginner that doesn’t have any experience in music or even playing real instrument.

Experimental Demo

The following Image and Video is the experimental demo of our Virtual Instruments including Virtual Drum, Virtual Guitar(previous method), Virtual Guitar(with new proposed method)and Spider King.

Virtual Guitar Proposed Method

This is the result of our Proposed method-in Virtual Guitar2 video, compare with our previous method implement in Virtual Guitar1 video, our proposed method could play various music genre. also its give player a real experience of playing guitar

also some previous Virtual instrument that we build and perform together in the Musical Concert Performance at NCU Taiwan

Live Performance Concert

The following Video is the result of our Virtual Instrument Live Performance Concert in National Central University Taiwan.

Conclusions

An efficient finger tracking algorithm was proposed in this paper, including the application for MIDI controller to play the chords on a virtual guitar. Our proposed method provides a fast algorithm for tracking fingertips in real-time, 22 to 30 frame per second and the accuracy performs well (90% for slow tempo song, 85% for high tempo song). Our application is simple and user-friendly, therefore it is easy for a beginners to play the music even if they never plays a guitar before. Moreover, many features, including music notes, chords, and playing position or tones of the instrument are implemented. The restriction in our finger tracking method is the speed of the song. Although it could not play high bpm songs, it is stable and already could be used in a professional music performance with the slightly slow songs. In the future, the use of Neural Network approach and the new version of Kinect Sensor V2 believed will solved the previous mentioned issues. In addition, extra features such as more different guitar modes, virtual character, and visualization of musical score, could be added to the current system in future. In addition, we could consider to improve the current interface using 3D models to match different kinds of guitars to get a realistic feeling of playing an instrument.

End Note

Due to copyright issues, additional testing data cannot be posted on this website, interested readers are welcome to contact the author for the locations of additional testing data.
Please cite it if you use any part of the images, data in this website. For any question or comments contact Timothy K. Shih or Noorkholis Luthfil Hakim.