Designing Virtual Instruments by Real-Time Finger Tracking System

Noorkholis Luthfil Hakim Mu-Hsen Hsu Shih-Wei Sun* Timothy K. Shih*
National Central University, Taiwan National Central University, Taiwan Taipei National University of Art, Taiwan National Central University, Taiwan


Kinect, 3D sensing device from Microsoft invoke the Human Computer Interaction research evolution. This device which mainly aiming used for games, now implemented in many scope area including music. One is use to be a MIDI controller device named Virtual Instrument, which using natural gestures to produce synthetic sound similar with real musical instrument. From related works, found that the use of big joint such as hand, arm and leg are inconvenient and limit the way of playing virtual music instruments. Thus this study proposed fast and reliable finger tracking algorithm yet suitable for musical instrument playing. In addition, Virtual guitar application, which is one kind of Virtual instrument which represent real music instrument named guitar, developed as an implementation of proposed algorithm. Experimental result show that proposed method able to be used to play many variety of songs with the restriction of the speed of song is 120bpm. Furthermore proposed application is easy to use even for beginner that doesn’t have any experience in music or even playing real instrument.

Experimental Demo

The following Image and Video is the experimental demo of our Virtual Instruments including Virtual Drum, Virtual Guitar(previous method), Virtual Guitar(with new proposed method)and Spider King.

Virtual Guitar Proposed Method

This is the result of our Proposed method-in Virtual Guitar2 video, compare with our previous method implement in Virtual Guitar1 video, our proposed method could play various music genre. also its give player a real experience of playing guitar

also some previous Virtual instrument that we build and perform together in the Musical Concert Performance at NCU Taiwan

Virtual Guitar2 First Video

Virtual Guitar2 Second Video

Virtual Guitar1

Virtual Drum

Spider King Instrument

(1) The program virtually draws a circle around the user. We divide the circumference into several intervals. Then users’ hands are used to control the key and volume. (2) When the hand is closer to the circumference, the volume becomes louder. The sound is also from MIDI signal as with drum and guitar, and we connect MIDI signals to different audio libraries.

Live Performance Concert

The following Video is the result of our Virtual Instrument Live Performance Concert in National Central University Taiwan.

Live Concert 1/2

Live Concert 2/2

Parent Teacher Meeting CSIE

CSIE Live Concert


An efficient finger tracking algorithm was proposed in this paper, including the application for MIDI controller to play the chords on a virtual guitar. Our proposed method provides a fast algorithm for tracking fingertips in real-time, 22 to 30 frame per second and the accuracy performs well (90% for slow tempo song, 85% for high tempo song). Our application is simple and user-friendly, therefore it is easy for a beginners to play the music even if they never plays a guitar before. Moreover, many features, including music notes, chords, and playing position or tones of the instrument are implemented. The restriction in our finger tracking method is the speed of the song. Although it could not play high bpm songs, it is stable and already could be used in a professional music performance with the slightly slow songs. In the future, the use of Neural Network approach and the new version of Kinect Sensor V2 believed will solved the previous mentioned issues. In addition, extra features such as more different guitar modes, virtual character, and visualization of musical score, could be added to the current system in future. In addition, we could consider to improve the current interface using 3D models to match different kinds of guitars to get a realistic feeling of playing an instrument.

End Note

Copyright 2015 - Timothy K. Shih and MINE Lab., National Central University