3D Finger Tracking and Recognition Image Processing for Reail-time Music Playing with Depth Sensors

Enkhtogtokh Togootogtokh* Timothy K. Shih* W.G.C.W. Kumara* Shih-Jung Wu** Shih-Wei Sun# Hon-Hang Chang*

*MINE Lab, Dept. of CSIE, National Central University, Taiwan
#Taipei National University of the Arts
**Department of Innovative Information and Technology, Tamkang University, Taiwan (R. O. C.)


In this research, we propose a state-of-the-art 3D finger gesture tracking and recognition method. We use the depth sensors for both hands in real time music playing. In line with the development of 3D depth cameras, we implemented a set of 3D gesture-based instruments, such as Virtual Cello and Virtual Piano, which need precise finger tracking in 3D space. For hands tracking, model-based tracking for left hand and appearance-based tracking for right hand techniques are proposed. To detect finger gestures, our approaches consist number of systematic steps as reducing noise in depth map and geometrical processing for Virtual Cello. For Virtual Piano, we introduce the Neural Network (NN) method to detect special hand gestures. It has Multilayer Perceptron (MLP) structure with back propagation training. Literature has few examples using touch screen as medium, with fixed-coordinates, and 2D-gestures to control MIDI input. The end users should no longer carry anything on their hands. We use Senz3D and Leap Motion due to a few technical benefits. Senz3D and Leap Motion use a closer distance to hands, thus detailed finger gestures can be precisely identified. In the past years, we announced a set of virtual musical instruments and the MINE Virtual Band. Our research work is tested on lab environment and professional theatrical stage.

Keywords: 3D Finger Gesture; 3D Finger Tracking; 3D Finger Recognition; Gesture Recognition; Neural Network; Senz3D; Leap Motion; Virtual Musical Instruments

The design and implementation


We proposed a state-of-the-art 3D multiple gesture tracking and recognition application on two virtual musical instruments namely, Virtual Cello and Virtual Piano with RGB-D regular sensors. Our approach tested on special and professional environment as on theatre stage in real time performance. We presented one of the new modern trends in multimedia interactive virtual music introducing with modern technology advances. Our approach is the setting up a competitive state of the art performance of gesture recognition of real time multiple objects. We used Senz3D for the Virtual Cello and Leap Motion for the Virtual Piano. Senz3D and Leap Motion are better in capturing objects in near range in comparison to Kinect. Since we use the hand and finger gestures in playing the virtual instruments we need to track fingers in the near range. The MINE Virtual Band is comprised of several virtual musical instruments including Virtual Cello, Virtual Piano, Drum, Spider King, and Bass Guitar. Demonstration and live performance video are available in http://video.minelab.tw/DETS/VMIB/. We plan to experiment the virtual instruments with professional music players to compare with the real instruments, to improve the tracking speed for fast songs, and to develop a learning system for virtual musical instruments. We hope the virtual instruments can be a very good alternative to the real musical instrument in the future.


   Copyright 2016 - Timothy K. Shih and MINE Lab., National Central University, Taiwan.