3D Finger Tracking and Recognition Image Processing for Reail-time Music Playing with Depth Sensors

Abstract

In this research, we propose a state-of-the-art 3D finger gesture tracking and recognition method. We use the depth sensors for both hands in real time music playing. In line with the development of 3D depth cameras, we implemented a set of 3D gesture-based instruments, such as Virtual Cello and Virtual Piano, which need precise finger tracking in 3D space. For hands tracking, model-based tracking for left hand and appearance-based tracking for right hand techniques are proposed. To detect finger gestures, our approaches consist number of systematic steps as reducing noise in depth map and geometrical processing for Virtual Cello. For Virtual Piano, we introduce the Neural Network (NN) method to detect special hand gestures. It has Multilayer Perceptron (MLP) structure with back propagation training. Literature has few examples using touch screen as medium, with fixed-coordinates, and 2D-gestures to control MIDI input. The end users should no longer carry anything on their hands. We use Senz3D and Leap Motion due to a few technical benefits. Senz3D and Leap Motion use a closer distance to hands, thus detailed finger gestures can be precisely identified. In the past years, we announced a set of virtual musical instruments and the MINE Virtual Band. Our research work is tested on lab environment and professional theatrical stage.

Keywords: 3D Finger Gesture; 3D Finger Tracking; 3D Finger Recognition; Gesture Recognition; Neural Network; Senz3D; Leap Motion; Virtual Musical Instruments

The design and implementation

Gesture Image Processing on Virtual Cello

The cello or violoncello is a bowed string instrument with 4 strings. It is a member of the violin family, which includes the violin and viola. The cello is used as a solo musical instrument and is the second-largest bowed string instrument in the modern symphony orchestra. We separate one string to 2 prats, means one string have 2 positions, as first position (near cello head) and second position (bottom). When player move hand to a specific location, the string color change to yellow and when system detect user touch the chord the finger color changes to red.
Presentation	Image




Gesture Image Processing on Virtual Piano

When user presses the key, key moves down, and sends a MIDI signal until user fingertip moves up. Users can stand or sit in front of the Leap Motion to use the virtual piano. We designed adjustment hand gestures to change position of keys, distance between keys, size of keys, viewing direction, timbre, pitch up/ down, etc.
Presentation	Image




Demonstration

Following video shows the demonstration of Virtual Cello and Virtual Piano with other virtual instruments including Virtual Drum and Spider Piano)




Other Virtual Instruments

Virtual Drum
Virtual Guitar
Virtual Piano

Demonstration of other virtual instruments

On-Stage Live Performance

2015

Song 1		Song 2		Song 3




2014

Song 1		Song 2		Song 3




2013

Song 1

Conclusions

We proposed a state-of-the-art 3D multiple gesture tracking and recognition application on two virtual musical instruments namely, Virtual Cello and Virtual Piano with RGB-D regular sensors. Our approach tested on special and professional environment as on theatre stage in real time performance. We presented one of the new modern trends in multimedia interactive virtual music introducing with modern technology advances. Our approach is the setting up a competitive state of the art performance of gesture recognition of real time multiple objects. We used Senz3D for the Virtual Cello and Leap Motion for the Virtual Piano. Senz3D and Leap Motion are better in capturing objects in near range in comparison to Kinect. Since we use the hand and finger gestures in playing the virtual instruments we need to track fingers in the near range. The MINE Virtual Band is comprised of several virtual musical instruments including Virtual Cello, Virtual Piano, Drum, Spider King, and Bass Guitar. Demonstration and live performance video are available in http://video.minelab.tw/DETS/VMIB/. We plan to experiment the virtual instruments with professional music players to compare with the real instruments, to improve the tracking speed for fast songs, and to develop a learning system for virtual musical instruments. We hope the virtual instruments can be a very good alternative to the real musical instrument in the future.

Download

Selected video are available to download below. For any questions or comments regarding this content please contact Timothy K. Shih.



Virtual Cello	Virtual Piano	Playing Demo	Live performance 2015
Download	Download	Download	Download