YoHand Sebastian Bach

A gesture control musical instrument and synthesizer designed for musicians using Leap Motion controller

YoHand is a digital instrument for live performance runs on Linux and OSX that uses Leap Motion controller to provide an intuitive way to manipulate sound and add effects. It was designed in a way that players with or without experiences are able to pick up easily without spending hundreds of hours learning and practicing.


Project type

Group Project with Nick Jones, Chad Price, Dylan Swiggett, Jack Conger, and Autumn Johnson.

Project Aspects

Sketching, Interface design, Interaction Design, User Testing, Coding.


                                                                  Discussing the features of YoHand

                                                                 Discussing the features of YoHand

The Goal

Our aim was to create an entirely new instrument that can be easily picked up and play, but with a reasonably high skill ceiling and level of expressiveness. Many Leap Motion projects closely resemble existing instruments, but our goal was to create a new and easy-to-use control scheme. 

In the beginning, we did a quick competitive analysis of current Leap Motion applications. After a few tests with the applications, we found that although most of the applications offered a variety of functionalities, it requires a lot of practices, and the 3D interface makes it difficult for the user to control the notes and effects. Based on this findings, we determined that our design should: 

                                                          Final software structure of YoHand

  • Display the information in 2D only so the user does not need to spend time learning about 3D interface.
  • Give a clear indication of where user's hands and fingers are, and the user should be able to tell which notes they are playing.
  • Allow the user to jump between instruments and add effects in real-time so that it could be used for live performance.

After we had reached an agreement of what we were planning to do, we broke down the work into three parts: User interface and experience design, music theory algorithm design with PureData, and code implementation with Qt.



Interface Design

Based on the original concept, I designed two initial drafts of the interface and presented them to all group members. 
Although the design seemed to work in the first place, after evaluated some few other Leap Motion applications, I found that some of the input methods I designed might not be suitable for our application. For instance, slide bars and dials required a precise control that was beyond the capability of Leap Motion Controller. Since in-air gestures do not have any haptic feedback, it will be difficult for users to interact with small objects. 
So I went back and had another discussion with the team about what kinds of information should be displayed and what will be the ideal way for users to interact with the application. In the end, we decided that most of the control should be done by using keyboard instead of gestures, so the user could focus on performing the music without triggering any other functions by accident.
The refined interface only contents information of palm and finger positions, and I used half circle to represent the orientation of the hand. For instrument selection, key incrementation, and tempo adjustment, they were all controlled with the keyboard, but the information will be displayed on the screen. This design gave user's hands a larger space to perform the music.

                                                    The instruction manual with all available gestures


After the released of the initial version, I conducted usability tests to observe how the user interact with the application. Four insights I got from the tests were

  1. A tutorial might be essential. If YoHand is going to be released to the web, we might need a tutorial to help the user learns how it works, especially if we are going to implement more gesture control functions.
  2. Currently, the window size is too small. We might need to make it full screen, so users will have more space to move their hands.
  3. We might need to decrease the sensitivity of note switching since users could barely stabilize their hands at the same location. It will be better if we give user's hands a larger space to move before jumping to another note.
  4. The metronome displayed on the interface was difficult to understand and might need to be redesigned. 


control scheme

Apart from the interface, we also spent time investigating what kind of gestures should be used. Our goal was to create an instrument that could be played in an intuitive way, so we need to lower the learning curve and reduce user's cognitive workload. The final design of gestures was based on the frequent of use and intuitiveness. We had removed some of the gestures that were either difficult to perform, hard to memorize, or might interfere with other gestures. 


Logo and Icon

Logos to designers is like personas to UX Researchers.  Logos and icons should demonstrate the core value of the application as well as to convey the design intention of the program to the user. So I designed several icons and logos based on our core value and presented them to the team. The final version of the logo and icon were later shown on the poster during the open house demo session.

Final interface Design

Open House Demo

YoHand was presented during the open house poster session at the University of Washington. We received an overwhelmingly positive feedback. During our session, users were able to test twelve unique timbres and control key signature, BPM, melodic direction, rhythm, and arpeggiation. We created a working, user-friendly program that demoed well even with people who had never heard of our project. Within a minute or two of explanation, users could successfully play music with YoHand, discover the controls naturally as they played. 

Final thoughts

Although we had received lots on positive feedback, there is still lots of things that need to be improved. Since we had integrated many functions into the application, the learning curve was significantly higher than our expectation. More usability tests are required to create the best user experience. Also, the program currently could only be used on Linux or OSX, and it requires the user to make the file by themselves which is difficult for people without experience. 

Overall, we have demonstrated that it is possible to create an entirely new instrument based on musical progressions and effects rather than the physical wood and metal of existing instruments, and people are able to pick up and play in a short period time. We hope that in the future, we could keep refining YoHand and finally turn it into an instrument that can be played on the stage.