Handshape recognition

CS 585 HW 2

Yingqi Zhang


Problem Definition

Use the webcam to record different handshapes. write a program to recognize these handshape correctly and display it in real time.

Method and Implementation

The workflow of my program looks like this:


Object Detection

First, I use skin-color detection to extract objects like face, hands and anything in nude color

Next, I use opencv function findContour to track these objects and calculate their area using contourArea. Assuming that the contour of hand should have the largest area, I delete all other objects from the binary image, which is the image I will work with.

Feature Calculation

I use three features:

1. Compactness: I compute the compactness of the whole object.

2. Centroid: I calculate the centroid of the object and use it to splite the object into two parts (the top part and the bottom part.)

top part & bottom part

3. compactness in each part.


I use a simple decision tree to classify the handshape according to three features: the overall compactness, compactness of the top part and the compactness of the bottom part.

decision tree


The results are shown below:

Object Detection

Since I only use skin-color to extract hand, it is often the case that objects like face, arm and skin-color wall be reserved in the binary image. This method is not very robust.

success one failed one

Feature Calculation and Classification

I choose five handshapes for this recognition task. They are open palm, fist, thumb up, thumb down and victory.

In the experiment, the features of different hand shapes can be concluded as follow:

1. Open palm and vicotry have rather large overall compactness while the other shapes have lower compactness.

2. The difference between the compactness of the top part and bottom part is usually large because the top part contains fingers and the botttom part contains palm. So 'fist' is the only hand shape that has similar copactness of both parts.

3. The 'thumb up' shape has a larger top compactness while the 'thumb down' shape is the opposite.

Because these features vary a lot on different hand shape, the classification performance is quite well.

palm open fist thumb up thumb down victory


We can successfully recognize up tp five simple hand shapes merely using features like compactness.

The program is not very robust due to the inconsistancy of object detection:

1. Using only skin-color detection will fail easily because face, arms and skin-color objects may have larger area than hand. Adding motion information improve the robustness because skin-color staionary objects may be removed from the binary image. The removal of face and arms from the binary image may need template matching.

2. The current object detection in highly sensitive to lightening changes, usually making the contour very jagged and influencing the compactness.