CS585 Homework 2

Lian Duan lduan@bu.edu

Problem Defination

Design and implement algorithms that recognize hand shapes (such as making a fist, thumbs up, thumbs down, pointing with an index finger etc.) or gestures (such as waving with one or both hands, swinging, drawing something in the air etc.) and create a graphical display that responds to the recognition of the hand shapes or gestures.

Method and Implementation

I mainly used skin color detection, and binary image processing (Emin/Emax) methods

Firstly, I processed the image with skin color detection following the guidance from the Lab sessions. I extracted the skin image out to be a binary image.

Then, I calculated the position of the centroid, using the function void findCentroid(Mat& image, int* position). And I used the equations taught in the class.

Now, with the centroid given, I am able to calculate the image moment related parameters. I used void findMInertia(Mat& image, int* position, int* abc, float* mInertia) to calculate the parameter a, b, c. And the Emin and Emax of the image.

Finally with the different Emin/Emax results, I am able to tell different hand shapes. And I output text in the image in realtime with OpenCV, as a response to the different hand shapes.


Below are screen shots of the program running.

On the left is the result of the skin color detection. On the right in the window is the real time image display, together with the response text I have. Below is the terminal showing the current image moment parameters.

My program will recognize 4 different hand shapes: the traditional game rock, scissors, and paper. And one extra flat hand.

Here are the 4 different results:

Scene 1 : Rock

Scene 2 : Paper

Scene 3 : Scissors

Scene 4 : Flat

Additional fun aspects and skills

For the problem I found that sometimes noise speckles can cause lots of trouble for recognizing the Emin/Emax. Because for far away noise pixels, they have a large r^2 that could count huge towards the calculation. For this reason I did some trick to eleminate these points, to maintain the stableness of the results.

Confusion Matrix

The confusion matrix is shown here

Conclusion and Analysis

By using skin color detection and binary image moment calculation. It is possible to distinguish different hand gestures and shapes. And a good results can be achieved using these methods.

In the meantime, some challanging points lies in the problem:
1, the skin color parameters needs to be set to a reasonable amount.
2, The noise pixels may cause big weights in the second moment calculation, some tricks would hugely improve the result.