CS585 HW2: Hand Gesture Recogniation

CS 585 HW 2
Ding Jin
9/20/2017


Problem Definition

Design and implement algorithms that delineate hand shapes or gestures
and create a graphical display that responds to the recognition of the hand shapes or gestures.


Method and Implementation

The general apporoach is to use template matching to cut the apporiate rectangle for drawing the contour
then compare the contour with different presett templates.

_____________________________________________________________________________________________
Prepocessing:

Before using template matching, I firstly do the processing for better matching perfromance:
--For converting RGB image into grayscale image, I used my version greyscale function which takes
the average of RGB value.
--I also blur the image for better static frame normalization later; for this part, I used the opencv
function blur() which uses kernel smoothing.
--I used my own binary image converting function which sets on the threshold of grayscale value 128; this
binary converting is to avoid the effect of brighness as much as possible.The preset template are also binary image.
_____________________________________________________________________________________________
Template Matching:

--For template matching, I used the opencv function matchTemplate() which loads an current static frame and the preset image template and perform a template matching procedure with normalized cross correlation. --After the matching, I also normalized the results and localize the minimum and maximum values in the result matrix by using minMaxLoc()to get the best matches. _____________________________________________________________________________________________
Contour comparing:

--For the contour part, I used the opencv function findContours() and drawContours().
--Since I already match the hand and rect the area, I simply draw the contour in the rectangle.
--For the four given hand gestures, I draw the static contours of the template image.
--The final step is to use function matchShapes() which compares two shapes(in this case, the shapes are the contours) using image moment; this outputs four values which respect to each template, each gesture; The least value means the least differences which can be used to distinguish between four gestures.


Results


Confustion Matrix
five fist thumb fist
five 6 1 2 2
fist 1 4 3 2
thumb 1 3 3 2
yes 2 2 2 4
_____________________________________________________________________________________________




Testing Video




_____________________________________________________________________________________________






Recognizing image...
"five" result
"fist" result
"thumb" result
"thumb" result(fail)
"yes" result


Credits and Bibliography

[1]--The skeleton code and the VS environment are coming from the second cs585 lab
[2]--The opencv template matching tutorial: http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html