The problem of this assignment is to design and implement algorithms that can recognize hand shapes or gestures, and create a graphical display that responds to the recognition of the hand shapes or gestures.
- Your algorithm should detect at least four different hand shapes or gestures.
- It should also work on cases when two hands are both in view and recognize each one separately.
- Template matching is required for this assignment.
Method and Implementation
- In order to detect shape by matching with the templates, I firstly preprocess the orignal frame to filter out only skin color pixels.
- Then using the threshold skin color image to find contours. Before finding the contours, I used function medianBlur to smooth image so that one hand only has one contour.
- When there is one contour (one hand in frame), apply all the templates and find the corresponding shape. When there are more than one contour (indicates two hands), apply each hand templates separately.
- MatchTemplate function returns the best match location, so that I can draw a bounding rectangle box around my hand, and the box size can be determined by template size.
I am applying below template to detect hand shapes.
|right hand template|
|left hand template|
The graphical display is built by rectangle and putText OpenCV library functions. As I know the best matching location, I make a bounding box around it and put
the corresponding shape text.
Discuss your method and results:
The skin color detection algorithm is highly affected by the lightness. Template matching method is influenced by the direction of camera and orientation of hands.
During the experiment, I have to set these factors as same as I take the picture of templates.
My method can effectively recognize four different hand shapes and handle two hands successfully.
Most of the hand shapes can be recognized correctly if I put my hands the way as the templates.
In conclusion, I think template matching may not the best way to recognize hand shapes since the other factors could influence the result easily. We may combine other measure algorithm in the future to improve the success rate of matching.
Credits and Bibliography
https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html accessed at 10/06/2018
https://docs.opencv.org/2.4/modules/imgproc/doc/filtering.html?highlight=medianblur#medianblur accessed at 10/06/2018
Worked with Kaikang Zhu