Design and implement algorithms that recognize hand shapes (such as making a fist, thumbs up, thumbs down, pointing with an index finger etc.) or gestures (such as waving with one or both hands, swinging, drawing something in the air etc).
Create a graphical display that responds to the recognition of the hand shapes or gestures.
Your algorithm should detect at least four different hand shapes or gestures.
It should also work on cases when two hands are both in view and recognize each one separately.
Template matching is required for this assignment.
1. Pre-process frames in order to separate hands and the background and detect skin;
2. How to use template;
3. Pre-process the templates;
4. Match image(frame) and template;
1. Use the camera of the laptop to get video.
2. Output results:
There are three windows to show result. The first one is used to show the original frame and draw the contours of the hands. The rest of the windows are used to show result. If there is a match, then show the hand shapes, else show none.
3. The precision:
The precision is pretty good. We calculated the situation in the video in two and a half minutes, having 40 times of gestures changes with one hand and two hands, and the accuracy rate was 100%.
4. Wave hand:
For this part, the output will tell there is a wave in the frame and also the direction of the wave, from left to right/up to down.
Images to show match result
Result: the screenshot1
Result: the screenshot2
Result: the screenshot3
Result: the screenshot4
Result: the screenshot5
Result: the screenshot6
Result: the screenshot7
Result: the screenshot8
Result: the statistical data1
Result: the statistical data2
Result: wave hand1
Result: wave hand2
Result: wave hand3
When get this assignment, we tried to use templateMatch() function to match the template and the frame, but the result is not good. If the size of template is bigger than the frame or just a little bit smaller, the result is very bad. Many examples show that when the template is a part of a screenshot of the frame, the result is perfect.
After this failure, we tried to use contours to match and we found the function matchShapes(). At first, we want to separate the background using difference between the first frame and the latter frame, but the result is not so good, the contours is always not clear enough, and the speed is really slow.
Then we convert frame to HSV version and give threshold to detect skin which has great precision (Of course before the match process, we erode and dilate the frame to delete some noise in the frame).
In good condition, the precision of the program can be 100%. Really really happy about the result in the end!
1. Help offered by TA (The idea of how to get two hands using area threshold), 10/09/2018
2. OpenCV documents to find some methods and functions: https://docs.opencv.org/3.2.0/annotated.html, 10/04/2018-10/09/2018
3. Some functions of OpenCV to compute contours: https://blog.csdn.net/wangshuai610/article/details/79913600, 10/06/2018
4. Matchshapes(): https://docs.opencv.org/2.4/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html, 10/06/2018
5. An example of matchshapes(): https://blog.csdn.net/luoyouren/article/details/65633170, 10/06/2018