The goal of this assignment is to implement algorithms that can recognise shapes and gestures made by the hand.
The program should take video from the webcam as input, analyze and process it to recognise the hand gesture.
Parts of the assignment:
1. Skin detection
2. Template Matching
3. Recognize hand gestures.
Method and ImplementationTask 1 - Skin Detection
- Converted the input RGB frame to HSV and estimated HSV values of skin color to function as the benchmark for thresholding input frame according to skin color.
- Obtained ellipsoid structuring kernel and applied morphological operations like erosion and dilation to it followed by Gaussian Blur. This gives us the skinmask
- Skinmask used in future template matching operations to detect the hand shape in an input frame
Task 2 - Hand Shape Recognition using Templates
- Created segmented templates for different hand gestures.
- Implemented Multiscale Multiple Template matching between gesture templates and the incoming video frames so that the template matching operation is invariant to the scale difference between the object in the template and the incoming frame.
- The bounding boxes and the skinmask used to identify the hand and scaled according to the proportion of change in the size of the hand as it moves near and farther away from the camera.
- Implemented gesture matching for the following gestures:
- Thumbs Up
- Peace Out
Task 3 - Temporal Gesture Recognition
- Found frame-to-frame difference, then used template matching to find the gesture.
- Implemented gesture matching for the following gesture:
The following are the resulting images from the program:
A confusion matrix can be obtained by changing the handshape slightly to see what will the program recognize it as. Here is the result.
|Hand Shape||Palm||Peace||Thumbs Up||Wave|