Assignment 3

CS 585 HW 3
Jeanpiere Solorzano
October 10 2018


Problem Definition

Design and implement algorithms that recognize hand shapes or gestures, and create a graphical display that responds to the recognition of the hand shapes or gestures.



Method and Implementation

The general idea of my implementation was to:

(1) Create Templates: Take pictures of my hands with different gestures and create templates/masks that will be used to compare to the frames acquired via the camera live feed. I used 4 scalings on the images to make sure that the algorithm detects more shapes even if distance of hand to camera differs.
(2) Frame Capture Capture a frame of the video and use "MedianBlur" in order to get rid of extra pixels that can mess up with skin detection.
(3) Skin Detection Used the Algorithm provided by lab3 in order to detect the skin color of my hands.
(4) Template Matching We check each frame of the original live and use the masks to check where the image is most similar to the templates we have available. If a match exists we draw a box around the hand to let users know a match has been found.


Experiments

For the experiments I made the hand gestures to the camera multiple times and checked to see if a rectangle was form when it detected a match.
The goal of my algorithm was to recognize 4 different hand shapes which can be found below.


Gestures Template
Ok_Sign okay sign
Palm palm
Peace_Sign peace sign
ThumbsUp thumbs up


Results

Here are some results of the hand detection. This also includes Detecting 2 hands


Gestures Template
Ok_Sign okay sign
Palm palm
Peace_Sign peace sign
ThumbsUp thumbs up
Two Hands thumbs up

Here is a Confusion Matrix to see how many times each hand gesture was detected correctly (out of 10 trials)


Hand Gesture Ok_sign Palm Peace_sign Thumbs_up
Ok_Sign 7 2 1 0
Palm 0 10 0 0
Peace_Sign 2 0 8 0
ThumbsUp 1 0 0 9


Discussions

The results showed that the algortihm was efficient in checking and detecting hand gestures. There are several factors that can greatly affect the hand detection especially brigthness levels in the room. My room is dimly lit and I had to make sure I found a part of the room where the skin detection algorithm would work more efficiently. Some hand gestures can be mistaken by others as can be seen in the matrix, but these problems were very minimal.



Conclusions

There needs to be a better implementation in skin detecting based on different levels of brightness levels, which I will need to work on for future projects. That has been the most difficult part on this experiment. The accuracy of the detection can also be improved if we provide more templates for a broader matching reference



Credits and Bibliography

Used the following documentation site to write the program: https://docs.opencv.org/3.3.0/

Used Skin Detection algorithm found in Lab3 of the CS585 website: http://www.cs.bu.edu/~betke/cs585/

Followed steps on this OpenCv website to create myTemplateMatching: https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html