# Programming Assignment 3

CS 585 HW 3
Kaiyuan Fan
Kaikang Zhu
10/10/2018

## Problem Definition

The problem of this assignment is to design and implement algorithms that can recognize hand shapes or gestures, and create a graphical display that responds to the recognition of the hand shapes or gestures.

1. Your algorithm should detect at least four different hand shapes or gestures.
2. It should also work on cases when two hands are both in view and recognize each one separately.
3. Template matching is required for this assignment.

## Method and Implementation

1. In order to detect shape by matching with the templates, I firstly preprocess the orignal frame to filter out only skin color pixels.
2. Then using the threshold skin color image to find contours. Before finding the contours, I used function medianBlur to smooth image so that one hand only has one contour.
3. When there is one contour (one hand in frame), apply all the templates and find the corresponding shape. When there are more than one contour (indicates two hands), apply each hand templates separately.
4. MatchTemplate function returns the best match location, so that I can draw a bounding rectangle box around my hand, and the box size can be determined by template size.

Functions: void mySkinDetect(Mat& src, Mat& dst); //Function from lab3 solution, detects whether a pixel belongs to the skin based on RGB values void filter(Mat& src, Mat& dst); //Function that filter orignal frame left with skin color pixels only void detectBoth(Mat& src, Mat& dst); //Function that detect shape using all templates void detectRighthand(Mat& src, Mat& dst); //Function that detect shape using right hand templates void detectLefthand(Mat& src, Mat& dst); //Function that detect shape using left hand templates int contourDetect(Mat& src); //Function that returns the number of contours double nccMax8(double a, double b, double c, double d, double e, double f,double g,double h); //Function that returns the maximum of 8 doubles double nccMax4(double a, double b, double c, double d); //Function that returns the maximum of 4 doubles OpenCV library functions: medianBlur findContours matchTemplate rectangle putText

## Experiments

I am applying below template to detect hand shapes.

Name Rock Scissors Paper Hole
right hand template
left hand template

## Results

### Results

Trial Processed Image
Filtered frame
Trial 1
Trial 2
Trial 3
Trial 4
Trial 5
Trial 6
Trial 7
Trial 8
Trial 9
Trial 10

Confusion Matrix:

## Discussion

Graphics Program:

The graphical display is built by rectangle and putText OpenCV library functions. As I know the best matching location, I make a bounding box around it and put the corresponding shape text.

The skin color detection algorithm is highly affected by the lightness. Template matching method is influenced by the direction of camera and orientation of hands. During the experiment, I have to set these factors as same as I take the picture of templates. My method can effectively recognize four different hand shapes and handle two hands successfully. Most of the hand shapes can be recognized correctly if I put my hands the way as the templates.

## Conclusions

In conclusion, I think template matching may not the best way to recognize hand shapes since the other factors could influence the result easily. We may combine other measure algorithm in the future to improve the success rate of matching.

## Credits and Bibliography

Lab3 solution

https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html accessed at 10/06/2018

https://docs.opencv.org/2.4/modules/imgproc/doc/filtering.html?highlight=medianblur#medianblur accessed at 10/06/2018

Worked with Kaiyuan Fan