Programming Assignment 3

CS 585 HW 3
Kaiyuan Fan
Kaikang Zhu

Problem Definition

The problem of this assignment is to design and implement algorithms that can recognize hand shapes or gestures, and create a graphical display that responds to the recognition of the hand shapes or gestures.

  1. Your algorithm should detect at least four different hand shapes or gestures.
  2. It should also work on cases when two hands are both in view and recognize each one separately.
  3. Template matching is required for this assignment.

Method and Implementation

  1. In order to detect shape by matching with the templates, I firstly preprocess the orignal frame to filter out only skin color pixels.
  2. Then using the threshold skin color image to find contours. Before finding the contours, I used function medianBlur to smooth image so that one hand only has one contour.
  3. When there is one contour (one hand in frame), apply all the templates and find the corresponding shape. When there are more than one contour (indicates two hands), apply each hand templates separately.
  4. MatchTemplate function returns the best match location, so that I can draw a bounding rectangle box around my hand, and the box size can be determined by template size.

Functions: void mySkinDetect(Mat& src, Mat& dst); //Function from lab3 solution, detects whether a pixel belongs to the skin based on RGB values void filter(Mat& src, Mat& dst); //Function that filter orignal frame left with skin color pixels only void detectBoth(Mat& src, Mat& dst); //Function that detect shape using all templates void detectRighthand(Mat& src, Mat& dst); //Function that detect shape using right hand templates void detectLefthand(Mat& src, Mat& dst); //Function that detect shape using left hand templates int contourDetect(Mat& src); //Function that returns the number of contours double nccMax8(double a, double b, double c, double d, double e, double f,double g,double h); //Function that returns the maximum of 8 doubles double nccMax4(double a, double b, double c, double d); //Function that returns the maximum of 4 doubles OpenCV library functions: medianBlur findContours matchTemplate rectangle putText


I am applying below template to detect hand shapes.

Name Rock Scissors Paper Hole
right hand template
left hand template



Trial Processed Image
Filtered frame
Trial 1
Trial 2
Trial 3
Trial 4
Trial 5
Trial 6
Trial 7
Trial 8
Trial 9
Trial 10

Confusion Matrix:


Graphics Program:

Discuss your method and results:


In conclusion, I think template matching may not the best way to recognize hand shapes since the other factors could influence the result easily. We may combine other measure algorithm in the future to improve the success rate of matching.

Credits and Bibliography

Lab3 solution accessed at 10/06/2018 accessed at 10/06/2018

Worked with Kaiyuan Fan