# Assignment 2

CS 585 HW 2
Yifu Hu
No Teammate
Sept. 20, 2017

## Problem Definition

Using "template matching" and other basic computer vision techinique to build a program that can recongnize at least four different hand shapes in real-time.

## Method and Implementation

The general idea of my method is:

1. Use skin color detection to detect the potential hand area
2. Use horizontal and vertical projections along with thresholding method to further extract the exact bounding box of hand, which is called by me AoI (Area of Interest)
3. Binarize the AoI, load the templates and convert them into grayscale images then binarize then using the same threshold
4. Resize the template so that the template image has the same height of the AoI image
5. Apply template matching techinique by calculate the adjust correlation index(ACI), which is definded as
ACI = max(cor(AoI,template))/max(cor(template,template)), where coris the cross-correlation funciton and max is the funciton to find the maximum value of a matrix
6. The template with the hightest ACI is the hand shape we are looking for

I wrote all steps within the main function. The funcitons I called from opencv are:

• medianBlur: to remove the skin-color noise using median filter
• reduce: to calculate the projections of the potential hand area
• minMaxIdx: to find the maximum value (and its index)of a matrix
• threshold: to binarize an image with a given threshold
• rectangle, putText: to draw graphics responds
• filter2D: to calculate the cross-correlation of two matrices

## Experiments

The goal of my experiment is to recongnize four different hand shape, the template for each hand shape are listed in the table below

NameTemplate
Palm
Hole
Peace
ThumbUp

## Results

Here are several recognition results of all four hand shapes:

Hand Shape NameResult
Palm
Hole
Peace
ThumbUp

A real-time demo of the above recognition process can be viewed in the video below.

As the background pixels may also be detected as "skin color", to help improve the accuracy of the bounding box, a median filtering is applied to the raw skin-color bulbs. As shown below.

A confusion matrix can be obtained by changing the handshape slightly to see what will the program recognize it as. Here is the result.

Hand ShapePalmHolePeaceThumbUp
Palm5011
Hole0700
Peace3162
ThumbUp0015

## Discussion

• As shown in the Results, the accuracy of the recognition is largely depend on the hand area extraction. Since my program utilize some basic skin-color detection techinique to locate the hand area, it's not robust and the performance is heavily varied under different lighting condition
• The template matching method is used for the classification of a detected hand shape, thus it is required that the test hand shape should have the exact same direction and same shape of the template, a slight change in shape may cause wrong recognition result
• The confusion matrix demonstrated that the "palm" can be mistakenly recongnized as "peace" more easily than "peace" being recognized as "palm"

## Conclusions

Based on the discussions, I think there are a lot of ways we can further improve the accuracy and performance of the above hand shape recognition algorithm.

## Credits and Bibliography

Cite any papers or other references you consulted while developing your solution. Citations to papers should include the authors, the year of publication, the title of the work, and the publication information (e.g., book name and publisher; conference proceedings and location; journal name, volume and pages; technical report and institution).

Material on the web should include the url and date of access.

Credit any joint work or discussions with your classmates.