Assignment 2

CS 585 HW 2
Yifu Hu
No Teammate
Sept. 20, 2017


Problem Definition

Using "template matching" and other basic computer vision techinique to build a program that can recongnize at least four different hand shapes in real-time.


Method and Implementation

The general idea of my method is:

  1. Use skin color detection to detect the potential hand area
  2. Use horizontal and vertical projections along with thresholding method to further extract the exact bounding box of hand, which is called by me AoI (Area of Interest)
  3. Binarize the AoI, load the templates and convert them into grayscale images then binarize then using the same threshold
  4. Resize the template so that the template image has the same height of the AoI image
  5. Apply template matching techinique by calculate the adjust correlation index(ACI), which is definded as
    ACI = max(cor(AoI,template))/max(cor(template,template)), where coris the cross-correlation funciton and max is the funciton to find the maximum value of a matrix
  6. The template with the hightest ACI is the hand shape we are looking for

I wrote all steps within the main function. The funcitons I called from opencv are:


Experiments

The goal of my experiment is to recongnize four different hand shape, the template for each hand shape are listed in the table below

NameTemplate
PalmPalm Template
HoleHole Template
PeacePeace Template
ThumbUpThumbUp Template

Results

Here are several recognition results of all four hand shapes:

Hand Shape NameResult
PalmRecongnition of a palm
HoleRecongnition of a hole-shaped hand
PeaceRecongnition of a V-shaped hand
ThumbUpRecongnition of a thumb-up

A real-time demo of the above recognition process can be viewed in the video below.

As the background pixels may also be detected as "skin color", to help improve the accuracy of the bounding box, a median filtering is applied to the raw skin-color bulbs. As shown below.

A confusion matrix can be obtained by changing the handshape slightly to see what will the program recognize it as. Here is the result.

Hand ShapePalmHolePeaceThumbUp
Palm5011
Hole0700
Peace3162
ThumbUp0015

Discussion


Conclusions

Based on the discussions, I think there are a lot of ways we can further improve the accuracy and performance of the above hand shape recognition algorithm.


Credits and Bibliography

Cite any papers or other references you consulted while developing your solution. Citations to papers should include the authors, the year of publication, the title of the work, and the publication information (e.g., book name and publisher; conference proceedings and location; journal name, volume and pages; technical report and institution).

Material on the web should include the url and date of access.

Credit any joint work or discussions with your classmates.