Using "template matching" and other basic computer vision techinique to build a program that can recongnize at least four different hand shapes in real-time.
Method and Implementation
The general idea of my method is:
- Use skin color detection to detect the potential hand area
- Use horizontal and vertical projections along with thresholding method to further extract the exact bounding box of hand, which is called by me AoI (Area of Interest)
- Binarize the AoI, load the templates and convert them into grayscale images then binarize then using the same threshold
- Resize the template so that the template image has the same height of the AoI image
- Apply template matching techinique by calculate the adjust correlation index(ACI), which is definded as ACI = max(cor(AoI,template))/max(cor(template,template)), where coris the cross-correlation funciton and max is the funciton to find the maximum value of a matrix
- The template with the hightest ACI is the hand shape we are looking for
I wrote all steps within the main function. The funcitons I called from opencv are:
- medianBlur: to remove the skin-color noise using median filter
- reduce: to calculate the projections of the potential hand area
- minMaxIdx: to find the maximum value (and its index)of a matrix
- threshold: to binarize an image with a given threshold
- rectangle, putText: to draw graphics responds
- filter2D: to calculate the cross-correlation of two matrices
The goal of my experiment is to recongnize four different hand shape, the template for each hand shape are listed in the table below
Here are several recognition results of all four hand shapes:
|Hand Shape Name||Result|
A real-time demo of the above recognition process can be viewed in the video below.
As the background pixels may also be detected as "skin color", to help improve the accuracy of the bounding box, a median filtering is applied to the raw skin-color bulbs. As shown below.
A confusion matrix can be obtained by changing the handshape slightly to see what will the program recognize it as. Here is the result.