A set of images is given as input. Each image contains one to four different shape blobs(square, triangle, circle) in different colors.
The requirents to the program are:
- For each shape image, determine its background color and label each and every shape blobs according to their color.
- For each shape blob, implement a border following algorithm to find its outermost contour (i.e. the border pixels) and compare it with OpenCV's "findContours" function.
- For each shape blob, classify its border pixels (in other words, segment its outermost contour) into three types: borders against the background, borders against another shape blob and borders that are a border of the whole image.
- For each shape blob, come up with an algorithm that can recognize its shape type (square, circle, or triangle).
Method and Experiments
We used the original dataset as our input.
Step1. Denoising and Determine the background color
The original image is in JPEG format. Due to the compression algorithm used in JEPG images, the original images are high-noised. As illustrated below, the colors on the edges of blobs are somehow blurred, indicating that in one blob there are several different colors. We need to denoise the image before finding the contours.
To denoise the image, we used the average filter function Blur:
void blur(InputArray src, OutputArray dst, Size ksize, Point anchor=Point(-1,-1), int borderType=BORDER_DEFAULT ).
In our case, the parameter ksize = Size(3, 3).
To determine the background color, we examine the color of each pixel and calculate the number of their appearances. The color that appears the most is considered the background color.
Step2. Label shape blobs
To label the shape blobs, we scan the image pixel by pixel. If the current pixel satisfy both of these assumptions:
(1) The color of the current pixel is not the same as or close to(threshold = 10) the background color.
(2) The current pixel is not at the intersection of two shape blobs
mark the current pixel white.If the pixel has color of the background, mark it black.
Then we convert this image to binary image and feed it to the FindBinaryLargeObjects(const Mat &binaryImg, vector
Step3. Find Contours of Image
The labeled and recolored image is ready for us to find its contours. For each pixel in the image, we examine the RGB value of itself and of every pixel in its N8 neighbourhood. If the RGB value of this pixel is different from the RGB value of one of the pixels in its neighbourhood, this pixel is marked as contour.The width of the contours we got from this step is 1 pixel.
Step4. Classify the borders
We could classify the borders now that we have contours, the background color and colors in each shape blob is unified.We simply go through all the pixels in the image. If the current pixel is next to a background pixel, the current pixel will be labeled as border against the background.If none of the current pixel's N8 neighbor is background pixel, the current pixel is considered border between two shape blobs. If the pixel is on the boundary of the image with color different from the background, it is border against the boundary of the image.
We use different colors to represent different borders.
Step5. Determine shapes
We use slopes to determine the shape. We select up to 32 pixels on the blob's border against the background and store them in a vector.
We use white to represent squares, orange to represent triangles and indigo to represent circles.
Step6. Calculate the precision
To calculate the presicion, we calcuate the centroid of each blob. We check whether it is on one of the white blobs of the annotation masks. If so, we compare the filename with estimated shape type to get TP and FP.
|Trial||Source Image||Background color||Shape blobs||Borders||Classified Borders||Blobs||Annotations|
Discuss your method and results:
- The precison of circle is not satisfying. We think its because some squares and triangles not recognized correctly are classified as circles.
- We implemented our findContours() function. Compared to the findContours() in OpenCV, our function is not able to store and output information of contours. Moreover, our function only marks out one thick layer of contour whereas the function in OpenCV is capable of finding the most exterior contour.
- Implement a way to recognize circles with slopes.
- Find better parameters and methods to denoise the images.
- Improve the shape recognization by calculating borders from other directions.
- The convexity of borders can be applied to classifying the blobs.
- Some other methods could be used to perform this task, machine learning methods for example.
- Code of Lab2
- Find the Center of a Blob (Centroid) using OpenCV:https://www.learnopencv.com/find-center-of-blob-centroid-using-opencv-cpp-python/ Access date:09/26/2018
We successfully find ways to implement a program that meets all the mandatory requirements. Our programs could determine the background color of all images in the dataset and detect most of the shapes, find their contours, classify the borders and segment each blob according to their colors. The precision is fine but there are still lots of room for improvments.
Credits and Bibliography