Welcome to Seunghun Oh's web page
Assignment 4: Segmentation and Multi-Object Tracking
Collaborated with Lang Gao
The goal of this assignment is to design and implement algorithms that can segment and track
moving objects in a video.
We are given a video of two tanks with eels and hermit crabs moving
in them. The setup consists of having one side of the tank filled with AstroTurf, a habitat that
they can hide in, and an open feeding area with a white bottom. The eels are dark and so stand
out against this white bottom. The hermit crabs are a little more difficult to see.
- The segmentation, detection, and tracking of the eels and the hermit crabs were rather difficult. The hermit crabs were difficult to see, sometimes even with humans eyes, and were very difficult to detect and calculate specific center of masses for.
- Eels wer easier to detect based on multiple approaches. We were able to segment and track their motions to some degree.
- I was able to distinguish the water tanks from the rest of the scene by first sharpening the frame by putting the frame through a Laplacian filter. Then, The image was put through a standard binary threshold and then put through a distance transformation. The transformation gave peaks of brightness for the two water tanks. After making them markers and watershedding them, the boundary boxes for the two tanks were established.
- Sometimes, depending on the amount of lighting on the walls, the walls would be labeled as a water tank. However, when that was the case, corrective measures were taken.
- In order to segment the creatures, we tracked their motion history as well as their difference in color and hue from the background. The eels in both video types had high motion, so we coded motion history and colored in all of the recent paths for the eel. However, the crabs' movements were negligible in most cases. Thus, in order to locate the crabs, gaussian thresholding and morphology was applied. However, regardless of any attempt to retrieve a good binary image for center of mass processing, we were unable to specifically only detect the hermit crabs only and only determine their center of mass. We wee also unable to track the animals' movements.
- With more time, we would have attempted a different approach in order to isolate the animals. Perhaps focusing more on the color aspect (specific RGB values or HSV values within the area) would have worked better. Howeve, we were not able to use this approach, because of the amount of objects with the same color ranges as the eels and the crabs. Also, the background noise and water deflection prevented from properly selecting eel or crab colors. Another potential approach would have been to focus on the optical flows of the animals and track the center of mass, as well as movement patterns that way.
- Either way, isolation of exactly the animals would have allowed for tracking as well as calculation for center of mass and undulations. In order to calculate the unulations, we concluded that optical flow was necessary.
Overall, we continue to believe that object segmentation, detection, and tracking are incredibly hard concepts to follow in real-life settings. We hope that in the upcoming project and in the future that we will be able to apply more of the aspects that we have learned.
Below are the screenshots of our results:
Boundary boxes for the water tanks (with some interference) as well as motion history of eels
Segmented image of eels and crabs, left and right respectively
All work was done together with Lang Gao and through mutual research.