Assignment 4

CS 585 HW 4
Yizhang Bai
Haoyu Zhang, Hui Li

Problem Definition

The goal of this assignment is to design and implement algorithms that can segment and track moving objects in a video. We are given a video of two tanks with eels and hermit crabs moving in them. The setup consists of having one side of the tank filled with AstroTurf, a habitat that they can hide in, and an open feeding area with a white bottom. The eels are dark and so stand out against this white bottom.

The ultimate goal of this homework is to map duration of the eels and crabs in the open feeding side of the tank (empty, white bottom, top portion of video).

Specifically, our tasks are listed below:

Find the boundary box of the two tanks. Note that this boundary box should be determined such that your code can run with the location of the camera moved

Automatically find the portions of the video with animal action.

Segment the eels and crabs from the background using the techniques you have already learned

Track the movement of the eels and the crabs i.e., identify the same part of eels (its tail, head and center of the body) and the center of mass of the crabs from frame to frame

Method and Implementation

Finding Bounding Box
Since the white color of the tank, we apply projection detection to find the bounding box. By doing projection in both x and y axis, we find the steep change in white pixel, where it is the boudary of the tanks.

Background Subtraction
We have tried several methods to do background subtraction: MOG2 and percentage threshoulding. An interface, a track bar, is created to visualize the different outcome when choosing different percentage.

We associate two objects in adjacent frames by finding the object with the minimum distance.

Kalman filter is used to track objects in the video.


We implements our algorithms and run it on the video.


The result of background subtraction:


The strength is that our algorithms are simple and robust. With a simple algorithms, it always means a lower probability to have bugs, and if bugs exist, then it is easy to debug.

The weakness of our algirhms is that the matching part could be improved if we have more time. We could have tried the Hungarian algorithm to do matching, which have a better performance and result than the simply finding the minimum distance.


Tasks for this assignment is really difficult. First of all, the background subtraction is not easy at all since the eels are always moving, and noise occurred by the reflection and light changing from movement of eels exists. Apart from background subtraction, the tracking is also difficult, again the noise keep annoying the matching. Besides, due to the minor failure in subtraction which removes certain part of eel causing the eel becomes more than one part, increase the number of objects.