# Homework Assignment 4: Object Tracking, Multiple Object Tracking, Behavior Analysis

### Problem Definition

We were given 2 videos of eels being subjected to differnt test conditions in BU marine biology labs. We were responsible for segmenting and tracking the eels, then computing their average movements per second, and outputting it to a CSV file. Each tank had multiple eels.

### My Method

My method uses a series of computer vision techniques to segment, track, and perform behavioral analysis in video 2. The first componenet of my algorithm is finding the fish tank. This is simple enough, since it never really changes. I have a template image of the region of the tanks that I am interested in, and use template matching to locate this region of the current video frame. Since template matching is a relatively heavy, and the camera basiacally never moves, I took the liberty of only re-computing the coordinates of the tank every 300 frames. For all the frames in between, I use coordinates from the last computation that are stored in golabl variables. Using these coordinates, I seperate out the area of interest in a new Mat object for each frame. Then, the first thing I do is apply background differencing, to isolate the eels. This fairly accurately finds the eels, and outputs a binary image. However, to counteract inaccuracies in this image, I then apply a series of dilations and erosions, as well as a gaussian blur to improve the shape i get in my binary image. This final binary image is stored after every cycle, and is then used for optical flow computations, which gives me a vector of the movement between the two frames in each image. I then use this vector to calculate the average motion of the eels.

### Experiments

I wound up doing a lot of experiments in this assignment. The problem was that I started off with a completely different approach to how I wound up finishing this project. At first, I thought I would be able to segment out the eels based on their color, but their movement towards and away from light sources became problematic. I thne tried converting the image to black and white and leaving only the elements that are as dark or darker than the eels, which also failed for two reasons: the movement of the eels caused them to reflect light and become brighter than the static threshold, and the movement of the water in the tank changed the lighting. After this, I decided to give optical flow a try. This worked much better, however, I still struggled with movements in the water, causeing inconsistencies. When the water moves, it tends to reflect or refract light. This proved to be really troublesome. I first attempted to combat this with a simple operation that converts pixels brighter than a static threshold to 0 intensity. However, this was blatently naive, and actually highlighted the change in brightness caused by the water to become even clearer, and made my optical flow worse. Instead, I decided it was best to mitigate this with a strong Gaussian Blur, which was far more effective. The blur averaged the brightness of the surrounding regions, reducing the glare, and sometimes eliminating it. However, this was not enough. I followed the blur up with dilations and erosions in order to try to get rid of as many glare spots as I could without damaging my eel segmentation. As it turns out, the background differencing detects far fewer glare spots than optical flow does, so I wrote the function, backgroundSubtraction, to compare the optical flow output to the background differencing output, and zero any pixels that dont match up. This wound up smoothing things more, however, it is not perfect. I think with the enough tweaks, I could get to clean up the first few frames.

### Results

#### Optical Flow

I have created the following three visualizations to show the results of my algorithm. The first shows the blurred and morphologically adjusted output of my background differencing algorithm. As you can see, it sometimes struggles to detect the eel, especially when it is still, or being reflected off the sides of the tank. The next image is a thresholded version of the background differenced image for more clarity. The last one shows the background of the original image wherever the algorithm detects the eel. Then, it uses the optical flow vectors to draw lines showing the estimated motion of the eels. For the most part, my algorithm is fairly successful at detecting the eels. I actually show one of the worse segments in my example here. However, it does struggle with a lot of lighting issues caused by movememts in the water, and reflections off the glass tank. These were issues that I attempted to mitigate, but ultimately failed to remove. This unfortunately throws off my calculations, which I didnt have time to print to formal CSV files. However, it appears that there is a bug in my math anyway. Currently, my algorithm just prints these values to console.

### Discussion

Frankly, I still can't think of a way to handle the problems I have been having with the water reclecting light, and the tanke reflecting the eel. The only solution I can think of is to change the camera angle. I wish I had more time to get the segmentation right, and actually work on creating vectors for each object in the image. Again, I struggled to get the Connected Component function from OpenCV's library to work, which was a major road block. However, had I gotten that to work, I think it would have been pretty easy to create a vector for an object, then compare its average movement against that of previous frames, using the vectors generated by optical flow, to calculate the average movements per second. If you look at the drawn vectors in my optical flow example, you can see that dispite being thrown off by the various false positives, they are fairly accurate. However, the largest problem I had was with the false negative: the eel that stayed statically in the bottom left corner of video 2. As you can more clearly see in the optical flow example, my algorithm sometimes will find it at random, however when it does, it also comes with an explosion of false positives. I actually discovered that this happens because when that eel moves, it pushes the sponge it was hiding behind, which in turn moves the sponge, 2 plants, and displaces some water. This sequence of events gets differenced from the background and detected by optical flow, since it is seen as a moving object, and therefore not part of the background, and because it manipulates the brightness levels at parts of the image, which triggers the optical flow as well.