Multiple Object Tracking

CS 585 HW 4

Shivam Satwah

Aaron Jacob Varghese

3rd April 2020

Problem Description

The goal of this  programming assignment is for us to learn more about the practical issues that arise when designing a tracking system. We tracked moving objects in video sequences, i.e., identifying the same object from frame to frame:

Method and Implementation


The bat dataset did not require segmentation. For the cell dataset we used frame difference with the average frame in the video. Then we used both absolute and adaptive thresholding and compared the results of tracking on both these segmentation outputs. For the absolute thresholding, we applied a Gaussian blur to filter out noise.


For the tracking pipeline, there are the steps we followed -

  1. KalmanFilter class, which implements the Kalman Filter and provides functions for prediction and correction.
  2. Created a Tracks class, which keeps track of the details of a single track. Each instance of the Tracks class keeps an instance of the KalmanFilter class and the current prediction.
  3. A Tracker class, to predict and update each track. Within the main function of this class, we -
  1. First compute the cost matrix between the detections and the current predictions of each track. This is done using a simple euclidean distance function.
  2. Use the Hungarian algorithm to compute the best matches between the detections and the tracks.
  3. Once the assignments are done, we remove those assignments that have a cost i.e. distance greater than a certain threshold
  4. We keep track of the number of times a certain track has not been assigned to any detection and if the number goes beyond a threshold, we delete the track i.e. the object has gone out of the frame.
  5. If there are any detections that are still not matched to any track, we assume these belong to new objects and create new tracks to correspond to these objects.
  6. Finally, with all the assignments made, we create new predictions.


We tried adaptive and absolute thresholding for the bats dataset.


Absolute Thresholding

Adaptive Thresholding





As one can see, absolute thresholding provides reasonable results, whereas adaptive fails to find the cells.



        The following are the screenshot images of tracking videos for bats and the          cell dataset



Frame 1

Frame 2




Bat Tracking -


In this image, box 83 is fairly close to other boxes and still maintains its track very well. More of this can be seen in the video

In this image, box 2 is a bit detached from the actual detection, likely because of the proximity to four other boxes.

  1. The tracking is fairly stable for most cases
  2. One issue is collisions between tracks i.e. when bats come too close to each other, causing the tracks to oscillate between the bats, until a certain distance is passed between them. We tried to mitigate as much of this as possible by using a distance threshold for assignments.
  3. We tried modeling just the positions between subsequent frames and also the positions and velocity. The latter method seemed to provide more stable results. This would make sense, as there is a lot of erratic movement by the bats and modeling velocity provides a further refinement.

Cell Tracking -

  1. The tracking is unstable for many cases. This could be due to the erratic nature of cell movement, since the Kalman filter used can only provide accurate results for Gaussian or Linear models.
  2. Collisions also cause issues here, as do cell splittings. While the algorithm does correctly identify new cells, it’s behaviour when the splitting happens is unstable.
  3. Cell segmentation is more or less accurate with absolute thresholding and blurring.

        Potential Improvements - It might help to improve the modeling of the system, either by adding additional parameters like acceleration or by tweaking existing parameters.


We have built a multi-object tracking system and demonstrated it on two different image sequences of bats and cells, by using the Kalman Filter and Hungarian algorithm.

Credits and Bibliography


Discussion with Rahul Ramachandran