Track Everything Package

2020
This project started in 2020.

This project is an open-source package built-in Python, it uses and combines the data form object detection models, classification models, tracking algorithms and statistics-based decision making. The project allows you to take any detection/classification models from any Python library like TensorFlow or PyTorch and add to them tracking algorithms and increase the accuracy using statistical data gathered from multiple frames.
 

Track Everything - Pipeline Overview

You can find this project in one of my repositories here.

The Pipeline

The pipeline starts by receiving a series of images (frames) and outputs a list of tracker objects that contains the objects detected and the probability of them being in a class.

Breaking it Down to 5 Steps

1st Step - Get All Detections in Current Frame

First, we take the frame and passe it through an object detection model, we can use any Python model, then filter out redundant overlapping detections using the Non-maximum Suppression (NMS) method and add all of the detection to the detections list.

2nd Step - Get Classification Probabilities for the Detected Objects

After we have the detections from step 1, we put them through a classification model to determine the probability of them being in a certain class (if no classification model is supplied the classification is applied during the previous step). We do this by cropping the frame to the object bounding box and then pass it through the classification model. We add this data as a vector of probabilities to each of the detection in the detections list.

3rd Step - Updated the Trackers Object List

We have a list of trackers object which is a class that contains among other things an OpenCV tracker object, unique ID, previous statistics about this ID and indicators for the accuracy of this tracker. In the first frame, this trackers list is empty and then in step 4, it’s being filled with new trackers matching the detected objects. If the trackers list is not empty, in this step we update the trackers’ positions using the current frame and dispose of failed trackers.

4th Step - Matching Detection with Trackers

Using intersection over union (IOU) of a tracker bounding box and detection bounding box as a metric. We solve the linear sum assignment problem (also known as minimum weight matching in bipartite graphs) for the IOU matrix using the Hungarian algorithm (also known as Munkres algorithm). The machine learning package SciPy has a build-in utility function that implements the Hungarian algorithm. 

matched_idx = linear_sum_assignment(-iou_matrix)

The linear_sum_assignment function by default minimizes the cost, so we need to reverse the sign of IOU matrix for maximization.
The result will look like this:

For each unmatched detector, we create a new tracker with the detector’s data, for the unmatched trackers we update the accuracy indicators for the tracker and remove any that are way off. For the matched ones, we update the tracker position to the more accurate detection box, we get the classification data and use the StatisticalCalculator class to adjust the results.

5th Step - Decide What to Do

After step 4 the trackers list is up to date with all the statistical and current data. The tracker class has a method to return the current classifications and confidence of those scores, we then update the detectors and iterate through them. A detector with a low confidence score probably came from a tracker with not enough data or the detection is poor, we can mark those using the uncertainty parameters in the VisualizationVars. We can then draw all the results or get the results directly from the detections list.

Lang/Lib/Pro Version
Python 3.8.1
OpenCV 4.2.0.34
NumPy 1.18.4
SciPy 1.4.1
Pillow 7.1.2
TensorFlow 2.2.0

Type Python Package
Input Camera/Video Feed
Output Enhanced Object Tracking & Classification