Lab 3: Video tracking in MTF

428/615, Martin Jagersand
Submit code and report electronically through the upload link on the e-Class course webpage.
Marks may be given for any of the questions in the assignment, so please answer them all
(many only require a one sentence answer).
Demo: In your lab session.

This assignment is about experimenting with the trackers in MTF and finding the best one for different tracking scenarios while also developing insights about the challenges that cause tracking failures and how these can be overcome.
You are given 10 challenging sequences and your task is to find the tracker that works best on each of these.

Measuring tracking performance:
The performance of a tracker is defined in terms of both its accuracy and speed.

Tracking accuracy in each frame is measured by the mean corner distance (MCD) error between the ground truth and the tracked bounding boxes. This is defined as the mean Euclidean distance between the four corners of the two bounding boxes.
- The mean MCD error over all the frames is one possible metric to summarize tracking accuracy for a particular sequence.
- However, once a tracker fails, the tracking errors for the remaining frames often become meaningless because high DOF trackers tend to fail abruptly and their results make no sense after failure.
Therefore, instead of just using the mean MCD error, we compute the success rate (SR) which is defined as the fraction of frames where the MCD error is less than a given error threshold.
Further, as a single threshold is insufficient to summarize the tracking performance, we vary the threshold for a range of values from 0 to 20, compute the SR for each threshold and then take the mean SR over all the thresholds.
This gives a number between 0 and 1 that becomes the final measure of tracking accuracy. We can also generate and examine a plot of SR vs error threshold.
Tracking speed is measured by the number of frames per second (FPS) tracked by the tracker.
This does not include the time needed to capture and preprocess the images and only considers the time that the tracker takes to process each frame.

The overall performance of a tracker is a combination of its speed and accuracy. Accuracy is usually more important as speed can often be increased by faster hardware or more efficient implementation. Therefore, accuracy is given 4 times greater weightage than speed while computing the overall tracking performance in this assignment.

Evaluation:
The evaluation for this assignment will be relative, i.e. the tracker that gives the best value for a particular metric (i.e. maximum SR or maximum FPS) in a given sequence will receive full marks for that metric and sequence while the remaining submissions will be graded relative to it. This grading will be independent for each metric and sequence since a tracker that gives maximum accuracy in a particular sequence will not necessarily also give the best speed in that sequence. Same holds true for performance across different sequences.

All submitted trackers will be ranked by both their average SR and the average FPS for each sequence.
Marks will then be allotted out of 1 for each metric based on the ranking - the best tracker will get 1, the second best 0.9 and so on.
The overall marks for each sequence will be computed as:
(4*[SR marks] + [FPS marks])/5
so that accuracy is given four times the weightage of speed as mentioned before.
The marks for all 10 sequences will be summed and scaled to get final performance marks out of 6.
The remaining 4 marks will be given for a report in which you will document all the trackers you experimented with for each sequence along with the corresponding SR and FPS figures and any other observations you may have made.
- The report should be written in such a way as to clearly show how you got to the optimal tracker.
- The report is also subject to 2 bonus marks if it contains insightful analysis of the results with interesting conclusions about what types of trackers work well with which tracking challenge.
- Tracking challenges include, but are not limited to, illumination variations, reflections, shadows, extreme pose/viewpoint changes, occlusions and fast motion.
- Any ideas or suggestions for improving an existing module to handle these challenges better will be considered for bonus marking too and might also be used as the basis of your course project.
- If any particularly interesting or novel results are obtained, these might be used in a journal paper that is currently being prepared and the submitter will be a co-author.

What you are provided with:
You will be using only cfg files for setting the tracking parameters in this assignment. You are provided with 10 sets of cfg files and each one is pre-set to read images for the corresponding sequence. The cfg files are available here.

You can use these to track each sequence using:
$ runMTF config_dir ‹path to cfg files for that sequence›
Tracking result will be shown as a red box along with the corresponding ground truth in green.
Tracking error and FPS will also be shown on the top left corner of the image.
After the tracker is done running on a sequence, the average tracking error, FPS and SR will be printed on the terminal.
A summary of results will also be written to a file called tracking_stats.txt so results from multiple runs can be easily analysed.
The complete SR data for all thresholds will also be written to an ASCII text file whose name will be indicated on the terminal too. This file can be loaded in Matlab with the importdata command to generate the SR vs threshold plot.

Please refer to chapter 9 of MTF thesis, especially the performance summaries in Fig. 9.18 and 9.25 to get a general idea of the performance of different tracking modules.

Note, however, that this chapter only provides overall results combining 4 different datasets with over 200 sequences so these might not be good indicators of the best tracker for any given sequence.
Dataset specific results are also provided in Appendix B.

What to submit:
You can change the settings in these files and submit all 10 sets along with the report PDF as a single zip file.
Following are some guidelines about setting the parameters:

The main parameters that you should start experimenting with in mtf.cfg are mtf_sm, mtf_am, mtf_ssm and mtf_ilm that define the core tracking modules and mtf_res/res_from_size that define the sampling resolution.
Once you have chosen the core tracker configuration, module-specific settings for the chosen modules can be adjusted in modules.cfg.
MTF also provides a number of third party (non-registration) trackers that can also be used for this assignment. These typically work better than registration based trackers when the object to be tracked is not planar which is the case for some of the sequences.
Settings for third party trackers can be adjusted in thirdparty.cfg.
runMTF is a general purpose tracking tool and provides many functionalities like automatic tracker reinitialization on failure, frame skipping and image resizing that you are not allowed to use for this assignment.
All parameters for controlling these features are in mtf.cfg and following are some of those that must not be changed:
reinit_on_failure
reset_at_each_frame
reinit_at_each_frame
reset_template
frame_gap
img_resize_factor
init_frame_id
start_frame_id
end_frame_id
n_trackers
use_reinit_gt
use_opt_gt
overwrite_gt
tracking_err_type
This list is not necessarily exhaustive so if you have doubts about any parameter, please ask the TA.
Changes in any of these parameters will cause the submission to be disqualified and receive no marks for this assignment.

Running MTF on your laptop:
You can carry out the experimentation on your own laptops too by following these instructions:

Install MTF by following the instructions on its git repo.
The test sequences are available at ~vis/ugrad_vision_w13/data/lab3. You can copy these to your laptop and set db_root_path in mtf.cfg or over command line to point runMTF to it. All sequence files are also present as a single zip file called lab3_data.zip in this folder.
Note that your submitted cfg files must be able to run on lab machines to be evaluated. If your laptop configuration is different from lab machines, you may have access to optional MTF components that are unavailable there. For instance, lab machines have OpenCV 2.4.9 so if your laptop has OpenCV 3, you will have access to additional third party and feature trackers that won't work on lab machines.
If you want to run batch scripts to test many different combinations of methods, refer to the scripts in this repo for a general idea of how to get started. These are a bit outdated, undocumented and some might not work but should give you an idea of how to setup your own batch scripts.