Lab software
In the labs we will be using Matlab augmented with mexMTF.
Matlab is a package for matrix computations and linear algebra.
As images can be represented as matrices and geometry computations
are linear algebra this is a convenient tool. mexMTF allows us
to bring live video into Matlab, as well as provides real-time
tracking (used for point correspondence in the labs).
Matlab
Matlab is available on the lab machines with cameras attached
For remote logins you can also use matlab on the central server
ohaton. For the machines in labs machines the console user has first priority to the CPU cycles, so be mindful if you
log on remotely.
Useful Matlab links:
mexMTF
mexMTF is the Matlab interface to the MTF visual tracking library.
mexMTF commands are invoked using the MATLAB function
[success, out1, out2, ...] = mexMTF('command',p1,p2,...) where p1, p2, ... are the parameters required for the command and out1, out2, ... are the outputs produced, if any. The first output success is a flag that indicates whether or not the command invocation was successful.
mexMTF has both single and multi threaded versions where the latter, called mexMTF2, is recommended as being faster and more stable.
A complete example of using it can be found in
~vis/ugrad_vision_w13/src/MTF/Examples/matlab/runMTF2.m (or runMTF.m for the single threaded version).
Preliminaries: Setting your environment variables and path
- Put this line in your .bashrc:
source ~vis/ugrad_vision_w13/scripts/vision_exports
This will set up environment variables to access the MTF video and tracking software.
export MATLABPATH=$MATLABPATH:~vis/matlabdirs/
This will set up mexMTF path for matlab.
- It can be useful to test the camera with the linux software
coriander. Start coriander in a shell:
$coriander
Check that the camera name shows (e.g. "pyro" or "PtGrey Grasshopper")
shows. Click the services tab. Click receive then display.
When done with coriander, exit it. This will let other programs use the camera (such
as mexMTF).
Using mexMTF to capture images:
-
In a terminal type:
$ matlab
to start Matlab
-
In Matlab, create an MTF input pipeline as:
>>success = mexMTF2('init','pipeline v img_source f vp_fw_res 640x480');
for "new" point-grey firewire cameras at 640 x 480 resolution
OR
>>success = mexMTF2('init','pipeline c img_source u');
for USB webcams at 640 X 480 resolution assuming that no firewire cameras are attached.
-
This command only needs to be run once at the beginning of each Matlab session.
-
USB cameras have to be accessed using the OpenCV pipeline as the V4L2 module required by the ViSP pipeline is not installed correctly on lab machines.
OpenCV pipeline does not allow detailed manipulation of camera settings so this is the only available mode.
-
For initialization of firewire cameras at other resolutions and adjusting other settings like ISO, FPS and colour format, please refer to the description of MTF ViSP input pipeline on MTF configuration page.
-
As mentioned there, MTF parameters can be provided either in ASCII text files or as pairwise command line arguments. Both of these are supported by mexMTF too except that the latter are represented by a single optional string argument into which all pairs have been combined (as shown above).
-
Note that the FlyCapture SDK based ViSP Point Grey firewire pipeline (
img_source p
) does not work on lab machines so its corresponding parameters for setting camera shutter speed (
vp_pg_fw_shutter_ms
), exposure (
vp_pg_fw_exposure
), gain (
vp_pg_fw_gain
) and brightness (
vp_pg_fw_brightness
) are not available either.
-
Update pipeline, capture an image and show it:
>>[success, im] = mexMTF2('get_frame');
>>figure
>>imshow(im);
Using mexMTF to track objects:
- Create a tracker using:
>>tracker_id = mexMTF2('create_tracker','mtf_sm esm mtf_am ssd mtf_ssm 2', init_location);
-
This will create a tracker based on Efficient Second-order Minimization (ESM) search method, Sum of Squared Differences (SSD) appearance model and translation (2 DOF) state space model.
-
For other types of trackers supported by MTF, refer to the documentation for mtf_sm, mtf_am and mtf_ssm on its configuration page
-
As mentioned before, MTF parameters can be specified in cfg files or as a string of pairwise arguments supplied while creating the tracker.
In general, the best idea is to have most settings in cfg files whose location is set using config_dir and then provide tracker specific settings in the string.
A set of cfg files is present in ~vis/ugrad_vision_w13/src/MTF/Config/ which can be copied to a local folder and then modified.
-
tracker_id will be 0 if tracker creation was unsuccessful so it doubles as the success flag. An error message may also be produced.
-
init_location is either a 1x4 or 2x4 matrix. In the former case, it specifies a rectangle as [x, y, width, height] where (x, y) are the coordinates of the top left corner of the rectangle. In the latter case, it specifies an arbitrary quadrilateral whose x, y coordinates are arranged as: [top left, top right, bottom left, bottom right] such that the x and y coordinates are in the first and second row respectively.
-
If init_location is omitted, the user will be asked to select the initial object location interactively.
By default, this window will update the frames while the object is being selected but this can be disabled by setting mex_live_init to 0 if, for instance, tracking is to be performed on a pre-recorded video sequence so that initialization is to be performed in the first frame.
Note, however, that the input pipeline still captures new frames continuously from the time it is created. If new frames are to be captured only when it is accessed from Matlab, please use the single threaded version of mexMTF instead.
-
Creating a tracker also creates a new window where the result of tracking is shown in each frame. This can be disabled by setting mex_visualize to 0 if, for instance, Matlab is to be used for visualization. If MTF visualization is enabled, tracking can be stopped by pressing the Escape key while the window is in focus. However, the window should not be closed by clicking on the close button as this causes Matlab to crash.
- Update tracker and obtain the current location of the tracked object:
>>[success, corners] = mexMTF2('get_region', tracker_id);
Depending on the tracker settings, the raw RGB image obtained from the input pipeline might need to be converted to grayscale before passing it to the tracker.
- Remove a tracker:
>>success = mexMTF2('remove_tracker', tracker_id);
-
Delete all pipelines and trackers:
>>success = mexMTF2('quit');
Using MTF without Matlab:
If Matlab crashes a lot while using mexMTF or the tracking results are not needed for online processing in Matlab, MTF can also be run directly from the terminal using this command:
$ runMTF ‹pairwise_arguments›
where the pairwise_arguments are the same as passed to mexMTF with init and create_tracker commands except that both must be combined into a single command. For example, to start tracking a manually selected object on firewire camera pipeline, following command can be used:
$ runMTF pipeline v img_source f vp_fw_res 640x480 mtf_sm esm mtf_am ssd mtf_ssm 2
A pre-recorded video can be tracked using:
$ runMTF actor_id -1 img_source m db_root_path ‹path to video file› seq_name ‹video file name without extension› seq_fmt ‹video file extension›
-
This command will attempt to read the initial object location from a ground truth text file in the same folder as the video file and having the same name but with extension txt.
Each line in this file must contain the object location for the corresponding frame in the sequence in the format:
frame ulx uly urx ury lrx lry llx lly
.
-
Several benchmark dataset sequences along with the associated ground truth are available on MTF website.
-
If the ground truth file is not found, the user will be asked to select the object manually.
Tracking results can be saved in a text file in the same format as the ground truth by setting
write_tracking_data to 1.
The output file name can also be specified using
tracking_data_fname.
If this is not provided, the file name will be constructed automatically from the tracker settings.
This file can be loaded in Matlab for further processing of the tracking result using:
>> tracking_res = importdata(‹result file name›)
tracking_res.data will then contain the object locations for all frames as an N x 8 array where N is the number of tracked frames.
As mentioned before, it is usually more convenient to set most of the parameters in cfg files and change only a few (or none) for each run. In that case, MTF can be run as:
$ runMTF config_dir ‹location of the folder containing the cfg file›
For instance, the default cfg files can be copied as:
$ mkdir ~/mtf_cfg
$ cp ~vis/ugrad_vision_w13/src/MTF/Config/*.cfg ~/mtf_cfg
and then MTF called to read parameters from the copied files as:
$ runMTF config_dir ~/mtf_cfg
If other command line arguments are supplied too, then
config_dir
must be the first one.
When something doesn't work
If you get an error message "no camera found" in coriander, or
MTF will not work, one of the following can help:
- Quit programs using the camera and potentially hogging the
interface.
- If the problem still persists, try reloading camera drivers:
- For webcams - first try resetting using coriander, if this does not work try plugging out and then plugging back in the camera.
- For "new" Point Grey cameras - ALWAYS use coriander to reset these cameras. NEVER plug out these cameras, they are very expensive and get ruined when the screw lock connectors are repeatedly connected and disconnected.
- As a final resort log out of your account and then log back in (this will almost always fix the problem), but NEVER reboot the computers (IST does NOT like this).