Lab 1: Optical flow

a (1). Capture images

Write a function to capture a sequence of images using Matlab .
Open Matlab and install following packages: here
vid = videoinput('dcam', 1, 'F7_YUV422_812x612_mode1');
vid.FramesPerTrigger = 1;
vid.ReturnedColorspace = 'rgb';
preview(vid);
in the command window to create a firewire camera pipeline, then you can capture one frame using:
start(vid);
stoppreview(vid);
imwrite(getdata(vid), 'image_path_to_save');
where success is a flag that indicates whether the image capture was successful.

Write a function to capture a sequence of images using Python .
import cv2
cam = cv2.VideoCapture(0, cv2.CAP_FIREWIRE)
ret_val, img = cam.read()
cv2.imwrite(filename, img)

b (2). Image differences

Compute the temporal image derivatives approximated by successive image differences. In the case of a fixed camera and a moving object (e.g. person walking by), try to reliably threshold the temporal derivative to extract only the moving object.

• If you do not use thresholding, what could mistakenly be identified as motion?
• You need to convert from colorful images to gray scale images before going to next step.

Solve for the optic flow. Write a routine that outputs the optic flow vectors given an image sequence. Try for both moving camera and moving objects. Chapter 9 from Stockman and Shapiro has a basic presentation of the required concepts: Motion from 2D Image Sequences. Read the introductory sections cursorily. Pay attention to the definitions in Section 9.3 and then closely look at the optic flow equations in Section 9.3.5

• Tile the image into blocks of user specified size (try sizes 4x4, 8x8, 16x16)
• For each block, consider if there is motion (Use the thresholds you derived above). If no temporal change set motion vector to 0.
• If there is temporal change, compute the spatial derivatives and solve for the x and y optic flow vector.
• Plot the flow vectors on top of the image using quiver.
• Submit a movie of your results (images with overlayed flow field). You can use getframe to create a movie. Please make sure the movie file is relatively small.

Example use of quiver and getframe

X = (1 : blockSize : imageWidth-1) + blockSize/2;
Y = (1 : blockSize : imageHeight-1) + blockSize/2;
[X,Y] = meshgrid(X,Y);
loop through all images (i){
  ...
  loop through all blocks (ix,iy){
    compute motion vectors ...
    U(ix,iy) = xMotion;
    V(ix,iy) = yMotion;
  }
  ...
  plot image;
  hold on;
  quiver(X,Y,U,V);
  hold off;
  M(i) = getframe;
}

movie(M);
save myMovie M;

If you are getting inconsistent results, you could try testing with artificial motion first. For example:

%im1 = image at time 1
%im2 = image at time 2
im2 = zeros(size(im1));
im2(1:end-1, 1:end-1) = im1(2:end, 2:end);

You may also find it useful to display some of your intermediate images/data.

• Can this optic flow account for any type of motion? If not, give two distinct cases when it will not work well.

Try to extract 3-6 dimensional motion. Parameterize motion using e.g. 4 (x,y,rotation,scale) and 6 (affine) parameters. How many motion parameters can be reliably extracted? How does the block size affect the results? Try both camera motion and object motion sequences. Compare results.

• Each time you may only be able to show 2dof by using quiver. You can show images two times if you are trying 4 dof, 3 times for 6dof.
• While visualizing your extracted motion parameters is difficult, you can also validate your solution by warping the image (or just individual blocks) based on the extracted motion parameters, and checking if the result is more similar to the next frame than if you computed the warped image using your solution to Exercise 2.

Test sequences (Use as desired — It is not mandatory for you to submit results on them)

• How to read and interpret the image sequences. matlab m- file
• A 60 image sequence with a jade tree Flower60im3. mat
• A 32 image sequence with articulated arm movement a rmD32im1.mat. This is the same sequence as used in the movie above.