Lab 1.1 Optical Flow

2-DoF optical flow on a soccer ball.

Overview

In this lab, you’ll implement a basic 2-DoF optical flow algorithm. These flow vectors describe the motion in a scene. Equation 1 is the basis of optical flow. We can capture the temporal shift between two frames by using a first order approximation:

\begin{equation} -I_t=\nabla I^T\mathbf{u} \end{equation}

where \(I_t\) is the temporal gradient, and \(\nabla I^T=\left[I_x,\ I_y\right]\) is the spatial gradient. For each image in our sequence, we want to find its temporal and spatial gradients and solve for \(\mathbf{u}\).

This zip file contains the sample videos you can use to complete this lab. However, you are encouraged to capture your own and evaluate which videos perform well and which perform poorly.

Important: You can obtain 2 bonus marks for each section you finish and demo while present during the lab period we introduce the lab.

0. Capturing Images with OpenCV

The following code grabs frames from a camera and displays them in a new window:

import cv2
cam = cv2.VideoCapture(0, cv2.CAP_FIREWIRE) # Remove second arg if using webcam
while True:
    ret_val, img = cam.read()
    cv2.imshow('test', img)

Modify this code to use imwrite to save a sequence of images we will use for this lab.

1. Single Window Optical Flow (20)

Complete the following steps for one timestep:

a) Temporal Image Gradient

  • Convert a sequenced pair of images to grayscale.
  • Find the temporal difference between the two images.
  • Threshold the temporal derivative by setting values that do not meet our threshold to zero.
  • Test different threshold values until motion is isolated and noise is minimized.

Deliverables:

  • Display the final thresholded temporal derivative in your report.

Report Question 1a: What does the temporal image gradient tell us about our image pair?

b) Spatial Image Gradient

  • Write a function that finds the spatial image gradients, \(I_x\) and \(I_y\), along the first and zeroth axes of our image, respectively. Compute an approximation of \(I_x\) and \(I_y\) using the definition of a gradient. Avoid using numpy.gradient for this exercise.

Deliverables:

  • Display these results in your report.

Report Question 1b:

  • What features are prominent in each spatial gradient?
  • Report Bonus (2) Include examples of images with prominent features and images without prominent features. How does this affect things like the condition number of the image and subsequently the quality of the computed (u)?

c) Solve for \(\mathbf{u}=[u,v]^T\)

  • Plot \(\mathbf{u}\) onto the center of our image using matplotlib.pyplot.quiver.

Deliverables:

  • Display these results in your report.

2. Single Window Optical Flow Live (20)

a) Implement single window optical using live video

  • Code in Question 0 gives the structure to retrieve live frames.

Deliverables:

  • Save a video of single window flow with the overlaid motion vector to include in your submission. Ensure the video is less than 10 mb.

Report Question 2: What is one limitation of finding a single optical flow vector on the entire window?

3. Optical Flow on Patches (20)

Complete the following steps for one timestep:

a) Divide Gradients into Patches

  • Divide your gradients into patches of size block_size x block_size.

Report Question 3a: How does changing block_size affect your results? What are some benefits and drawbacks of increasing block_size?

b) Find Flow Vectors for Each Patch

  • For each patch, find the flow vectors \(u\) and \(v\).

Report Question 3b: What sort of motion have we captured here? Name two types of motion we cannot account for with our current method.

c) Plotting

  • For each tile, plot the motion vector \(\mathbf{u}\) onto the center of the tile using matplotlib.pyplot.quiver.

Deliverables:

  • Display these results in your report.

4. Optical Flow Video (20)

a) Implement a live implementation of optical flow.

  • Explore what video characteristics enable good or bad performance for our algorithm.

Report Question 4a: Name three assumptions we make for optical flow to work.

b) Capture one video that results in ideal performance for your optical flow algorithm.

  • Apply your optical flow function to the video.

Deliverables:

  • Save the video with motion vectors to include in your submission. Please put it in a folder media/good. Ensure the video is less than 10 mb.

Report Question 4b: Why is this video suitable? Relate your answer to the assumptions named in Report Question 4a.

c) Capture one video that results in poor performance for your optical flow algorithm

  • Apply your optical flow function to the video. You may artificially generate a non-trivial video for this question.

Deliverables:

  • Save the video with motion vectors to include in your submission. Please put it in a folder media/bad. Ensure each video is less than 10 mb.

Report Question 4c: Why is this video unsuitable? What assumptions does your video violate?

5. Rotation and Scale (20)

This question is a bonus for undergraduate students.

Grad students: this question is mandatory.

a) Rotation and Scale Gradients

  • Write a function to find rotation and scale gradients \(I_r\) and \(I_s\) for a single image.

Deliverables:

  • Display these results in your report.

b) Solve Motion Vectors for Rotation and Scale

  • Write a function to solve for motion vectors \(r\) and \(s\) using a formulation similar to Equation 1 in a single window (i.e., no tiles) of a video with rotation and zoom with respect to the camera axis.
  • Plot your results by mapping the x-vector to rotation and the y-vector to scale.

Deliverables:

  • Save these results to include in your submission.
  • Report Bonus (2) Include examples of images with prominent features and images without prominent features. How do these differ from the prominent features you found in \(I_x\) and \(I_y\)?

6. Linear Taylor Approximation

This question is a bonus for all students.

Report Bonus (4) How accurate is your linear assumption? What features make the assumption better or worse? What would an image pair look like if H.O.T. was truly 0? Generate an image pair where H.O.T. is 0 and paste them into your report.

Submission Details

  • Include accompanying code used to complete each question. Ensure they are adequately commented.
  • Ensure all functions are and sections are clearly labeled in your report to match the tasks and deliverables outlined in the lab.
  • Organize files as follows:
    • code/ folder containing all scripts used in the assignment.
    • media/ folder for images, videos, and results.
  • Final submission format: a single zip file named CompVisW25_lab1.1_lastname_firstname.zip containing the above structure.
  • Your combined report for Lab 1.1 and 1.2 is to be submitted at the end of the topic at a later date. The report contains all media, results, and answers as specified in the instructions above. Ensure your answers are concise and directly address the questions.
  • Total marks for this lab is 80 for undergraduate students and 100 for graduate students. Your lab assignment grade with bonus marks is capped at 110%. Report bonus marks will be applied to the report grade, also capped at 110%.