Lab 4: 2D projective geometry and homography estimation

Consider the 3D point (10,10,10). Perform the following operations first using the affine formulation of the transforms (3 coordinates) and then rewrite them in homogeneous form (4 coordinates) . Draw a sketch with the initial and the transformed point (one figure).

• translate point 20 along positive Y axis
• rotate point around Z axis 30 deg
• rotate point around Y axis -10 deg
• write the resulting composed affine and homogeneous transformations to perform the operations in the listed order and simplify to the canonical affine and homogeneous form respectively.

What is the corresponding transformation if you perform the same operations to transform the coordinate frame? Write the homogeneous and affine representation of the transformation. Draw a sketch of the initial and transformed coordinate system (one figure).

• translate origin 20 along positive Y axis
• rotate coordinate system around Z axis 30 deg
• rotate around Y axis -10 deg
• write the resulting composed affine and homogeneous transformations to perform the operations in the listed order and simplify to the canonical affine and homogeneous form respectively.

Write a matlab function that takes a 3-D point, a translation vector and a rotation vector as input, and returns the new coordinate of the point and the composite transformation matrix T. Use this function to repeat the exercise above.
Note: Consider the rotations like in the following figure. Your rotation vector will be a vector of the rotation angles around each coordinate axis. Apply the rotations in the appropriate order to get the same results from your MATLAB function as the results you derived by hand.

Refer to Lecture 4

Exercise 2: 2D projective Geometry: Points, lines and their incidence. (3)

References: lec06ProjG2D.pdf slides 3-5 and lec01bCRV16-2018.pdf slides 46 - 47

a. Construct the mid-point and mid-lines of the football field in the image below. (1)

• Click on the corners of the field using ginput.
• Convert the points to homogeneous coordinates.
• Compute the diagonal lines l₁ and l₂ (line between two points: l = p₁ x p₂).
• Compute the mid point using the diagonal lines: p_m= l₁ x l₂.
• Compute vanishing point as the point of intersection of either of the two opposite edges of the field: p_∞ = l₃ x l₄
• Compute mid-line using intersection of mid-point and vanishing point: l_m = p_∞ x p_m.

b. Constructing mid line in video sequence (1)

• Place a book or poster with a textured cover on the table. Position the camera to stream video of the book.
• Initialize 4 MTF trackers on the 4 corners: user should be able to click on the 4 corners and the trackers should be initialized on a patch with the clicked point as the center and its size specified as a parameter.
• Write a program that reads the trackers and draws graphic marking on the video to plot the center point on the book, and the line going across the middle of the book.
• Move the camera, and verify that although the video projection changes, the markings stay fixed on the object.

• Land a helicopter UAV in the middle of an asphalt rectangle.
• Land a fixed-wing UAV along the midline of a road.
• In Augmented Reality (AR), we could use a projector to project lines and points in the real world.

c. Parallel lines and line-to-line (1)

• Use e.g. two books with textured covers or papers / posters. Lay the books on a table a bit away from each other.
• Construct the parallel lines constraint.
  • 
  • parallel lines: epar(l) = (l₁ x l₂) x (l₃ x l₄)
• Move one book and track two lines on it using trackers initialized at the endpoints of the lines as in part (b). The lines on the other book do not need to be tracked, but can be selected in the first frame and assumed to be fixed, as the book is stationary.
• Display the value of the constraint in the image (as a text annotation). Note that it is vector-valued. Verify that its magnitude decreases when the books are near parallel, and that its direction changes when the books cross the point of being parallel.
• Similarly construct the line-to-line constraint, and display its value in the image.
  • 
  • line-to-line: ell(y) = y₁ . (y₃ x y₄) + y₂. (y₃ x y₄)
• Verify that both constraints work both when the books are side-by-side and on top of each other.
• The projective lines are ideal, infinitely long lines. Not limited to the physical edge of the book / object.
• To avoid having very large numbers for the constraints, normalize the points and lines before computing the constraints.

• Computer vision-guided robot control (as in the images above).
• Building inspection: For commercial buildings a final inspection is done before hand-over from the builder to the new owner to ensure doors, windows walls etc. are all lined up correctly.

Reference: lec06ProjG2D.pdf, slides 40-44

Given two images of a planar scene and a set of four or more 2D point correspondences between the images ( x_i <--> x'_i ), compute the homography, H, which relates one image to the other, and then apply H to the first image to see if you get similar image as the second one.

Some useful commands in matlab to select files and points: imgetfile, cpselect, uigetfile, getpts and possibly many others. Use doc in matlab command window or google the relevant commands.

(a). (1.5) Write a matlab program that displays two images. Allow the user to select four or more points in each image. Use these correspondences to estimate the homography by the direct linear transform algorithm (DLT). Apply the homography to the first image, such as using imwarp. For imwarp, you may need to use the transpose of your homography matrix, and may want to set the 'OutputView' argument to preserve the size of the image.

(b). (1) Recall that in DLT we need to constrain the magnitude of the homography H to be strictly positive, which is usually done by constraining the Euclidean norm of H to be 1 (as in (a)). We also need a loss function on the residual Ah, which is again usually done by the Euclidean norm (as in (a)). Replace the loss function by the L1 norm (sum of absolute values) and do (a) again.

A numerical procedure to solve the L1 norm based DLT is outlined here. Graduate students are welcome (but not compulsory) to come up with their own numerical procedures.

(c). (1) Add the normalization procedure into the DLT algorithm (in both (a) and (b)): For undergraduate students, translate the points such that they center around the origin, then scale the points such that their sum of distances to the origin is fixed to root 2; For graduate students, use principal component analysis to center the points and normalize the variances in different principal axes to unity. Please do NOT forget to apply the inverse normalization after estimating the homography.

(d). Discussion (1)

Compare (a) and (b) by deliberately picking some pair(s) of points that do not correspond to each other (i.e., outliers). Here is what I will do for the comparison: Choose perfect correspondences first and estimate the homography H by (a) and (b), then ruin some correspondence(s) and run (a) and (b) again, how significantly will the estimated homography change? (You can apply the homography to the image and see how the recovery changes). Repeat what you have done with (c) turned on.

Use the sample images below or capture your own.

Provide comments for each step of your implementation. I expect your implementation to have flags that can switch between (a) and (b), and can turn normalization (c) on and off.

• After the homography is applied, do the overlapping areas in the two images look exactly the same? Why / why not?
• Correspondences are not always perfect. What happens if some of the correspondences are incorrect? How can this be handled?
• This method is being used to align two images of a planar surface. Can it be used to align images of non-planar objects? What conditions need to be met?
• What if we only have four or more corresponding pairs of lines, instead of points? Or some pairs of points and some pairs of lines? Will the algorithm still work (under obvious modifications)?

• Track 4 points on a textured planar surface (e.g. poster, book cover).
• Compute the homography in real time.
• In the camera video, project a point or line drawing onto the book cover. Move the camera and verify that the drawing stays fixed.
• Variation for 1 bonus mark: Project a new cover image on the book. Verify as above.