University of Alberta

           CMPUT610: Hand-Eye Coordination and Vision-based Interaction

 Martin Jagersand
CMPUT610: Hand-Eye Coord & Visual Interaction by Martin Jägersand

Schedule:

Winter 2002, Tue, Thu 9:30-10:50am in CSC-B41
(Basement in the new Comp Sci building)

Syllabus

Participation:

Course participation consisits of:
  1. Following readings on schedule, posing questions, and actively participating in lectures.
  2. Three small warmup labs to get familiar with the lab environment.
    Topics: Intro to matlab and image processing. Capturing images and tracking motion. Animation and control of a linked structure (arm).
  3. An individual project. Either:
    1. Do a significant literature research project. You are welcome to propose your own topics, or we'll meet and discuss your interests and then I can help with suggestions.
    2. Do a practical project in vision/robotics consistent with the course.
    For the project you will:
    1. Hand in a written proposal.
    2. Prepare one presentation on your topic for the class.
    3. Hand in a written final report (about 6-8 pages in the format and style of a research report.

Examination:

You will present your findings and progress during the course, and write a final report. Grading: 50% Project, 30% labs, 20% Active class participation.

People not taking it for credit (sitting in or auditing) are expected to keep up with the course readings, and maybe do a smaller reading project.

Lecture notes

  • Lecture 1: Introduction and overview html,
  • Lecture 2: Motion and optic flow html,
  • Lecture 3: Subspaces for optic slow html,
  • Lecture 4: PCA and linear subspaces html,
  • Lecture 5: Intro to biological vision html,
  • Lecture 6: Biological image flow and motion detection html, Some demos.
  • Lecture 7: Project plans and discussion ,
  • Lecture 8: Vision based motion control html,
  • Lecture 9: Estimation of Motor-Visual functions html,
  • Lecture 10: Visual task definition html,
  • Lecture 11: Other sensory control: force html,
  • Lecture 12: Intro to motor physiology and anatomyhtml,
  • Lecture 13: Temporal superposition of movement html,
  • Lecture 14: Eye movements and the Ocular-Motor system html,
  • Lecture 15: Hand-Eye systems Sw Eng html,
  • Lecture 16: Deciphering the Human Visuomotor System. Cheryl html
  • Lecture 17: Model Independent Visual Servoing. Zhen html, ,
  • Lecture 18: Image Based Animation, Keith, html
  • Lecture 19: Tracking and XVision, Chris
  • Lecture 20: Robot arm hardware, Sean, .html
  • Lecture 21: Vision-Based Interactive Systems html,

Readings:

  • Image variability and Motion We'll start by studying the classic x,y optic flow, then gernralize it to include translations, rotations and scale:
    1. Chapter 9 from Stockman and Shapiro: Motion from 2D Image Sequences. Read the introductory sections cursorly. Pay attention to the definitions in Section 9.3 and then closely look at the optic flow equations in Section 9.3.5. Notice that we can write image variability in more than x,y image plane translations as in Eq 1 in Hong Zhou's report (just look at Eq. 1. No need to read the rest unless you want to)
    2. Efficient Region Tracking With Parametric Models of Geometry and Illumination (with P. Belhumeur). IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(10), pp. 1125-1139, 1998.  (34 pages, 2.7M compressed postscript) © IEEE Greg and Peter look at image variability in a quite general form, but it is essentially the same as in Hong's report. We'll study the image variability modelling in Section 2 early in the course, then follow up later with the tracking.
    3. Jägersand M. Image Based View Synthesis of an Articulated Agent In Proc of CVPR '97, also TR 595, Computer Science, Univ. of Rochester.
    4. Black, M.J., Yacoob, Y., Jepson, A.D., and Fleet, D.J. (1997) Learning parameterized models of image motion. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, June, pp. 561-567 (compressed postscript) How to learn a low dimensional subspace which organizes the flow of individual points.
    5. J.L. Barron, Fleet, D.J., and Beauchemin, S. (1994) Performance of optical flow techniques International Journal of Computer Vision, 12(1):43-77. An overview paper for those interested in what other optic flow techniques there are.(Cursory reading sufficient)
    6. J.L. Barron, and Beauchemin, S. Computation of optical flow Complementary reading to lectures.
  • Biological visual processing and motion detection.
    1. Review of the human visual system: Read through The eye and Visual pathways . Focus on functional aspects rather than anatomy and names. (These readings are written for med students). Also the first few sections of Kaiser provide a good overview of early visual processing in humans.
    2. Overview of human motion detection . Feel free to check the other nice demos on Johannes home page .
    3. Chapter 2 of Abbot and Dayan provides a more detailed and formal description of neural processing in visual cortex.
    4. Directionally Selective Complex Cells and the Computation of Motion Energy in Cat Visual Cortex Emerson, R. C., Bergen, J. R., and Adelson, E. H. Vision Research, 32:203-218 (1992). pdf   Postscript.z
    5. (Optional reading) "Computing Optical Flow Distributions Using Spatio-Temporal Filters" Simoncelli, E. P., and Adelson, E. H. MIT Media Laboratory Vision and Modeling Technical Report #165 (1991) Abstract pdf , Postscript.z,
      "Spatiotemporal Energy Models for the Perception of Motion" Adelson, E. H., and Bergen, J. R. Journal Optical Society of America A, 2(2):284-299 (1985). Abstract   pdf   Postscript.z. These two papers make the connection between the Reichardt detector used to model motion detection in humans and animals, and the optic flow equations favored in computer vision.
  • Vision and other sensory based control.
    1. Start by reading through Chapter 1 and 2 of Alexa's thesis.
    2. Try to understand in detail sections 3, 3.1, 3.2, and 3.3 of this tutorial to uncalibrated control.
    3. Read through Image Based Visual Simulation and Tele-Assisted Robot Control by Jägersand M. In IROS '97, Proc. New Trends in Image Based Visual Servoing.
    4. Try to get the idea of parallel and serial compositions of primitives for composing tasks from our paper.
    5. (Probably optional reading, haven't decided) Visual-motor and other viewing models, Vision and force hybrid motion control , IROS 2000. Jeanelle and Gary's paper considers a modified Newton method to account for time dependencies.
  • Human motor control.
    1. Tutis tutorial on Motor Cortex, (?), PDF
    2. Chapter 3 and 4 of Alexa's thesis.
    3. Tutorial on eyemovements, (?) (PDF)
    4. (Optional reading) Jeff Pelz thesis Visual Representations in a Natural Visuo-motor Task
  • User interfaces for robots and other machines.
    1. Kunioshi et al: "Seeing, Understanding and Doing Human task", "Robot See, Robot Do, Ikeuchi et al: "Assembly plan from observation.
    2. Polly K. Pook, "Teleassistance: Using Deictic Gestures to Control Robot Action", TR 594 and Ph.D. Thesis, Computer Science Dept., U. Rochester, September 1995. 95.tr594thesis.Teleassistance_Deictic_gestures_control_robot_action.ps.gz

Course and Reference Literature

Hand-Eye Coordination

Computer Vision

Human Vision and Neuro Science

Applications and Demos

Supplementary and introductory material