Markerless Motion Tracking

Faculty: Song Zhang

Graduate Mentor: Ken Kopecky

Interns: Kelly Byron, Linh Pham, Joshua Situka

Abstract: Current motion tracking typically involves multiple cameras, markers, or other sensors. This project focuses on markerless motion tracking using a single camera and mathematical prediction models and computer vision techniques to track, for example, eyes, mouth, hands in real time with high precision. This challenge is particularly important for facial features because the brain's face detection systems are so fine tuned. When implemented successfully, a system like this can be used by a human to control an avatar much like a real-world, highly-realistic puppet.

Introduction: Marker-based tracking is commonly used in films and video games. The technology they commonly used is known as "Motion Capture", or "Mo-Cap". The physical markers have to been put on real human body or face, multiple cameras are used to capture the markers and the movement of the marker (motion), and the computer is used to generate 3D model having the same motion at the marker locations. This means that only the geometry motion at the marker points are known, while the rest has to be animated by the computer (or the modeler). Even though it is a very painful process for movies and games, they still used until recently, such as Polar Express, and Beowulf Beowulf.

Recently, Mova recently developed a less painful motion capture technology, which sprain random patterns "speckles" on human face (or body). It is a much better technology that captures more detailed motion. The most recent movie Benjamin Button used this technology for their movie. However, it is still a marker-based technology.

Goal: The research objective is to develop a tracking system without using markers. A single camera is used to capture the motion, and the mathematical prediction models and computer vision techniques to track to the motion and generate the photo-realistic animated 3D models. This technique is very useful especially when we combine this tracking technique with my 3D motion capture system (see my lab), when both 2D images and 3D geometries are captured at the same time. Below shows typical frames of human facial expressions captured by my system.

• Markerless_Motion_Tracking_poster_final.pptx: Markerless Motion Tracking Poster