The HyperMask Project

Description

HyperMask is a system which projects an animated face onto a physical mask, worn by an actor. As the mask moves within a prescribed area (the stage), its position and orientation are detected by a camera, and the projected image moves with it. Also, if the orientation of the mask changes, the projected image changes with respect to the viewpoint of the audience. The lips of the projected face are automatically synchronized in real time with the voice of the actor, who also controls the face's expressions.
As a theatrical tool, HyperMask enables a new style of storytelling, and we plan to demonstrate this style in a short performance. However, the same technology also has applications in offices, schools and theme parks. For example, in the so-called ``Office of the Future'', we would like to be able to project dynamically images and information onto moving, irregularly-shaped objects.

Technologies in use

Calibration (camera, projector)
Tracking system
Movement prediction & noise reduction
Pose estimation
Real-time Lip Synchronization
Face animation system, emotion controller
etc.

Team and contributors

Kim BINSTED, Sony CSL Inc.
Shigeo MORISHIMA, Seikei University
Frank NIELSEN, Sony CSL Inc.
Claudio PINHANEZ, MIT MediaLab
Tatsuo YOTSUKURA, Seikei University
Will UTHER, Carnegie Mellon University

Publications

HyperMask: Virtual Reactive Faces for Storytelling, Emerging Technologies: The Millenium Motel, Siggraph 1999.
Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track, Siggraph 1999.
Hypermask: Projection onto 3d moving surfaces, The Interactive Experiences, Interact 1999.

Live demos at Siggraph 1999, Interact 1999 and Sony CSL.

Photo & Movies album. Soon!