1994 visual-perception-of-human-bodies-and-faces-for-multi-modal
The paper presents a suite of computer‑vision techniques developed at the MIT Media Lab for real‑time, non‑invasive perception of human users in multimodal interfaces. The authors describe a three‑stage pipeline: (1) figure‑ground segmentation using chromatic or motion differencing and connected‑component analysis to isolate a user from the background; (2) an attention and tracking module that localizes the user’s head and hands in 3‑D based on geometric cues and context‑dependent search heuristics; (3) gesture and expression analysis that employs both user‑independent point‑location methods and user‑dependent view‑based distributed representations. View‑based models consist of a set of 2‑D canonical poses (“views”) whose distances to the current image are used for recognition via spatial and temporal clustering, dynamic‑time‑warping, and for driving a physically based facial animation system through a radial‑basis‑function mapping. The system is demonstrated in two prototypes: the ALIVE system, which enables full‑body wireless interaction with a virtual world, and a talking‑head application that synchronizes lip‑shape and facial expression with speech. Reported recognition rates exceed 90 % for user‑specific gestures, and the implementations run in real time on hundreds of users in an unconstrained office setting. The work illustrates how vision can complement speech in multimodal interfaces, providing richer cues for both perception and generation of human communication.
Topics
Figure‑ground segmentation for non‑invasive user isolation
Chromatic and motion differencing combined with connected‑component blob analysis to separate the user from static or dynamic backgrounds.
Attention and 3‑D tracking of head and hands
Context‑dependent search heuristics and geometric projection that locate key body parts in three dimensions using the bounding‑box size and pose estimates.
View‑based distributed representation for hand and face modeling
A set of 2‑D canonical pose views is learned via clustering; distances to these views form feature vectors for spatial and temporal gesture recognition.
Dual‑level gesture recognition: user‑independent vs user‑dependent pipelines
User‑independent point‑location recognition combined with user‑specific view‑based models that employ dynamic‑time‑warping and spatio‑temporal pattern matching.
Expression generation through view‑distance to facial animation mapping
Radial‑basis‑function mapping from view‑distance scores to motor parameters of a physically based skin‑and‑muscle face model, enabling real‑time talking‑head synthesis.
ALIVE system: full‑body wireless interaction with a virtual world
A demonstration platform where isolated users control and are perceived by virtual creatures in real time, validating the vision pipeline with over 500 users.
Vision‑speech integration for multimodal interfaces
Discussion of how visual cues (lip‑reading, facial expressions, gestures) complement acoustic signals to improve speech recognition and user interaction.