The report presents the MIT Media Lab’s Perceptual Computing Section’s effort to create computers that can perceive and understand human actions in real time. Central to this effort are Interactive Video Environments (IVE), instrumented office spaces that provide synchronized video and audio streams for multimodal interpretation. The paper describes a modular architecture in which separate processing modules run on dedicated SGI Indy machines and communicate over video, audio, ISDN, and Ethernet. The core modules are: (1) Pfinder, a real‑time 3‑D tracker that models a person as a set of color‑spatial Gaussian blobs and updates them with Kalman‑filter predictions; (2) a face‑recognition system that uses eigen‑faces and maximum‑likelihood detection to locate and identify individuals with 99 % rank‑one accuracy; (3) an expression‑recognition system that builds spatio‑temporal motion‑energy templates for facial expressions, achieving 98 % overall recognition; (4) vision‑directed audio that steers a microphone array by head position and renders 3‑D sound through a binaural spatializer and transaural system; (5) action‑recognition pipelines that treat human behavior as a hidden Markov model, demonstrated on real‑time American Sign Language classification and driver‑intention detection (86 % accuracy within 0.5 s, 97 % within 2 s). The paper concludes with several applications that exploit these capabilities, such as gesture‑controlled interfaces, avatar‑based telepresence, and immersive virtual environments. Throughout, the authors emphasize the importance of removing restrictive input devices and enabling natural, unencumbered interaction.
Topics
Why Perception‑Enabled Computing Is Essential
The introduction argues that traditional computers are “deaf and blind,” relying on keyboards and mice, which limits their ability to act as intelligent assistants.
Interactive Video Environments (IVE) as Real‑Time Testbeds
Describes the networked, instrumented office spaces that provide synchronized video and audio streams, and the modular SGI Indy architecture that supports interchangeable perception modules.
Pfinder: Real‑Time 3‑D Tracking via Gaussian Blob Models
Details the statistical representation of a person as a set of spatial‑color Gaussian blobs, the scene texture model, Kalman‑filter prediction, likelihood estimation, and the support‑map segmentation loop that yields head, hand, and foot positions at 10 Hz.
Eigen‑Face Based Face Recognition with Maximum‑Likelihood Detection
Explains the two‑stage detection pipeline (multiscale head search, warping, contrast normalization) and the use of a low‑dimensional eigenspace to compute likelihoods, achieving 99 % rank‑one recognition.
Facial Expression Classification Using Motion‑Energy Templates
Introduces spatio‑temporal motion‑energy templates for six basic expressions, the template‑matching classification process, and experimental results (98 % overall accuracy).
Vision‑Directed Audio: Beamforming and Transaural Rendering
Describes how head position from Pfinder steers a microphone array (electronic beamforming) and how a binaural spatializer combined with a transaural system creates 3‑D sound for the listener.
Action Recognition via Hidden Markov Models
Presents the HMM framework for inferring internal human states from multimodal cues, with two concrete systems: real‑time ASL classification (near‑perfect on a 40‑word subset) and driver‑intention detection (86 % within 0.5 s, 97 % within 2 s).
Applications: Gesture Control, Avatars, and Telepresence
Shows how the perception modules enable natural gesture‑based interfaces, shared virtual spaces, low‑bandwidth avatar telepresence, and immersive games such as SURVIVE.