Abstract
Many human activities follow a loosely defined script in which individuals assume roles. Encoding such scripts in a formal representation makes it possible to build systems that observe and understand human activity. In this paper, we first present a conceptual framework in which scripts for human activity are described as scenarios composed of actors and objects within a network of situations. We provide formal definitions for the underlying concepts for situation models, and then propose a layered, component-based, software architecture model for constructings systems to observe human activity. Both the conceptual framework and architectural model are illustrated with a system for real-time composition of a synchronized audio-video streams for recording activity within a meeting or lecture.