Research

Open-vocabulary scene understanding reflects a paradigm shift away from recognition systems built around a small, fixed set of predefined categories. Classical segmentation and detection models can only recognize the specific classes they were trained on — a "chair" model has no way to name or localize a "stool" if that word was never in its label set. Open-vocabulary perception instead couples visual recognition with language, letting a model segment, ground, and reason about an arbitrary object category described in natural language at test time, whether or not that category ever appeared in training.

Active perception is grounded in a paradigm shift that computer vision underwent as it moved beyond passive, all-at-once scene analysis. Rather than treating a camera or a model as a fixed device that must take in an entire scene and try to make sense of everything it sees, active perception treats the observer as something that can choose — where to look, how to move through and interact with a scene, or which view to prioritize — in order to answer a specific question as efficiently as possible.

The lab's embodied AI line investigates why end-to-end vision-language-action (VLA) models struggle with long-horizon manipulation, and proposes a modular approach centered on open-vocabulary affordance grounding rather than mapping instructions directly to low-level actions. The core hypothesis is that manipulation becomes tractable when it is decomposed into intermediate, physically grounded representations — where on an object a task-relevant region is, how a hand should contact it, and how the whole body should move to get there — rather than learned as one opaque instruction-to-action mapping. This is a deliberate move away from black-box, end-to-end learning toward a more analytical and explicit formulation of manipulation, where each intermediate representation can be inspected, interpreted, and reasoned about on its own terms.

Understanding a visual scene often requires more than naming the objects in it — it requires parsing the scene into a structured representation of entities and the relationships between them. A scene graph makes this explicit: nodes for objects, edges for their relationships, turning an image into a graph rather than a flat list of detections. When the entities being related are people — interacting with objects, forming groups, or simply needing to be perceived accurately in the first place — this general scene-parsing problem becomes human-centric, and the lab's work in this space treats human-object interaction, group and social behavior, and even pose perception as specialized instances of the same underlying graph-parsing challenge.

Beyond recognizing what is in a single frame, video understanding requires reasoning about how content unfolds over time — segmenting a stream into meaningful units, anticipating what comes next, locating precise moments of change, answering questions that require following a narrative, and tracking specific objects reliably as they move and occlude. The lab has a sustained program across all of these temporal reasoning problems, and a defining trend across the group's recent work is treating generative modeling — diffusion in particular — and large language models not as end tasks in themselves, but as flexible backbones for representing and reasoning about temporal structure.

Before a system can reason about actions, events, or long video narratives, it first needs a way to represent motion itself — and for a long stretch of the lab's history, that representation problem was treated as unsolved rather than assumed. Standard video architectures apply spatio-temporal convolution or attention uniformly across space and time, treating motion as something that emerges implicitly from stacked appearance features. This line of the lab's research instead argues that motion has its own structure — correspondence and relation between neighboring regions across time — and that structure should be modeled explicitly rather than left to convolution or attention to discover on its own.

Finding reliable correspondences between images, shapes, and point clouds is one of the lab's longest-running research threads, tracing back over a decade to early work on semantic correspondence with object proposals and continuing through today's group-equivariant formulations. The core problem is deceptively simple to state — given two visual observations of related content, which parts correspond to which? — but solving it robustly across viewpoint change, intra-class variation, symmetry, and partial observability has proven to be a deep and recurring research challenge.