Research

Actions, Events & Long-Horizon Reasoning from Videos

Beyond recognizing what is in a single frame, video understanding requires reasoning about how content unfolds over time — segmenting a stream into meaningful units, anticipating what comes next, locating precise moments of change, answering questions that require following a narrative, and tracking specific objects reliably as they move and occlude. The lab has a sustained program across all of these temporal reasoning problems, and a defining trend across the group's recent work is treating generative modeling — diffusion in particular — and large language models not as end tasks in themselves, but as flexible backbones for representing and reasoning about temporal structure.

본문

This program spans several closely related problems. One line unifies action segmentation and long-term action anticipation, treating both as instances of a single sequence-completion problem — segmenting the observed part of a video while generating plausible predictions for its unobserved future — first through a query-based transformer and more recently through a diffusion model that handles both visible and invisible portions of a sequence within one framework. A second line detects generic event boundaries, the natural points where a video shifts from one meaningful unit to the next; because human annotators often disagree on exactly where these boundaries fall, this work moves from a single deterministic prediction toward a generative formulation that produces diverse, plausible boundaries, and separately toward processing video boundaries online as it streams in rather than after the fact. A third line tackles video-language reasoning, decomposing complex video question answering into interpretable modular stages of event parsing, grounding, and reasoning rather than a single opaque model, and using large language models to evaluate and aggregate frame importance for coherent video summarization. A fourth line addresses video object segmentation, extending promptable segmentation to remain robust under real-world visual corruptions rather than assuming clean, idealized video input.

Across these strands, the shared thread is representing temporal structure — segments, boundaries, questions, and object identities — through generative and language-model-based reasoning, treating time itself as something to be modeled probabilistically and interpretably rather than predicted as a single deterministic output.

Research

본문

Related papers