Research

Relational Representation Learning for Video Understanding

Before a system can reason about actions, events, or long video narratives, it first needs a way to represent motion itself — and for a long stretch of the lab's history, that representation problem was treated as unsolved rather than assumed. Standard video architectures apply spatio-temporal convolution or attention uniformly across space and time, treating motion as something that emerges implicitly from stacked appearance features. This line of the lab's research instead argues that motion has its own structure — correspondence and relation between neighboring regions across time — and that structure should be modeled explicitly rather than left to convolution or attention to discover on its own.

본문

This program traces a clear conceptual progression. It begins by replacing the standard practice of pre-computing external optical flow — expensive and disconnected from the rest of the network — with a lightweight, trainable module that learns to establish frame-to-frame correspondences internally and convert them directly into motion features. The next step generalizes this idea beyond simple two-frame correspondence into a richer representation built on spatio-temporal self-similarity: each local region is represented not by its raw appearance but by its pattern of similarity to neighboring regions in space and time, which captures long-range and fast motion far more robustly than fixed convolutional kernels. This self-similarity idea is then reframed as a general feature transform problem, arguing that both convolution and self-attention are fundamentally limited for video because neither dynamically models the relational, correspondence-like structure that motion requires — leading to a relational feature transform that generates kernels from spatio-temporal relations directly. The most recent work in this line extends self-similarity to higher orders and to structured correlation patterns across vision transformer layers, pushing the same relational principle into modern transformer-based backbones.

Across this progression, the unifying commitment is treating motion and structure not as incidental byproducts of deep appearance features, but as an explicit, relational representation problem — one that recurs whether the underlying architecture is a convolutional network, an attention module, or a vision transformer.

Research

본문

Related papers