Research

Scene Graphs & Human-Centric Scene Parsing

Understanding a visual scene often requires more than naming the objects in it — it requires parsing the scene into a structured representation of entities and the relationships between them. A scene graph makes this explicit: nodes for objects, edges for their relationships, turning an image into a graph rather than a flat list of detections. When the entities being related are people — interacting with objects, forming groups, or simply needing to be perceived accurately in the first place — this general scene-parsing problem becomes human-centric, and the lab's work in this space treats human-object interaction, group and social behavior, and even pose perception as specialized instances of the same underlying graph-parsing challenge.

본문

The central technical difficulty scene graph generation poses is combinatorial: given every possible pair of detected objects in an image, only a small fraction have a meaningful relationship, and treating every candidate pair as equally worth reasoning about drowns out the handful that matter. The lab's approach to general scene graph generation tackles this directly by decoupling the problem into two stages — first selecting which object pairs are plausible edges worth reasoning about at all, and only then applying rich contextual attention, jointly over node-to-node, node-to-edge, edge-to-node, and edge-to-edge relations, to disambiguate what the relationship actually is. This same edge-selection-then-context structure reappears once the entities are specifically human. Human-object interaction (HOI) detection is a scene graph restricted to a single, well-defined relation type — a person acting on an object — and the lab's work here moves from a rich context-exchange architecture that lets human, object, and interaction branches inform each other through unary, pairwise, and ternary relations, toward zero-shot settings where the challenge becomes adapting large vision-language models like CLIP — strong at coarse image-level semantics but weak on the fine-grained local detail needed to tell interactions apart — by injecting locality and interaction awareness directly into their representations. Group and social interaction detection extends the graph one level further, parsing not just pairwise relations but cluster structure among people: first building a practical benchmark and model for detecting an unknown number of social groups and their activities in realistic scenes, then pushing further to a bottom-up formulation that infers groups from fine-grained interpersonal cues like body parts and gaze rather than treating each person as a single holistic node. Beneath all of this sits a prerequisite: a scene graph over people is only as good as the perception of the people themselves, which motivates the lab's work on human pose estimation robust to extremely low-light conditions, built on a dedicated camera system and paired well-lit/low-light dataset.

Across these strands, the shared commitment is parsing visual scenes into structured, relational representations — general object graphs and their human-centered specializations alike — rather than treating a scene as an unstructured collection of independent detections.

Research

본문

Related papers