Research

Open-Vocabulary Scene Understanding

Open-vocabulary scene understanding reflects a paradigm shift away from recognition systems built around a small, fixed set of predefined categories. Classical segmentation and detection models can only recognize the specific classes they were trained on — a "chair" model has no way to name or localize a "stool" if that word was never in its label set. Open-vocabulary perception instead couples visual recognition with language, letting a model segment, ground, and reason about an arbitrary object category described in natural language at test time, whether or not that category ever appeared in training.

본문

At the lab, this paradigm is pursued primarily in 3D scene understanding, where extending open-vocabulary recognition from 2D images to 3D scenes introduces its own data and efficiency bottlenecks. One line of work tackles the data bottleneck, building large-scale pipelines that automatically generate 3D region-text annotations from multi-view images and vision-language models, since manually labeling 3D scenes with open-ended language is prohibitively expensive. A second line tackles the efficiency bottleneck, replacing slow multi-stage pipelines that lift 2D predictions into 3D with proposal-free, single-pass architectures that segment a scene by open-vocabulary category directly and far faster. A third line questions how much text supervision segmentation actually needs up front, showing that discovering object regions visually first and only later assigning language labels can match or exceed classification-first approaches without additional training. This same open-vocabulary grounding machinery is also extended from generic object categories to actionable affordances, grounding not just what an object is but how it can be used.

Across these approaches, the shared commitment is decoupling what a model can recognize from a fixed training vocabulary, and doing so efficiently enough — in data construction, inference speed, and viewpoint coverage — to make open-vocabulary 3D perception usable as an infrastructure layer for downstream tasks, including embodied and robotic agents that must interact with categories no fixed label set could anticipate.

Research

본문

Related papers