Research

Embodied AI & Language-Driven Robot Manipulation

The lab's embodied AI line investigates why end-to-end vision-language-action (VLA) models struggle with long-horizon manipulation, and proposes a modular approach centered on open-vocabulary affordance grounding rather than mapping instructions directly to low-level actions. The core hypothesis is that manipulation becomes tractable when it is decomposed into intermediate, physically grounded representations — where on an object a task-relevant region is, how a hand should contact it, and how the whole body should move to get there — rather than learned as one opaque instruction-to-action mapping. This is a deliberate move away from black-box, end-to-end learning toward a more analytical and explicit formulation of manipulation, where each intermediate representation can be inspected, interpreted, and reasoned about on its own terms.

본문

This modular approach spans the full pipeline from perception to action. One line of work addresses the data bottleneck behind open-vocabulary affordance grounding, since training such models requires far more annotated language-affordance pairs than exist naturally, building automatic pipelines to generate them at scale. A second line addresses a limitation inherent to real robots: cameras only ever capture partial, occluded views of an object, so this work reconstructs the unobserved 3D geometry and grounds affordances on the complete shape, including regions the camera never saw. A third line completes the pipeline from a grounded affordance region down to a physical grasp, introducing an intermediate representation that predicts which parts of the hand should contact which surface regions before generating the full hand configuration — explicitly bridging task semantics with physical feasibility rather than mapping straight from instruction to joint angles. A fourth line extends this same instruction-to-action structure from the hand alone to the whole body, using the wrist as an explicit interface between kinematically-tracked, contact-free body motion and separately learned, contact-rich finger behavior for humanoid manipulation. Underlying all of this is the planning problem: once an affordance and a grasp are grounded, the robot still needs to simulate and select among possible future actions, which motivates compact, explicit world-model representations that make decision-time planning computationally practical for real-time control.

Across this pipeline, the shared commitment is decomposing manipulation into grounded, inspectable intermediate representations — affordances, contacts, and compact world-model states — rather than treating instruction-following as a single end-to-end black box. This research direction is intended to grow beyond simulation and benchmark evaluation: the lab plans to extend this pipeline into robot manipulation research conducted with real robots in diverse, real-world environments, closing the loop from language grounding to physical deployment.

Research

본문

Related papers