Towards Generalizable and Intelligent System for Robotic Manipulation

Part of the Special ECE Seminar Series

Modern Artificial Intelligence

Title:

Towards Generalizable and Intelligent System for Robotic Manipulation

Speaker:

Hongyang Li, University of Hong Kong

Abstract:

A generalist robot should perform effectively across various environments. However, most existing approaches heavily rely on scaling action-annotated data to enhance their capabilities. Consequently, they are often limited to a single physical specification and struggle to learn transferable knowledge across different embodiments and environments. To confront these limitations, we propose UniVLA, a new framework for learning cross-embodiment vision-language-action (VLA) policies. Our key innovation is to derive task-centric action representations from videos with a latent action model. This enables us to exploit extensive data across a wide spectrum of embodiments and perspectives.

To mitigate the effect of task-irrelevant dynamics, we incorporate language instructions and establish a latent action model within the DINO feature space. Learned from internet-scale videos, the generalist policy can be deployed to various robots through efficient latent action decoding. We obtain state-of-the-art results across multiple manipulation and navigation benchmarks, as well as real-robot deployments. UniVLA achieves superior performance over OpenVLA with less than 1/20 of pretraining compute and 1/10 of downstream data. Continuous performance improvements are observed as heterogeneous data, even including human videos, are incorporated into the training pipeline. The results underscore UniVLA's potential to facilitate scalable and efficient robot policy learning.

Bio:

Hongyang Li is an Assistant Professor at The University of Hong Kong and has led OpenDriveLab since 2021. His research focus is on autonomous driving and embodied AI. He led the end-to-end autonomous driving project, UniAD and won the IEEE CVPR 2023 Best Paper Award. He created the first large-scale real robot ecosystem, Agibot World, that systematically investigated the scaling law principles for robotic manipulation. He served as Area Chair for CVPR, NeurIPS, ICLR, ICCV, ICML, RSS.