Towards Robust and Reliable Autonomous Vehicles with Foundation Models

Part of the Special ECE Seminar Series 

Modern Artificial Intelligence

Title:

Towards Robust and Reliable Autonomous Vehicles with Foundation Models

Speaker:

Jose Alvarez, Director at NVIDIA

Abstract:

In this talk, I will present our latest progress in vision-centric and data-driven methods for autonomous vehicles. First, I will introduce Hydra, our end-to-end architecture using a multi-target distillation paradigm, and then I will explain how we leverage MLLMs within the autonomous driving ecosystem. In the car, I will discuss Omnidrive, an LLM-Agent for driving with 3D perception, reasoning, and planning. I will introduce SSE in the cloud, a multimodal semantic data selection framework to scale up datasets for effectively training AV models. Finally, I will share some insights on the remaining challenges to achieving robust autonomous driving systems.

Bio:

Jose is a director at NVIDIA, leading the perception research for the Autonomous Driving team. The focus of the team and his interests are the scalability of deep learning and its applicability to autonomous driving. Prior to NVIDIA, he was a research scientist at TRI and NICTA, and worked as a postdoc researcher at NYU under Yann LeCun. He did his PhD in computer vision prior to the Deep Learning revolution, working on road scene understanding for autonomous driving when datasets were scarce and small.

Overall, his work has been centered around dynamic scene understanding and focused on the applicability of AI in real world problems such as bionic vision, medicine, agriculture, and autonomous vehicles.