Ted Xiao

I'm a research scientist at Google DeepMind, where I work on making robots smarter. My research focuses on robot learning, internet-scale foundation models, and reinforcement learning. I am particularly interested in methods that can scale and generalize in the real world.


Formal Bio     G. Scholar     Twitter      Blog     LinkedIn     GitHub    

xiaoted at gmail dot com

Some recent highlights from our research:

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents

Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, Zhuo Xu
Preprint
Website  •   PDF  •   Blogpost

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh*, Jeannette Bohg*, Stefan Schaal*
Preprint
Website  •   PDF

Physically Grounded Vision-Language Models for Robotic Manipulation

Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, Dorsa Sadigh
IEEE International Conference on Robotics and Automation (ICRA) 2024
Website  •   PDF  •   Video  •   Dataset  •   Model

How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies

Montserrat Gonzalez Arenas, Ted Xiao, Sumeet Singh, Vidhi Jain, Allen Z Ren, Quan Vuong, Jake Varley, Alexander Herzog, Isabel Leal, Sean Kirmani, Dorsa Sadigh, Vikas Sindhwani, Kanishka Rao, Jacky Liang, Andy Zeng
IEEE International Conference on Robotics and Automation (ICRA) 2024,
★ Oral Presentation ★ Robot Learning Workshop (WRL) at the Conference on Neural Information Processing Systems (NeurIPS) 2023
PDF

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration [>150 Authors]
IEEE International Conference on Robotics and Automation (ICRA) 2024
Website  •   PDF  •   Code  •   Dataset  •   Blogpost

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao
★ Spotlight ★ International Conference on Learning Representations (ICLR) 2024
Website  •   PDF  •   Blogpost

Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation

Annie Xie, Lisa Lee, Ted Xiao, Chelsea Finn
Robot Learning Workshop (WRL) at the Conference on Neural Information Processing Systems (NeurIPS) 2023
Website  •   PDF

Open-World Object Manipulation using Pre-Trained Vision-Language Models

Austin Stone*, Ted Xiao*, Yao Lu*, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, Karol Hausman
Conference on Robot Learning (CoRL) 2023
Webpage  •   PDF  •   Video

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
Conference on Robot Learning (CoRL) 2023
Webpage  •   PDF  •   Video  •   Blogpost

Language to Rewards for Robotic Skill Synthesis

Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien, Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia
★ Oral Presentation ★, Conference on Robot Learning (CoRL) 2023
Webpage  •   PDF  •   Code  •   Blogpost

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

Ted Xiao*, Harris Chan*, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, Jonathan Tompson
Robotics: Science and Systems (RSS) 2023
Webpage  •   PDF

Scaling Robot Learning with Semantically Imagined Experience

Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, Fei Xia
Robotics: Science and Systems (RSS) 2023
Webpage  •   PDF  •   Video

Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators

Alexander Herzog*, Kanishka Rao* Karol Hausman*, Yao Lu*, Paul Wohlhart*, Mengyuan Yan, Jessica Lin, Montserrat Gonzalez Arenas, Ted Xiao, Daniel Kappler, Daniel Ho, Jarek Rettinghouse, Yevgen Chebotar, Kuang-Huei Lee, Keerthana Gopalakrishnan, Ryan Julian, Adrian Li, Chuyuan Kelly Fu, Bob Wei, Sangeetha Ramesh, Khem Holden, Kim Kleiven, David Rendleman, Sean Kirmani, Jeff Bingham, Jon Weisz, Ying Xu, Wenlong Lu, Matthew Bennice, Cody Fong, David Do, Jessica Lam, Yunfei Bai, Benjie Holson, Michael Quinlan, Noah Brown, Mrinal Kalakrishnan, Julian Ibarz, Peter Pastor, Sergey Levine
Robotics: Science and Systems (RSS) 2023
Webpage  •   PDF  •   Video  •   Blogpost

RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
Robotics: Science and Systems (RSS) 2023
Webpage  •   PDF  •   Video  •   Code  •   Blogpost

Token Turing Machines

Michael S. Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, Anurag Arnab
Conference on Computer Vision and Pattern Recognition (CVPR) 2023
PDF  •   Code

Jump-Start Reinforcement Learning

Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, Karol Hausman
International Conference on Machine Learning (ICML) 2023
Webpage  •   PDF  •   Blogpost

Inner Monologue: Embodied Reasoning through Planning with Language Models

Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter
Conference on Robot Learning (CoRL) 2023
Webpage  •   PDF  •   Video

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, Andy Zeng
★ Oral Presentation, Special Innovation Award ★, Conference on Robot Learning (CoRL) 2022
Webpage  •   PDF  •   Video  •   Code  •   Demo  •   Blogpost

PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale

Kuang-Huei Lee, Ted Xiao, Adrian Li, Paul Wohlhart, Ian Fischer, Yao Lu
Conference on Robot Learning (CoRL) 2023
Website  •   PDF  •   Video

Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning

Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, Brian Ichter
International Conference on Learning Representations (ICLR) 2022
PDF  •   Blogpost

AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale

Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, Sergey Levine
Conference on Robot Learning (CoRL) 2021
Webpage  •   PDF

Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, Sergey Levine
International Conference on Machine Learning (ICML) 2021
Webpage  •   PDF  •   Video  •   Blogpost

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman*, Alexander Herzog*
International Conference on Learning Representations (ICLR) 2020
Webpage  •   PDF  •   Video

Learning Latent Plans from Play

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet
★ Oral Presentation★, Conference on Robot Learning (CoRL) 2019
Webpage  •   PDF

Adversarial Machine Learning

Phillip Kuznetsov, Riley Edmunds, Ted Xiao, Humza Iqbal, Raul Puri, Noah Golmant, Shannon Shih
Contributed Chapter, Artificial Intelligence Safety and Security (CRC Press)
Book

Goal-Driven Dynamics Learning via Bayesian Optimization

Somil Bansal, Roberto Calandra, Ted Xiao, Sergey Levine, Claire J. Tomlin
56th IEEE Conference on Decision and Control (CDC) 2017
PDF

Frame Rate Upscaling with Deep Neural Networks

Ted Xiao, Raul Puri, Gautham Kesineni
Technical Report
PDF

NVIDIA, Invited Talk

2021

UC Berkeley, Machine Learning at Berkeley Tutorial

2018

Xoogler, Machine Learning Keynote

2017

RobotX, NLP Tutorial

UC Davis, iidata Conference Keynote

DiversaTech, Technical Interview Workshop

2023

Co-Organizer, CoRL Workshop on Language and Robot Learning

Demo, Robotics: Science and Systems (RSS) - Large Language Models on Robots

Lead Organizer, ICRA Workshop on Learning from Diverse, Offliine Data

Co-Organizer, ICLR Workshop on Reincarnating Reinforcement Learning

Co-Organizer, CoRL Workshop on Language and Robot Learning

2022

Co-Organizer, NeurIPS Deep Reinforcement Learning Workshop

2018+

Reviewer, CoRL, RSS, RA-L, ICRA, NeurIPS, ICML, ICLR