Xavier Puig

About

I am a co-founder at Versor, a robotics company using simulation to accelerate robot deployments. Before this, I was a Research Scientist at FAIR, working on Embodied AI. I completed my Ph.D. at the Computer Science and Artificial Intelligence Laboratory (CSAIL) of MIT, advised by Professor Antonio Torralba. Before that, I obtained a double degree in Computer Science and Telecommunications at the CFIS program of UPC.

I am interested in building agents that can assist and collaborate with humans. My research focuses on developing agents that can understand and anticipate human goals, and coordinate with them in performing complex tasks. I also study how to represent humans in simulation environments, including generating realistic motions and plausible high-level behaviors.

If you are interested in these areas, and would like to collaborate, or are interested in joining our team, reach out!

News

Jan. 2026 I started Versor, a robotics company using simulation to accelerate robot deployments.

Oct. 2025 I left Meta, after three years working on Embodied AI at FAIR.

Oct. 2024 We released PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks.

Sept. 2024 I gave a talk about Human Foundation Models for Embodied AI at the Human Foundation Models for 3D Humans workshop at ECCV 2024.

July 2024 CHOIS and SIF were accepted to ECCV 2024.

Oct. 2023 Our paper Generating Continual Motion in Diverse 3D Scenes was accepted to 3DV 2024.

Oct. 2023 We released Habitat 3.0.

Jan. 2023 Our work NOPA was accepted to ICRA 2023.

Oct. 2022 I joined the Embodied AI team at FAIR, to work in human-centered Embodied Intelligence.

Sep. 2022 I defended my Ph.D. Thesis.

Jan. 2022 GenRep was accepted to ICLR 2022.

Apr. 2021 We are organizing the Social AI Virtual Gathering, at ICLR 2021.

Feb. 2021 We are organizing the Social Intelligence in Humans and Robots Workshop, to be held at ICRA 2021.

Jan. 2021 Watch-And-Help was accepted to ICLR 2021 as a Spotlight presentation.

Nov. 2020 Watch-And-Help was accepted to the Cooperative AI Workshop at NeurIPS 2020 as a Best paper award.

Oct. 2020 We released the code for our Watch-And-Help paper.

Sep. 2020 We released the new version of VirtualHome, as well as the Unity Source Code.

Publications

Planning with an Embodied Learnable Memory

Priyam Parashar, Jacob Krantz, Matthew Chang, Kavit Shah, Xavier Puig, Roozbeh Mottaghi

International Conference on Learning Representations (ICLR), 2026

Paper

COOPERA: Continual Open-Ended Human-Robot Assistance

Chenyang Ma, Kai Lu, Ruta Desai, Xavier Puig, Andrew Markham, Niki Trigoni

Neural Information Processing Systems (NeurIPS), 2025 Spotlight

Paper Code Webpage

Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning

Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, Roozbeh Mottaghi

Preprint, 2025

Paper

ADAPT: Actively Discovering and Adapting to Preferences for any Task

Maithili Patel, Xavier Puig, Ruta Desai, Roozbeh Mottaghi, Sonia Chernova, Joanne Truong, Akshara Rai

Preprint, 2025

Paper Code

RobotMover: Learning to Move Large Objects from Human Demonstrations

Tianyu Li, Joanne Truong, Jimmy Yang, Alexander Clegg, Akshara Rai, Sehoon Ha, Xavier Puig

Preprint, 2025

Paper

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, Tsung-Yen Yang

International Conference on Learning Representations (ICLR), 2025

Paper Webpage

Controllable Human-Object Interaction Synthesis

Jiaman Li, Alex Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig†, Karen Liu†

European Conference on Computer Vision (ECCV), 2024 Oral

Paper Webpage

Situated Instruction Following

So Yeon Min, Xavier Puig, Devendra Singh Chaplot, Tsung-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, Roozbeh Mottaghi

European Conference on Computer Vision (ECCV), 2024

Paper Webpage

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

Xavier Puig∗, Eric Undersander∗, Andrew Szot∗, Mikael Dallaire Cote∗, Tsung-Yen Yang∗, Ruslan Partsey∗, Ruta Desai∗, Alexander William Clegg∗, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai†, Roozbeh Mottaghi†

International Conference on Learning Representations (ICLR), 2024

Paper Webpage

Generating Continual Motion in Diverse 3D Scenes

Aymen Mir, Xavier Puig, Angjoo Kanazawa, Gerard Pons-Moll

International Conference on 3D Vision (3DV), 2024

Paper Webpage

NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants

Xavier Puig*, Tianmin Shu*, Josh Tenenbaum, Antonio Torralba

IEEE International Conference on Robotics and Automation (ICRA), 2023

Paper Code Webpage Video

Pre-Trained Language Models for Interactive Decision-Making

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyurek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, Yuke Zhu

Neural Information Processing Systems (NeurIPS), 2022 Oral

Paper Code Webpage