Xavier Puig

Research Scientist at FAIR

xavierpuigf@gmail.com
181 Freemont St, San Francisco

Resume • Google Scholar • LinkedIn • GitHub • Ph.D. Thesis

About Me

I am a Research Scientist at FAIR , working on Embodied AI. Previously, I completed my Ph.D. at the Computer Science and Artificial Intelligence Laboratory (CSAIL) of MIT, advised by Professor Antonio Torralba. Before that, I obtained a double degree in Computer Science and Telecommunications at the CFIS program of UPC.

I am interested in building agents that can assist and collaborate with humans. My research focuses in developing agents that can understand and anticipate human goals, and coordinate with them in performing complex tasks. I also study how to represent humans in simulation environments, including generating realistic motions and plausible high-level behaviors.

If you are interested in these areas, and would like to collaborate, or intern in my team, reach out!

News

Oct. 2024: We released PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks.
Sept. 2024: I gave a talk about Human Foundation Models for Embodied AI at the Human Foundation Models for 3D Humans workshop at ECCV'2024.
July 2024: CHOIS and SIF were accepted to ECCV'2024.
Oct. 2023: Our paper Generating Continual Motion in Diverse 3D Scenes, was accepted to 3DV'2024.
Oct. 2023: We released Habitat 3.0.
Jan. 2023: Our work NOPA was accepted to ICRA'2023.
Oct. 2022: I joined the Embodied AI team at FAIR , to work in human-centered Embodied Intelligence.
Sep. 2022: I defended my Ph.D. Thesis.
Jan. 2022: GenRep was accepted to ICLR 2022.
Apr. 2021: We are organizing the Social AI Virtual Gathering , at ICLR 2021.
Feb. 2021: We are organizing the Social Intelligence in Humans and Robots Workshop, to be held at ICRA'2021.
Jan. 2021: Watch-And-Help was accepted to ICLR'2021 as a Spotlight presentation.
Nov. 2020: Watch-And-Help was accepted to the Cooperative AI Workshop at NeurIPS 2020 as a Best paper award.
Oct. 2020: We released the code for our Watch-And-Help paper.
Sep. 2020: We released the new version of VirtualHome, as well as the Unity Source Code.

Publications

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander Tsung-Yen Yang
Preprint
Paper Webpage
Controllable human-object interaction synthesis
Jiaman Li, Alex Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig†, Karen Liu†.
In Proc. European Conference of Computer Vision (ECCV), 2024 -
Oral

Paper Webpage
Situated Instruction Following
So Yeon Min, Xavier Puig, Devendra Singh Chaplot , Tsung-Yen Yang Akshara Rai Priyam Parashar Ruslan Salakhutdinov Yonatan Bisk Roozbeh Mottaghi
In Proc. European Conference of Computer Vision (ECCV), 2024
Paper Webpage
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavier Puig∗, Eric Undersander∗, Andrew Szot∗, Mikael Dallaire Cote∗, Tsung-Yen Yang∗, Ruslan Partsey∗, Ruta Desai∗, Alexander William Clegg∗, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai†, Roozbeh Mottaghi†.
International Conference on Learning Representations (ICLR), 2024
Paper Webpage
Generating Continual Motion in Diverse 3D Scenes
Aymen Mir , Xavier Puig , Angjoo Kanazawa , Gerard Pons-Moll .
In Proc. International Conference on 3D Vision (3DV), 2024
Paper Code Webpage Video
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Xavier Puig* , Tianmin Shu*, Josh Tenenbaum , Antonio Torralba .
In Proc. IEEE International Conference on Robotics and Automation (ICRA), 2023
Paper Code Webpage Video
Pre-Trained Language Models for Interactive Decision-Making
Shuang Li , Xavier Puig , Chris Paxton , Yilun Du , Clinton Wang , Linxi Fan ,
Tao Chen , De-An Huang , Ekin Akyurek , Anima Anandkumar , Jacob Andreas ,
Igor Mordatch , Antonio Torralba , Yuke Zhu
In Proc. Neural Information Processing Systems (NeurIPS), 2022 -
Oral

Paper Code Webpage
Generative Models as a Data Source for Multiview Representation Learning
Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola.
International Conference on Learning Representations (ICLR), 2022
Paper Code Webpage
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Xavier Puig, Tianmin Shu, Shuang Li, Ziling Wang, Josh Tenenbaum, Sanja Fidler, Antonio Torralba.
Cooperative AI Workshop at NeurIPS, 2020 -
Best paper award

International Conference on Learning Representations (ICLR), 2021 -
Spotlight

Paper Code Webpage
Synthesizing Environment-Aware Activities via Activity Sketches
Andrew Liao*, Xavier Puig*, Marko Boben, Antonio Torralba. Sanja Fidler,
In Proc. Computer Vision and Pattern Recognition (CVPR), 2019.
Paper Code Webpage
Semantic Understanding of Scenes through ADE20K Dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, Antonio Torralba.
International Journal on Computer Vision (IJCV), 2018.
Paper Dataset Benchmark Page Challenge Page Toolkit Demo
VirtualHome: Simulating Household Activities via Programs
Xavier Puig*, Kevin Ra*, Marko Boben*, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba,
In Proc. Computer Vision and Pattern Recognition (CVPR), 2018 -
Oral

Paper Webpage Press Talk
Open Vocabulary Scene Parsing
Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, Antonio Torralba.
In Proc. International Conference in Computer Vision (ICCV), 2017.
Paper Page
Scene Parsing through ADE20K Dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, Antonio Torralba.
In Proc. Computer Vision and Pattern Recognition (CVPR), 2017.
Paper Dataset Toolkit Demo

Experience

2022.10 - Present: Research Scientist at FAIR
Working on the Embodied AI Team
2016.9 - 2022.9: Research Assistant, MIT Computer Science and Artificial Intelligence Laboratory
Advised by Prof. Antonio Torralba
2019.6 - 2019.8: Research Intern, Google Cambridge working in Contrastive Learning
Advised by Dilip Krishnan, Aaron Maschinot, Aaron Sarna
2017.6 - 2017.8: Research Intern, Google Seattle, working in Mobile Computer Vision and Adaptive Inference.
Advised by Li Zhang, Yukun Zhu, Maxwell Collins

Professional Service