RoboPapers

57:39

Ep#28: DreamGen: Unlocking Generalization in Robot...

RoboPapers ·

2025/09/09

Robotics has a data problem. Generative video models propose a solution: we can fine-tune these models with robot data to generate vast amounts of diverse task data, which in turn unlocks new behaviors and environmental generation.Abstract:We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages

en

0:52

Ep#27: DextrAH-RGB: Visuomotor Policies to Grasp...

RoboPapers ·

2025/09/09

How can we learn sim-to-real manipulation policies for grasping any object? Watch this episode with Ritvik Singh of NVIDIA to find out.Abstract:One of the most important yet challenging skills for robots is dexterous multi-fingered grasping of a diverse range of objects. Much of the prior work is limited by the speed, dexterity, or reliance on depth maps. In this paper, we introduce DextrAH-RGB, a system that can perform dexterous arm-hand grasping end2end from stereo RGB input. We train a teacher

en

69:03

Ep#26: ManiSkill3: GPU Parallelized Robotics...

RoboPapers ·

2025/09/06

Robotics as a field faces a large data gap, and one of the most promising directions for solving it is sim-to-real robot learning. We talked with Stone Tao about ManiSkill3: a powerful, easy-to-use framework for generalizable sim-to-real learning.In particular, we learned about how with Maniskill + HuggingFace LeRobot, you can easily train your own sim-to-real manipulation policies on a low-cost robot!Abstract:Simulation has enabled unprecedented compute-scalable approaches to robot learning

en

58:43

Ep#25: DexUMI: Using Human Hand as the Universal...

RoboPapers ·

2025/09/05

Collecting dexterous humanoid robot data is difficult to scale. That's why Mengda Xu and Han Zhang built DexUMI: a tool for demonstrating how to control a dexterous robot hand, which allows you to quickly collect task data.Abstract:We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between

en

67:01

Ep#24: CLONE: Closed-Loop Whole-Body Humanoid...

RoboPapers ·

2025/09/04

How can we control humanoid robots to get them to perform a variety of challenging, real-world tasks while moving around in their environments? Yixuan Li and Siyuan Huang of BIGAI tell us how their complex system works.Abstract:Humanoid teleoperation plays a vital role in demonstrating and collecting data for complex humanoid-scene interactions. However, current teleoperation systems face critical limitations: they decouple upper- and lower-body control to maintain stability, restricting natural

en

41:54

Ep#23: FALCON 🦅- Learning Force-Adaptive Humanoid...

RoboPapers ·

2025/09/03

One of the key advantages of humanoid robots is how they can handle heavier objects and exert forces in a more dynamic way, compared to wheeled robots. However, few research papers show this. We talked to Yuanhang Zhang about his really impressive work, FALCON, which shows humanoid robots pushing or pulling carts, lifting weights, and more.Abstract:Humanoid loco-manipulation holds transformative potential for daily service and industrial tasks, yet achieving precise, robust whole-body control with

en

63:01

Ep#22: DexWild: Dexterous Human Interactions for...

RoboPapers ·

2025/08/27

How can we collect large-scale manipulation data in the real world? DexWild proposes a solution: an easy-to-use wearable that lets operators perform robotic tasks in a wide variety of environments easily. With the added data diversity, this results in robot policies which can operate in a variety of different environments.Watch the video to learn more!Abstract:Large-scale, diverse robot datasets have emerged as a promising path toward enabling dexterous manipulation policies to generalize to novel

en

59:44

Ep#21: TesserAct: Learning 4D Embodied World Models

RoboPapers ·

2025/08/24

World models are an exciting area of research, wherein we predict video given robot actions and use these predictions for planning or training. But two dimensional image frames aren’t necessarily enough for robot planning; robotic tasks are inherently three dimensional, and so shouldn’t we be predicting how objects move around in 3d space?TesserAct does this, tuning a video generative model to produce 3d predictions. Watch the video to learn more!Abstract:This paper presents an effective approach

en

73:23

Ep#20: VideoMimic: Visual imitation enables...

RoboPapers ·

2025/08/22

Part of the advantage of humanoid robots is in their ability to interact with the environment: to climb stairs, to sit down, to navigate rough terrain, and so on. In VideoMimic, the authors produced a pipeline which let them:* Scan a scene into simulation using an iPhone* Train a whole-body control policy for manipulation and locomotion* Use this policy to control a humanoid robot as operated in diverse, real-world environments.Abstract:How can we teach humanoids to climb staircases and sit on

en

50:34

Ep#19: Learning to Drive From a World Model

RoboPapers ·

2025/08/21

Comma has been selling end-to-end driving kits for your car for years. Their Openpilot is available now, with source code on Github. Very interestingly, they train their lanekeeping behavior using generative world models. Watch to learn more.Abstract:Most self-driving systems rely on hand-coded perception outputs and engineered driving rules. Learning directly from human driving data with an end-to-end method can allow for a training architecture that is simpler and scales well with compute and

en

57:45

Ep#18 HuB: Learning Extreme Humanoid Balance

RoboPapers ·

2025/08/21

This time, Tong and Boyuan tell us about how to achieve some of the most impressive humanoid robot balancing that we have ever seen.Take a look!Here’s the abstract:The human body demonstrates exceptional motor capabilities—such as standing steadily on one foot or performing a high kick with the leg raised over 1.5 meters—both requiring precise balance control. While recent research on humanoid control has leveraged reinforcement learning to track human motions for skill acquisition, applying this

en

71:02

Ep#17: EgoZero: Robot Learning from Smart Glasses

RoboPapers ·

2025/08/17

Using human data is the key to unlocking the potential of general-purpose robotics: it’s much more plentiful and can potentially be collected quickly at scale for relatively little money. But how can we actually do this?Despite recent progress in general purpose robotics, robot policies still lag far behind basic human capabilities in the real world. Humans constantly interact with the physical world, yet this rich data resource remains largely untapped in robot learning. We propose EgoZero, a

en

43:24

Ep#16: TWIST: Teleoperated Whole-Body Imitation System

RoboPapers ·

2025/08/17

Learn how to do versatile, capable whole body teleoperation of humanoid robots — a key capability for unlocking the data we need to train general-purpose autonomous humanoids.Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior,

en

54:41

Ep#15: Navigation World Models

RoboPapers ·

2025/08/16

World models are one of the hottest research areas. Amir’s work on navigation world models shows how we can use the world models to achieve new goals tin unfamiliar environments.Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional

en

68:34

Ep#13: Instant Policy

RoboPapers ·

2025/08/15

How can we do in-context learning for robots? Watch this episode to find out. This work was an ICLR 2025 oral paper, and winner of Best Paper Award at the ICLR 2025 Robot Learning Workshop.Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly from just one or two demonstrations, achieving ICIL through two key components. First, we

en

68:43

Ep#12: VaViM and VaVAM: Autonomous Driving through...

RoboPapers ·

2025/08/14

How can world models be used for training autonomous driving? Learn by watching this episode with Florent Bartoccioni!We explores the potential of large-scale generative video models to enhance autonomous driving capabilities, introducing an open-source autoregressive video model (VaViM) and a companion video-action model (VaVAM). VaViM is a simple autoregressive model that predicts frames using spatio-temporal token sequences, while VaVAM leverages the learned representations to generate driving

en

68:23

Ep#14: In-Air Vehicle Maneuver for High-Speed...

RoboPapers ·

2025/08/14

Two papers in one episode! Learn about how we can use small amounts of data to train transformers capable of doing truly impressive stuff.Original Post on XDom, cars don’t fly!—Or do they? In-Air Vehicle Maneuver for High-Speed Off-Road NavigationWhen pushing the speed limit for aggressive off-road navigation on uneven terrain, it is inevitable that vehicles may become airborne from time to time. During time-sensitive tasks, being able to fly over challenging terrain can also save time, instead of

en

61:55

Ep#11: Sim-and-Real Co-Training: A Simple Recipe for...

RoboPapers ·

2025/08/14

Large real-world robot datasets hold great potential to train generalist robot models, but scaling real-world human data collection is time-consuming and resource-intensive. Simulation has great potential in supplementing large-scale data, especially with recent advances in generative AI and automated data generation tools that enable scalable creation of robot behavior datasets. However, training a policy solely in simulation and transferring it to the real world often demands substantial human

en

61:03

Ep#10 Human Policy ~ Humanoid Policy

RoboPapers ·

2025/08/12

It’s hard to collect data for humanoid robots at sufficient scale for generalization. The authors of “Humanoid Policy ~ Human Policy” have the answer: collect human data at scale, and retarget it to humanoid robots.This acts as a multiplier, letting you get away with using far less robot data to accomplish challenging robot tasks. Watch or listen to learn more.Abstract:Training manipulation policies for humanoid robots with diverse data enhances their robustness and generalization across tasks and

en

51:46

Ep#9: AutoEval: Autonomous Evaluation of Generalist...

RoboPapers ·

2025/08/12

Evaluating robot policies is challenging, because everyone has a slightly different testing setup and environment. Paul joined us to talk about his work AutoEval, which is a continuously-operating benchmark which allows you to test robot policies remotely, in any environment.Abstract:Scalable and reproducible policy evaluation has been a long-standing challenge in robot learning. Evaluations are critical to assess progress and build better policies, but evaluation in the real world, especially at a

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Engineering the Intelligent Enterprise

PRASAD BHONDE

Double Tap

Steven Scott, Shaun Preece, Steven...

The Good Tech Companies

HackerNoon

BlockHash

BlockHash LLC