Jim Fan, Senior Research Manager and Head of Nvidia’s Generalist Embodied Agent Research (GEAR) Lab, recently discussed the potential training methods for future artificial intelligence (AI) robots. He envisions a time when these embodied AI agents will be developed within simulated environments, allowing them to acquire specialized skills. Fan noted that entire cities, homes, and factories could be recreated in simulations, providing a realistic backdrop for AI training.
Nvidia’s Jim Fan Explores Future AI Robot Training Methods
In a recent post on X (formerly Twitter), Fan highlighted the City of Tokyo’s release of a high-resolution 3D digital twin of the city, which is available for public download. He believes this initiative exemplifies a growing trend of transferring real-world environments into digital simulations for AI purposes.
“It’s an inevitable trend that more and more cities, houses, and factories will be transported into simulations,” Fan stated.
Fan pointed out that, unlike current training practices, where robots operate within controlled settings, future AI will learn in a more interconnected environment. The conventional approach has limitations, as robots trained in isolation may struggle to adapt to real-world scenarios.
To tackle these challenges, Fan proposed that embodied AI agents will soon operate as part of a cohesive system, akin to an “iron fleet.” These agents, which are AI systems with physical or simulated forms, will interact and perceive their surroundings similarly to humans and animals.
The Nvidia executive elaborated on future training environments, stating that AI robots will utilize real-time graphics engines and will be scaled across vast networks. This approach is expected to generate trillions of high-quality training tokens. He projected, “The majority of embodied agents will be born in sim, and transferred zero-shot to our real world when they are ready.”
Upon completion of their training, these AI robots will be deployed in various settings, including factories and offices, where they will be connected through a hive mind. This collective intelligence could enable them to learn from numerous experiences and collaborate on complex tasks.
While some might view this concept as speculative, Fan’s leadership in Nvidia’s embodied AI division lends credibility to this promising vision for the future of robotic training. He noted that the company’s headquarters in Santa Clara are designed and rendered in Omniverse, a GPU-accelerated graphics platform, prior to their physical construction.
