NVIDIA Adds Four Open Models to Its Cosmos 3 Family to Simulate Worlds Interactively
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
NVIDIA is adding to Cosmos 3 four open-weight models that generate simulated worlds: two produce autonomous driving data with cameras and LiDAR, and two more serve robotics. Cosmos3-Nano-Sim-Bimanual, for instance, anticipates what a two-arm robot will see based on its movements. NVIDIA offers them on Hugging Face with the OpenMDW 1.1 license and with recipes to adapt them.
On November 5, NVIDIA presented on its Hugging Face blog four new open-weight models for Cosmos 3, its world model platform. These models produce video of an environment based on a set of commands and on what has happened before. Two of the new ones are meant for autonomous driving and the other two for robotics.
On the driving side, Cosmos3-Nano-Transfer-Auto creates synthetic data with up to 11 cameras, fisheye lenses included, and can produce images and LiDAR together; the developer picks the weather, the light or the time of day for each scene. Cosmos3-Nano-Sim-Auto is a distilled variant of the former that turns out, in a few steps and in short stretches, synchronized camera video and LiDAR, for long simulations in which driving keeps changing the commands.
On the robotics side, Cosmos3-Nano-Sim-Transfer converts the controls of a physics simulator plus a text description into realistic video, while the simulator stays in charge of the physics. Cosmos3-Nano-Sim-Bimanual starts from an image and predicts how the scene changes depending on what a two-arm robot or the camera does, with the YAM and AgiBot platforms; it can be driven by a control policy or by a person using joysticks.
All four are versions of Cosmos 3 Nano tuned after training, and they come out under the OpenMDW 1.1 license, which according to NVIDIA permits both commercial and non-commercial use on the terms it sets. The company pairs the models with the training recipes needed to adapt them to other data, sensors or robots. Running them requires Linux, an NVIDIA GPU and Docker; the published setup was tried on a single GB200, which NVIDIA stresses is not a minimum memory requirement. NVIDIA itself admits that it still has to measure in which cases these models truly help to train and evaluate control policies, and where they fall short.
Why it matters · analysis and opinion
Teaching an autonomous car to drive or a robot to handle objects takes far more situations than the ones on record, hence the interest in simulators that generate believable scenes. With these models, NVIDIA moves toward a simulation that responds to each action instead of just playing back a video. Open weights and open recipes let researchers and developers try them out without starting from scratch, although they will need NVIDIA GPUs to do so. The most valuable part of the announcement is what it admits: it is not clear how much of what is learned in these synthetic worlds carries over to the real world. That measurement will decide whether they move from test bench to working tool.
Official source: NVIDIA · Written with the help of AI: how we make the news
