NVIDIA releases Cosmos3-H-Dreams, an AI surgery simulator that generates video of the operation in real time
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
Cosmos3-H-Dreams recreates a cholecystectomy performed with the Versius robot: it takes a starting image and the instrument commands and returns what the camera would see. NVIDIA trained it on recordings of real procedures and has put it on Hugging Face, with code on GitHub. The model runs on one RTX PRO 6000 and is a research tool, with no validation for use on patients.
NVIDIA has released Cosmos3-H-Dreams on Hugging Face, an artificial intelligence model that recreates on video a gallbladder removal (cholecystectomy) performed with a surgical robot and reacts in real time to how the instruments move. The company announced it on Tuesday, October 20, on its Hugging Face blog.
The model starts from an initial image from the endoscopic camera, the one placed inside the abdomen, and from the movement commands the robot receives, and uses them to generate video of what that camera would see next. It was trained on video and robot motion data recorded during real operations with CMR Surgical's Versius system. NVIDIA says that, as far as it is aware, no earlier real-time generative surgical simulator had been trained on clinical recordings of this procedure and could be steered live.
The first version, Cosmos-H-Dreams, built on Cosmos-Predict2.5, was limited to suturing exercises on a tabletop with the dVRK system. Like that one, instead of calculating scene geometry and tissue deformation the way a numerical simulator would, it learns from video and robot movements how the image changes. Inside the body, a small movement can alter the whole scene: tissue stretches, reflections change, some structures hide others and smoke cuts visibility.
The new version builds on Cosmos3-H-Embodied-Nano, a model with 16 billion parameters from the Cosmos 3 family, NVIDIA's world models. It works with two visible instruments, a Cadiere grasper and a monopolar hook, and runs on a single NVIDIA RTX PRO 6000 graphics card. There are two versions: the two-step one is faster and the four-step one gives higher image quality. According to the model card, on the RTX PRO 6000 Blackwell Workstation Edition and at 832 × 480 pixels, the first generates about 43 frames per second and the second about 30.
The card says it is meant for surgical robotics research: testing in simulation the systems that control the robot before moving to a physical one, exploring the scene by operating it remotely and generating synthetic data. NVIDIA stresses that the model is meant for research and has not been validated for patient care, and warns that looking realistic only opens the door to validation and does not show that tissue behaves accurately.
The model can be downloaded from Hugging Face under the NVIDIA Open Model License, and the card states that it can be used for both commercial and non-commercial purposes. The code, examples and a step-by-step guide for adapting it to other data are on GitHub. Among its next lines of work, NVIDIA lists simulating the full operation, evaluating the model with surgeons and supporting more instruments, and it notes that none of these is yet an established clinical capability.
Why it matters · analysis and opinion
Training surgical robots takes a lot of practice that cannot be done on patients, and classic simulators need the geometry and mechanics of every tissue to be modeled explicitly. This approach changes the method: the simulator learns from real recordings and responds to the controls, which makes it easier to test control systems and produce training data. What is still missing for real use is spelled out by NVIDIA itself: a scene that looks believable does not guarantee that tissue reacts as it would in the operating room, and for now it covers a single procedure and a single robot. The decisive step will be evaluation with surgeons and pairing it with physical models that make errors measurable. For hospitals and manufacturers, it is a lab tool that deserves attention, not something to buy or deploy today.
Official source: NVIDIA · Written with the help of AI: how we make the news
