From the source
NVIDIA introduced Cosmos3-H-Dreams, a real-time generative surgical simulator trained on clinical cholecystectomy recordings and controlled live by robot kinematics.
- The company says it is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.
- It builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, and specializes a bidirectional teacher on curated cholecystectomy video and kinematics.
- The model uses 28 action dimensions covering pose and gripper information for both arms plus energy controls and thumbstick axes and buttons, with training on 832 x 480 video and actions aligned at 10 Hz.
- Two released checkpoints use four and two denoising steps per generated block; the two-step model is separately trained, and fewer steps mean faster inference at lower visual quality.
- Powered by Cosmos 3 and accelerated with FlashDreams, it runs interactively on a single NVIDIA RTX PRO 6000 GPU, with an adapted FadeMem module consolidating the key/value cache during long rollouts.
