From pixels to physics: AI models learn to simulate real-world scenes as code
A team at MirroS has flipped the script on how machines understand physical scenes. Instead of treating videos as flat images or captions as descriptions, their new approach—dubbed Code-as-World—converts real footage into executable physics simulations. The idea is simple but powerful: pixels are just evidence of what’s happening, not the actual mechanics behind it. A video model can predict future frames without ever grasping mass, gravity, or friction—but Code-as-World forces the model to reconstruct the mechanism itself.
Rethinking scene representation
At the heart of this work is the executable world representation (EWR), a structured triple that encodes a scene as code: composition (objects, their geometry, mass, and friction), evolution (initial states, forces, collisions), and appearance (camera settings, lighting). Unlike traditional video models that output pixels or captions, this code can be run in the MuJoCo physics engine to generate a full state trajectory. The team argues that existing methods—whether 3D reconstruction or captioning—only capture fragments of a scene’s true nature. Code-as-World captures its physics.
An iterative hunt for the right simulation
Recovering this executable code from a video isn’t a one-shot task. MirroS frames it as an abductive search: an agent proposes a simulation, executes it, renders the result, and verifies it against the original footage. This loop repeats for up to five rounds, refining the model’s hypothesis each time. Tools like SAM 3 for instance segmentation, VGGT-Omega for depth estimation, and SAM 3D for 3D meshes feed the agent’s pipeline. Discrepancies between the rendered simulation and real video guide revisions, ensuring the final code aligns with reality. When tested, this agentic approach outperforms independent sampling on metrics like object overlap and trajectory accuracy.
A new kind of training data
The real breakthrough may lie in what these verified simulations enable. Unlike raw video, executable code comes with exact physical labels—mass, velocity, acceleration—data that’s otherwise invisible in pixel form. MirroS leverages this to train its models, first on 73,335 image-space QA pairs covering object properties, then on the code itself. The result? Their largest model, Code-as-World-VL-9B, scores 55.4 MRA on the QuantiPhy benchmark, edging out Gemini-3.1 Flash (54.8) and surpassing the strongest open-weight baseline by roughly 15 points. Both the 4B and 9B variants are available under Apache 2.0 on Hugging Face, designed for research use with vLLM and OpenAI-compatible endpoints.
Why it matters
This isn’t just another video understanding model—it’s a shift toward mechanistic scene comprehension. By turning videos into verifiable physics simulations, MirroS enables AI systems to reason about the world in terms of causes, not just correlations. For industries like robotics, autonomous systems, or even graphics, this could mean more reliable training data, better alignment between perception and reality, and simulations that aren’t just plausible but provably correct. The open-source release lowers the barrier to experimentation, but the real test will be whether these executable worlds can scale beyond controlled lab conditions. If they do, we may finally have a way to bridge the gap between what AI sees and what it understands.
Source: MarkTechPost. AI-assisted editorial synthesis — TechnoExpress.