Teaser

Token-World teaser figure
Teaser figure placeholderassets/teaser.png

Abstract

A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World (r = 0.794 vs. 0.583), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction.

Method Overview

Token-World method overview
Method figure placeholderassets/method_overview.png
Token-World overview. (a) Compact Token Construction. Qwen3-VL visual tokens are compressed into a compact latent space. (b) Dynamics Modeling. A spatiotemporal Transformer models action-conditioned dynamics in compact token space. (c) Training Objective. The model is trained with diffusion forcing and weighted flow matching. (d) Autoregressive Rollout. Future compact tokens are generated with a sliding temporal window and recurrent state.

Qualitative Results

Open-loop Replay

Closed-loop Rollouts and Simulation Replay

Left: the VLA policy selects actions in closed loop using states predicted by Token-World. Right: the simulator replays the same action sequence.

Place burger and fries
Place a can in the basket
Place cans in the plastic box
Place the container on the plate
Place the empty cup
Put the object in the cabinet
Stack two blocks
Stack two bowls

Quantitative Results

Open-loop comparison of VLM feature fidelity and policy-action consistency on RoboTwin and real-world Franka tasks, measured by cosine similarity and NMSE
Open-loop prediction figure placeholderassets/open_loop_prediction.png
Open-loop prediction quality. Token-World achieves the strongest VLM feature fidelity and policy-action consistency across both RoboTwin and real-world Franka tasks, with higher cosine similarity and lower NMSE than RGB-based world models.
Policy evaluation scatter plots comparing Token-World and Ctrl-World against reference success rates, alongside average simulation step times
Policy evaluation and efficiency figure placeholderassets/policy_evaluation_efficiency.png
Policy evaluation and efficiency. Token-World’s predicted outcomes align more closely with actual policy success rates (r = 0.794 vs. 0.583 for Ctrl-World), while also achieving the fastest simulation speed at 0.359 s per step.

Ablations

Ablation studies comparing VLM and RGB VAE representations, raw and compressed VLM features, and compact latent dimensions of 8, 16, 32, and 48
Ablation figure placeholderassets/ablation.png
Ablation studies. (a) VLM-based compact states improve both feature fidelity and action consistency over RGB VAE representations. (b) Compressing raw VLM features substantially improves dynamics prediction. (c) A 16-D compact representation provides the best overall trade-off between feature and action consistency.