Fast-LeWM: Parallel Action-Prefix Prediction Slashes Latent World Model Planning Costs

Joint-Embedding Predictive Architectures (JEPAs), including the recent LeWorldModel (LeWM), have been hailed as a promising route to reconstruction-free visual world models. But when it comes to visual planning, they've hit a wall. The issue: LeWM evaluates candidate action sequences by repeatedly applying a local one-step latent transition model in an autoregressive fashion. That rollout is costly, and as the horizon stretches, errors pile up fast.

In a new paper titled Fast LeWorldModel, researchers describe a fix. Instead of rolling out predictions step by step, Fast-LeWM predicts the outcome of entire action prefixes in one go. The idea: encode the prefixes of a candidate action sequence and predict all future latents reached after executing those prefixes in parallel.

How Fast-LeWM Works

Given the current latent state and a candidate action sequence, Fast-LeWM encodes its prefixes, the first few actions, and predicts the future latent states that would be reached after executing those prefixes. By making action prefixes the basic prediction unit, the model directly learns the accumulated effects of actions over multiple horizons, rather than fitting only one-step state transitions.

During planning, the predictor can use the last prefix token from the encoded action sequence to evaluate the corresponding future latent without explicitly rolling through each intermediate imagined state. This design eliminates the autoregressive bottleneck and reduces the exposure to error accumulation that plagues long-horizon predictions.

Performance Gains

Across multiple tasks, Fast-LeWM improves average success rates over LeWM while substantially reducing planning time. The method achieves lower open-loop latent loss whose growth becomes significantly slower as the rollout horizon increases. This suggests that the prefix-level supervision forces the model to learn how states continuously evolve under different action prefixes, rather than only fitting one-step transitions.

The authors report that Fast-LeWM maintains the reconstruction-free advantages of JEPAs while overcoming their main limitation for planning: the computational cost of autoregressive rollouts. By parallelizing prediction, the method opens the door to real-time visual planning in robotics, autonomous navigation, and other domains requiring fast decision-making over long horizons.

Implications for Visual World Models

The work represents a practical advance in making latent world models more efficient for planning. While JEPAs have shown strong performance in representation learning and reconstruction-free modeling, their use in planning has been limited by the sequential nature of latent transition models. Fast-LeWM addresses this gap with a simple but effective architectural change.

Future work could explore extending the approach to continuous action spaces or integrating it with model-based reinforcement learning algorithms. The paper also suggests that the prefix-prediction paradigm could be applied to other autoregressive components in world models beyond visual planning.