Understanding World or Predicting Future? A Comprehensive Survey of World Models

In plain words

A world model is a learned stand-in for an environment: given the current situation and an action, it predicts what happens next. The survey argues that the term covers two jobs. One is building an internal picture that captures how the world works. The other is predicting how the world will change, so an agent can try options in its head before acting.

The paper sorts the research along those two jobs. It covers agents that learn a model to plan, language models that already carry knowledge about space, time and people, and video generators that act as simulators. It then shows where these ideas are used: games, robots, self-driving cars, cities and simulated societies. It ends with open problems, such as video generators that look realistic but break physical rules, and the lack of shared benchmarks. Read it as a map of the field rather than as a method to implement.

TL;DR

• A survey, not a method. It argues that world models serve two functions: implicit representation of the world for understanding, and prediction of future states for simulation and decision-making §1 §2.
• Covers model-based reinforcement learning, world knowledge in language models, video generation as world simulation, embodied environments, and applications in games, autonomous driving, robotics, urban analytics and social simulacra §3 §4 §5.
• The only formulas that an implementer can use are the one-step model-learning objectives and the model predictive control planning objective eq. 1 eq. 2 eq. 3.
• Useful for orientation and as a list of evaluation benchmarks tab. 8. It reports no experiments of its own, so none of its numbers can be reproduced from it.

Problem

The paper says the definition of a world model is still debated, and that the debate splits into two views: understanding the present versus predicting the future §1. Earlier surveys each cover a single application area, such as video generation, autonomous driving or agent applications, or they cover the move from multimodal models to world models. The paper says these surveys do not systematically define what a world model is or what each application needs from one §1 app. A.

Method

The paper's contribution is a taxonomy and a literature review, not an algorithm §1. Its organising claim is that all these lines share one goal: a world model should capture how the world changes step by step, which lets it project further ahead and inform decisions §2, Categorization. The paper frames this as two lenses:

1. Implicit representation of the external world, covering decision-making and world knowledge in models §3 fig. 3 fig. 4.
2. Future prediction of the external world, covering video generation and embodied environments §4 fig. 5.

Applications (§5) and open problems (§6) follow. The subsections below summarise each part.

Background and history (§2). The paper traces the idea from frame representations in the 1960s and transition models in model-based reinforcement learning to the recurrent world model of Ha and Schmidhuber (2018), the Dreamer series, LeCun's joint embedding predictive architecture (2022), then language models, video generators and interactive 3D scene generators §2. A roadmap figure shows this development fig. 1. Another figure shows the survey's framework and timeline fig. 2.

Model-based reinforcement learning (§3.1). The paper treats the world model as the transition dynamics M in a Markov decision process, since the reward function is usually known §3.1. Learning M is cast as supervised learning on real transitions, with either a squared-error loss eq. 1 or a probabilistic loss eq. 2. For images, the paper describes an autoencoder with latent dynamics (Ha and Schmidhuber) and a visual encoder with latent dynamics (the Dreamer line) §3.1.

Policy generation uses the learned model in three ways named by the paper:

• Model predictive control: choose the action sequence with the highest predicted return over a horizon τ eq. 3. The paper names Monte Carlo sampling of action sequences and probabilistic ensembles with trajectory sampling as ways to search §3.1.
• Monte Carlo tree search over learned values, including AlphaGo-style search and value prediction networks §3.1.
• Latent-space trajectory optimisation, for example TD-MPC2 §3.1.

The loop below combines the model-learning step eq. 1 with the planning step eq. 3. It is our condensed reading of §3.1. The paper does not give it as an algorithm.

# Model-based RL: learn the transition model, then plan with it  [§3.1]
# Input:  real transitions (state, action, next_state), reward function
# Output: learned transition model and a planned action sequence

def learn_transition_model(transitions, model):
    # Supervised fit of M_theta to real one-step transitions  [eq. 1]
    for state, action, next_state in transitions:
        prediction = model(state, action)
        loss = squared_error(next_state, prediction)
        update(model, loss)
    return model

def plan_actions(model, reward, state, horizon):
    # Candidate action sequences, e.g. by Monte Carlo sampling  [§3.1]
    candidates = sample_action_sequences(horizon)
    # Score each candidate by its predicted sum of rewards  [eq. 3]
    returns = {}
    for sequence in candidates:
        returns[sequence] = rollout_return(model, reward, state, sequence)
    # Return the best candidate; execution and re-planning not stated
    return max(candidates, key=returns.get)

Language-based world models (§3.1). Two patterns are described. In direct action generation, a language or video model outputs actions from a learned or prompted model of the environment §3.1. In modular use, the language model builds an explicit model that is paired with a planner or verifier. Examples given are PDDL domains written by GPT-4 and then solved by an off-the-shelf planner, a program-writing agent that refines its own world model, a web-navigation agent that predicts the outcome of actions, and a planning framework for web agents that simulates actions with an LLM §3.1.

World knowledge in models (§3.2). tab. 1 lists 33 works in four categories: common sense and general knowledge, knowledge of the global physical world (space and time, geography, cities), knowledge of the local physical world (cognitive maps, visual prediction, spatial reasoning benchmarks), and knowledge of human society (Theory of Mind and social norms) §3.2 fig. 4. The text reports that language models show linear representations of space and time §3.2, but that their urban knowledge is often coarse and inaccurate §3.2. It also reports that models perform well on several Theory of Mind tasks, with limits such as detecting faux pas §3.2.

Video generation as world models (§4.1). The paper treats video generators such as Sora as the main current route to world modelling, while noting that whether they are true world models is debated §4.1 §1. It groups the desirable capabilities into long-term predictive ability, multimodal integration, interactivity and coverage of diverse environments §4.1. The main limitations it names are weak causal reasoning and failures on physical laws §4.1. tab. 2 groups recent video models into five themes: extended-length generation, combining several input types, user-controllable generation, keeping frames consistent over time, and modelling varied scenes.

Embodied environments (§4.2). tab. 3 lists 26 simulators and generated environments, grouped as indoor, outdoor, and dynamic (action-conditioned video) environments §4.2 fig. 5. The text says most existing work builds static indoor and outdoor scenes and that the trend is towards dynamic, first-person, action-conditioned environments that respect physical constraints such as depth or normals §4.2.

Applications (§5).

• Game intelligence: the paper organises work by interactivity, consistency and generalisation across games §5.1.
• Embodied intelligence: learning implicit representations, predicting future states, and transfer from simulation to the real world §5.2 tab. 4.
• Urban intelligence: autonomous driving, autonomous logistics and urban analytics §5.3 fig. 6 tab. 5 tab. 6.
• Societal intelligence: social simulacra that mirror real societies, and agents' understanding of the external world inside such simulacra §5.4 tab. 7.
• Functions (§5.5): cloud-side world models can serve as data engines, reinforcement learning environments and policy evaluators, and edge-side world models can compress states into a latent space for planning §5.5.

Open problems (§6). The paper lists physical rules and counterfactual simulation, the social dimension of simulated worlds, benchmarks tab. 8, bridging simulation and reality, simulation efficiency, and ethical and safety concerns, namely data privacy, misuse for simulating harmful scenarios, and accountability for generated content §6.

Hyperparameters

Not applicable: the paper runs no experiments and reports no training configuration §1–§7. The symbols that appear in the formulas have no values stated.

Results

No experiments: the paper reports no results of its own §1–§7. The benchmarks in tab. 8 are listed for evaluation, and the paper does not report scores on them. Its quantitative statements are the authors' descriptions of cited works, not verified results. Examples are a DreamerV3 claim of solving over 150 tasks §3.1, a game engine claim of running at 20 frames per second §5.1, and a 17-billion-parameter game model §5.1.

Benchmarks catalogued in tab. 8 (§6), in our own wording:

Paper vs code

No official code. The abstract links https://github.com/tsinghua-fib-lab/World-Model, which the abstract describes as a summary of representative papers and their code repositories (abstract). That is a paper list for the survey, not an implementation of any method here, so no repository was cloned. The only other URL in the source is a footnote to an encyclopedia entry §2; no project page is linked.

Pitfalls

• No formal definition. The paper calls the definition of a world model contested §1. Its "consensus" that the purpose is to compute the next state with certainty or some guarantee is an assertion from the authors §2, Categorization. Implementers should not treat the taxonomy as settled.
• "World model" covers different things. The same term is used for latent dynamics models in reinforcement learning §3.1, for language-model knowledge §3.2 and for video generators §4.1. The paper itself asks whether video generators qualify §1 §4.1. When comparing papers, check which sense is meant.
• Equations are not complete recipes. eq. 1 is a squared one-step prediction error. eq. 2 is written as a log-ratio expectation under the true dynamics, which the text calls a KL objective §3.1. The paper does not say which model family to use for it. eq. 3 gives the planning objective but not how actions are executed or re-planned §3.1.
• Table versus text. tab. 8 describes UrbanVideo-Bench as 5.2k "urban egocentric clips", while the text in §6 says "5.2k drone clips" §6. Check the original benchmark before citing either.
• Selection is not described. The paper gives no inclusion criteria or search protocol for the works in its tables §1 tab. 1 tab. 3. Absence from a table does not mean a work is absent from the field.
• Author-stated claims. Several statements about model capability, such as Sora adhering to physical laws in the introduction §1 and a claim of "perfectly" adhering to physics, are descriptions by the authors, not measured results. Later sections give the opposite picture for physical laws §4.1 §6.
• Version. The source compiled here is the latest arXiv version (v4, updated 2025-12-10 in the metadata). The source's update history says the background section was rewritten and the application section reorganised in November 2025 app. C.
• Figures are described, not reproduced. Figure numbers are counted in source order. If a numbering check fails, cite the section instead.

Lineage

• Builds on: the recurrent world model of Ha and Schmidhuber (2018) and the Dreamer series §2 §3.1; LeCun's joint embedding predictive architecture (2022) and V-JEPA and V-JEPA 2 §2 §5.5.
• Compared against: six earlier surveys, on video generation, efficient multimodal models, and autonomous driving app. A.
• Superseded by: not stated.