related work of Facing Off World Model Backbones: RNNs, Transformers, and S4

Transformer world models. Transformer-based dynamics models, including TransDreamer, replace recurrence with self-attention, enabling parallel training and direct access to long histories. Their standard quadratic attention cost in sequence length makes long imagined rollouts and long-context visual modeling computationally expensive.

Structured state-space models. S4 is a structured state-space sequence model designed for long-range dependencies, combining a recurrent state-space interpretation with a convolutional form that permits parallel sequence training. The paper argues that this makes S4 a promising middle ground: linear-time recurrent inference and parallelizable training.

S4WM’s distinction. Rather than using S4 for conventional sequence benchmarks, S4WM incorporates it into a latent visual world model, with image encoding/decoding and stochastic latent dynamics, then evaluates it against RNN and Transformer backbones under a common world-modeling framework. The stated contribution is the first S4-based world model for generating high-dimensional image sequences via latent imagination.