Dreamer

2020 Dreamer V1

The very first working Dyna-style model-based RL implementation on pixel with data augmentation using imaginary rollout.

They reconstruct original observation using a decoder to catch all details unlike

TD-MPC so it is slower. Any on-policy algorithm can be used to train actor/critic.

First case that model-based RL could be successful.

Dreamer V2

DayDreamer

World Models for Physical Robot Learning

Dreamer V3
Pixel based RL

처음으로 인간 데이터 없이

Minecraft AI 다이아몬드 캠

학습하는 월드 모델의 구조와 목적 함수에 있습니다. DreamerV3는 변형된 목적 함수를 사용하여 효과적으로 동작할 수 있는 정책을 학습

월드 모델은 복잡한 환경에서 감각 입력의 압축 표현을 학습하고, 가능한 행동에 대한 미래의 표현과 보상을 예측

policy가 계획을 가능하게 한다

Actor Critic

Symlog Function

reward 가 크기 다양할 수 있는 걸 효율적으로 변환 발산방지 안정적 학습

V4 ‣

Proposes a scalable agent that learns general knowledge from large-scale video data and can optimize actions within imagination.

Introduces a Shortcut Forcing Objective, extending principles from diffusion models, to greatly improve world-model training and inference efficiency. The objective takes both the data noise level and the size of the requested timestep as inputs, enabling the model to directly predict the terminal future state with only a few sampling steps. Concretely, it is based on a flow-matching formulation that predicts the velocity vector between a clean data point and a Gaussian-noise point , approximating the solution of the probability differential equation: . Unlike prior diffusion models that require dozens of sampling steps, this enables accurate video prediction in as few as 4 steps.

Dreamer 4 departs from DreamerV3’s RNN-based RSSM and adopts an efficient Block-Causal Transformer architecture suitable for high-resolution video. The input video is tokenized into high-resolution patches (360×640), and specialized layers jointly process spatial and temporal information to maintain a long context (9.6 seconds / 192 frames). It also uses a Masked Autoencoder (MAE) tokenizer to effectively compress and reconstruct visual details.

Dreamer 4 is the first agent to obtain diamonds in Minecraft using only offline data (the VPT dataset), without online interaction with the environment.

Training Agents Inside of Scalable World Models

World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world models have been...