Diffusion Transformer

DiT

BERT/MLM is essentially a special form of text diffusion model, and by adding a masking schedule and iterative denoising, it can be used as a complete generative language model.

BERT (especially RoBERTa) is originally an encoder model that performs MLM (masked token recovery). However, by reinterpreting it as a discrete text diffusion process where the masking ratio changes like time steps, it can be transformed into a full generative model. By training with varying masking ratios from 0% to 100%, BERT can also generate text in a diffusion manner. By adding only variable masking and iterative denoising to RoBERTa and fine-tuning on WikiText, quite natural sentence generation was successfully achieved.

BERT is just a Single Text Diffusion Step

This article appeared on Hacker News. Link to the discussion here. A while back, Google DeepMind unveiled Gemini Diffusion, an experimental language model that generates text using diffusion. Unlike traditional GPT-style models that generate one word at a time, Gemini Diffusion creates whole blocks of text by refining random noise step-by-step. I read the paper Large Language Diffusion Models and was surprised to find that discrete language diffusion is just a generalization of masked language modeling (MLM), something we’ve been doing since 2018. The first thought I had was, “can we finetune a BERT-like model to do text generation?” I decided to try a quick proof of concept out of curiosity.

https://nathan.rs/posts/roberta-diffusion/

Omni

For image tokens apply diffusion loss for transformer omni

サメQCU on Twitter / X

bros, DiT is wrong.it's mathematically wrong.it's formally wrong. there is something wrong with it pic.twitter.com/OQZ8IcQfnA— サメQCU (@sameQCU) August 17, 2025

https://x.com/sameQCU/status/1957223774094585872

Traditional DiT (=Diffusion Transformer) relies on outdated VAE encoders → low-dimensional (latent 4ch), complex structure, weak expressiveness. Instead of VAE, using pre-trained representation encoders (DINO, MAE, SigLIP, etc.) + lightweight decoder combination = Representation Autoencoder (RAE). L1+

GAN+

LPIPS loss

arxiv.org

https://arxiv.org/pdf/2510.11690

Diffusion Transformer

DiT

Omni

Backlinks

Recommendations