Looped Transformer

Looped Transformers as Programmable Computers

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard,...

https://arxiv.org/abs/2301.13196

Looped Transformers are Better at Learning Learning Algorithms

Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative...

https://arxiv.org/abs/2311.12424

training

Training-Free Looped Transformers

We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning,...

https://arxiv.org/abs/2605.23872

moe transformer trick that reuse middle layers for training efficiency

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study...

https://arxiv.org/abs/2609.01343

RLT: Recurrent Looped Transformer

recurrent-looped-tranformer

yifanzhang-pro • Updated 2026 Oct 23 0:53

Diffusion Transformer

arxiv.org

https://arxiv.org/pdf/2609.40305

InfiLoop

기존 Looped Transformer는 마지막 recurrent state만 다음 iteration에 전달하기 때문에, 반복 횟수를 늘리면 이미 올바르게 계산한 중간 결과가 noisy update에 의해 사라지는 latent overthinking (

AI Overthinking) 문제가 발생할 수 있다. InfiLoop는 이전 iteration들의 출력을 content-based weighting과 learned temporal decay로 결합하는 새로운 residual mechanism을 제안한다. 정확한 streaming recurrence를 통해 과거 결과를 누적하므로 반복 횟수가 늘어도 persistent aggregation memory는 일정하게 유지된다. 추론 depth를 무작정 늘리는 대신, 이전 iteration에서 얻은 올바른 latent state를 보존하는 mechanism이 중요하다는 연구다.

Scaling to Tens of Thousands of Test-Time Iterations with...

In this paper, we argue that Looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We first introduce an interesting...

https://arxiv.org/abs/2610.11570

Looped Transformer

RLT: Recurrent Looped Transformer

Diffusion Transformer

InfiLoop

Backlinks

Recommendations