Seonglae ChoLooped Transformers as Programmable Computers
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard,...
https://arxiv.org/abs/2301.13196
Looped Transformers are Better at Learning Learning Algorithms
Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative...
https://arxiv.org/abs/2311.12424
training
Training-Free Looped Transformers
We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning,...
https://arxiv.org/abs/2605.23872
moe transformer trick that reuse middle layers for training efficiency
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study...
https://arxiv.org/abs/2609.01343