Seonglae ChoRecurrent Transformer
Weight Sharing with LoRA variants
Recursive Transformer Methods
RNN
History
Universal Transformers (2018) and ALBERT (2019) showed that weight sharing works. But they were either not at LLM scale, or they lost to regular (untied) models using the same compute.
Ouro, ByteDance Seed et al. is the key paper missing from the first answer. It pretrained a looped LLM on roughly 7.7T tokens and added an adaptive exit that learns when to stop looping. Saunshi et al. (Google, ICLR 2025) provided the theoretical grounding, proving that repeating a k-layer block L times approximates a kL-layer model, and that T iterations can simulate T steps of CoT.
Coconut correction: this is not depth-wise looping but feeding hidden states back along the sequence direction, so it should be treated as a nearby but separate branch. Mixture-of-Recursions (July 2025) routes a different recursion depth per token. Relaxed Recursive Transformers (2024) converts an already-trained LLM into a looped model.