Seonglae ChoIntroduces a variable-width architecture for decoder-only transformers, keeping the early and late layers wide while making the middle layers narrower.
Variable-Width Transformers do not change width during inference; the width simply differs by layer. Residual connections use parameter-free carry-forward residual resizing, which copies inactive coordinates so dimensional changes work smoothly without learnable projection layers.
