Seonglae ChoVariable-Width Transformers
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers,...
https://arxiv.org/abs/2606.18246
