Seonglae ChoLoRA uses the theoretical foundation that the intrinsic rank during adaptation is low
LoRA reduces the number of weight parameters by decomposing the adaptation layer's low intrinsic rank (Matrix Rank) into a product of low-dimensional matrices
Intrinsic Dimensionality Explains the Effectiveness of Language...
Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well...
https://arxiv.org/abs/2012.13255
Measuring the Intrinsic Dimension of Objective Landscapes
Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty...
https://arxiv.org/abs/1804.08838