Seonglae ChoLoRA 의 이론적 근거로 adaptation 중 intrinsic rank가 낮다는 것을 근거로 든다
Adaptation layer의 intrinsic rank(Matrix Rank)가 낮아서 그걸 저차원 행렬의 곱으로 분해해서 weight parameter 수를 줄인게 lora
Intrinsic Dimensionality Explains the Effectiveness of Language...
Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well...
https://arxiv.org/abs/2012.13255
Measuring the Intrinsic Dimension of Objective Landscapes
Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty...
https://arxiv.org/abs/1804.08838
