Intrinsic Dimensionality

Creator

Creator

Created

Created

2024 Jul 25 3:15

Editor

Editor

Edited

Edited

2024 Jul 30 8:54

Refs

Refs

LoRA 의 이론적 근거로 adaptation 중 intrinsic rank가 낮다는 것을 근거로 든다

Adaptation layer의 intrinsic rank(

Matrix Rank)가 낮아서 그걸 저차원 행렬의 곱으로 분해해서 weight parameter 수를 줄인게 lora

Intrinsic Dimensionality Explains the Effectiveness of Language...

Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well...

Intrinsic Dimensionality Explains the Effectiveness of Language...

https://arxiv.org/abs/2012.13255

Intrinsic Dimensionality Explains the Effectiveness of Language...

Measuring the Intrinsic Dimension of Objective Landscapes

Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty...

Measuring the Intrinsic Dimension of Objective Landscapes

https://arxiv.org/abs/1804.08838

Measuring the Intrinsic Dimension of Objective Landscapes

Recommendations

//////////