Texonom
Texonom
/
Science
Science
/Mathematics/Math Field/Statistics/Statistical Model/Model Generalization/Model Training/Fine Tuning/Model Adaptation/
Intrinsic Dimensionality
Search

Intrinsic Dimensionality

Creator
Creator
Seonglae Cho
Created
Created
2024 Jul 25 3:15
Editor
Editor
Seonglae Cho
Edited
Edited
2024 Jul 30 8:54
Refs
Refs
LoRA
의 이론적 근거로 adaptation 중 intrinsic rank가 낮다는 것을 근거로 든다
Adaptation layer의 intrinsic rank(
Matrix Rank
)가 낮아서 그걸 저차원 행렬의 곱으로 분해해서 weight parameter 수를 줄인게 lora
 
 
Intrinsic Dimensionality Explains the Effectiveness of Language...
Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well...
Intrinsic Dimensionality Explains the Effectiveness of Language...
https://arxiv.org/abs/2012.13255
Intrinsic Dimensionality Explains the Effectiveness of Language...
Measuring the Intrinsic Dimension of Objective Landscapes
Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty...
Measuring the Intrinsic Dimension of Objective Landscapes
https://arxiv.org/abs/1804.08838
Measuring the Intrinsic Dimension of Objective Landscapes
 
 

Recommendations

Texonom
Texonom
/
Science
Science
/Mathematics/Math Field/Statistics/Statistical Model/Model Generalization/Model Training/Fine Tuning/Model Adaptation/
Intrinsic Dimensionality
Copyright Seonglae Cho