LoRA

Creator

Creator

Seonglae Cho

Created

Created

2023 Jun 22 15:24

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Oct 16 20:12

Refs

Refs

microsoft • Updated 2024 Oct 16 17:40

Model Adaptation

Low-Rank Adaptation

LoRA is a technique that optimizes rank decomposition matrices with LoRA module rank .

The existing PEFT technology reduces the available sequence length of the model or expands the model depth for inference. LoRA usually applied to MLP by two low-rank matrix before activation.

Free weight and append adaptation layer composed by adaptation matrix with proving low-rank adaptation matrix is sufficient for fine-tuning. By dividing adaptation matrix like

Bottleneck layer, It makes total model size smaller and make it achieve same learning rate with less computing resources.

A learning rate of 1e-4 has become the standard when fine-tuning LLMs with LoRA. Although we occasionally encountered training loss instabilities, reducing the learning rate to lower values like 3e-5. LoRA’s weight is initialized randomly but techniques like

EVA uses activation vector decomposing it to initialize based on priority.

LoRA Usages

Model Regularization

LoRA Learns Less and Forgets Less

https://arxiv.org/pdf/2405.09673

It is enough to change
Cross-Attention layer for fine tuning

LoRA: Low-Rank Adaptation of Large Language Models

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full...

LoRA: Low-Rank Adaptation of Large Language Models

https://arxiv.org/abs/2106.09685

LoRA: Low-Rank Adaptation of Large Language Models

Fine-Tuning LLMs: In-Depth Analysis with LLAMA-2 | Anyscale

In this blog, we compare full-parameter fine-tuning with LoRA and answer questions around the strengths and weaknesses of the two techniques.

Fine-Tuning LLMs: In-Depth Analysis with LLAMA-2 | Anyscale

https://www.anyscale.com/blog/fine-tuning-llms-lora-or-full-parameter-an-in-depth-analysis-with-llama-2

Fine-Tuning LLMs: In-Depth Analysis with LLAMA-2 | Anyscale

[논문리뷰] LoRA: Low-Rank Adaptation of Large Language Models

LoRA 논문 리뷰

https://kimjy99.github.io/논문리뷰/lora/

Tips

Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)

Things I Learned From Hundreds of Experiments

Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)

https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms

Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)

Recommendations

//////////