Parameter-Efficient Fine-Tuning
Operating efficiency is greatly improved when deploying, as you only need to adjust some parameters, learn an external module for a new task, and save and load parameters for each task. PEFT also uses less memory.
PEFT Usages
IA3
Few layer-wise representation’s Scaling coefficient for loss with Activation Engineering in some sense it detour Layer Normalization.