Function Vector, Task feature
sparse SAE task vector fine-tuning (gradient-based cleanup)
Obtain a more accurate steering vector through gradient-based cleanup of the steering vector obtained from the SAE decoder since it has reconstruction error with linear combination of SAE features.
Gradient-based cleanup
Fine-tuning is applied to the target vector to efficiently reconstruct neuron activation patterns present in the residual using the SAE basis. Through gradient-based cleanup, features with small gradients were removed to create a compact SAE. This shows improved performance compared to the existing task vector and provides interpretability.
In-context learning
TVP-loss to emerge task vector of ICL into specific layer (2025)
Neuron SAE features mimic task vector steering based on Task detector features and task feature from separator token’s residual mean as task vector with Gradient based Cleanup (2024)
Task vectors play an important role in in-context learning and appear in latter layers (2023)
icl_task_vectors
roeehendel • Updated 2025 Feb 13 5:38
Finding visual task vector using Policy Gradient Learning
function vector