Seonglae ChoFunction Vector, Task feature, Task Circuit, Task representation
sparse SAE task vector fine-tuning (gradient-based cleanup)
Obtain a more accurate steering vector through gradient-based cleanup of the steering vector obtained from the SAE decoder since it has reconstruction error with linear combination of SAE features.
Gradient-based cleanup
Fine-tuning is applied to the target vector to efficiently reconstruct neuron activation patterns present in the residual using the SAE basis. Through gradient-based cleanup, features with small gradients were removed to create a compact SAE. This shows improved performance compared to the existing task vector and provides interpretability.
AI Task vectors
TaskVec 2023 EMNLP 2023
Task vectors play an important role in In-context learning (2023) .
ICL compresses a set of examples S into a single task vector , then generates answers to question x using only θ without directly referencing S. In other words, ICL can be decomposed into: learning stage + application stage . is extracted from intermediate layer representations and is a stable, distinguishable vector for each task. When is patched, even tasks different from the original examples are performed based on is dominant. This decomposition still maintains 80–90% of general ICL accuracy.
Function Vector 2023.10 ICLR 2024
A small number of intermediate layer attention heads causally transmit task information in ICL. Even when FV is inserted into zero-shot or natural language contexts, the task is executed with transferability. FV cannot be explained by output word distribution alone and triggers nonlinear computation.
Visual task vector using Policy Gradient Learning
SAE TaskVector 2024
Sparse Autoencoder features mimic task vector steering based on Task detector features and task feature from separator token’s residual mean as task vector with Gradient based Cleanup (2024)
In-context learning
TVP-loss to emerge task vector of ICL into specific layer (2025)
Instruct Vector from base model to instruction model
It presents a methodology for extracting and analyzing task vectors from the internal activations of vision–language models (VLMs). The main finding is that VLMs learn a shared task representation across different modalities (text vs. image) and across different input formats (few-shot examples vs. instructions). High cosine similarity between cross-modal task vectors supports the existence of a modality-invariant shared representation. It also demonstrates that a task vector extracted from one modality can be applied to inputs in another modality and still enable task performance, providing evidence for cross-modal transferability of task vectors.