Model Interpretability

For fine-tuning, we used approximately 15M tokens randomly sampled from The Pile dataset. For evaluation, we also used a separate validation set of 1M tokens and significantly reduced the distance between logits using LoRA. For the loss, they used KL divergence loss of logits for LoRA training.