Perplexity Interview

Anthropic

You are my coding assistant for a time-boxed ML takehome (5 hours). Priorities:
1) Every change must be reproducible: set seeds, log configs, save outputs.
2) When unsure, propose 2-3 options with tradeoffs and pick one.
3) Never invent results. If an experiment hasn’t been run, say so.
4) For any metric, report mean±std over >=3 seeds when feasible.
Ask me at most one clarifying question at a time; otherwise proceed with best assumptions.

Scan the repository and answer:
- What is the main entrypoint to run baseline experiments?
- What are the datasets / models used?
- Where are configs/args parsed?
- Where are results saved?
Then propose the fastest path to run a baseline end-to-end on GPU.
Provide exact commands to run.

Make the smallest changes needed to:
- run training on CUDA if available,
- set deterministic seeds (python/numpy/torch/cuda),
- log key metrics every N steps/epochs,
- save a checkpoint and a JSONL/CSV of metrics.
Show a patch/diff and the exact run command.
Do not refactor.

Add an experiment output structure:
runs/<exp_name>/<timestamp>/{config.json, metrics.jsonl, final.ckpt, stdout.log}
and ensure every run writes its config and git commit hash if available.
Keep changes minimal. Provide patch + command.

Given this research question:
<PASTE QUESTION>
Design an experiment plan that fits in 5 hours:
- 1 baseline
- 2-4 ablations/variants
- 3 seeds each where feasible
- expected runtime per run (rough)
- what plots/tables are needed
Also list possible confounds and how to mitigate them.
Output as a checklist.

Implement an ablation for <CHANGE> with a single config flag.
Requirements:
- default behavior unchanged
- logs include the flag value
- results saved under runs/<exp_name>
Provide patch + 2 example commands: baseline vs ablation.

Create a script analyze.py that:
- reads multiple runs folders
- aggregates metrics over seeds (mean/std)
- produces 2-3 plots (train curve, val metric, final comparison bar)
- writes a summary.md with key numbers
Use matplotlib only. Minimal dependencies.
Provide patch + example usage command.

We need to run local inference on a HF transformers model and extract:
- tokenized inputs
- generated outputs with sampling controls
- logits and logprobs for specific tokens
- optionally hidden states / attention
Write a minimal utility module hf_infer.py with functions:
load_model(name, device, dtype), run(prompt, **gen_kwargs),
get_logprobs(prompt, target_tokens), get_hidden(prompt, layer)
Provide a patch and a tiny demo command.
Avoid heavy abstractions.

Add knowledge distillation option:
- teacher provides logits (precomputed or on-the-fly)
- student loss = alpha*CE + (1-alpha)*T^2*KL(student||teacher) with temperature T
Implement with one flag: --distill --alpha --temperature.
Default training unchanged.
Provide patch + commands.
Also add a sanity check that distillation loss decreases on a small batch.

Before we trust results, audit the code for:
- train/val leakage
- eval mode vs train mode
- seed handling
- device placement bugs
- metric computation mistakes
- batching/tokenization alignment issues
Provide a prioritized list of risks + how to test each quickly.
If you find any bug, propose the smallest fix as a patch.

Draft report.md using this template:
- Question
- Setup (hardware, runtime, seeds)
- Methods
- Experiments (baseline + variants)
- Results (with placeholders referencing plots)
- Discussion (interpretation + confounds)
- Limitations + Next steps
Use bullet points and be precise. Do not fabricate numbers.
Insert TODO markers where numbers/figures should go.

(A) 0:00~0:40 baseline

Get baseline running now. Optimize for speed. Patch + command only.

(B) 0:40~3:20 ablations + seeds

Implement variant #1/#2/#3 with flags, then produce a run matrix (commands) for 3 seeds each.

(C) 3:20~4:10 analysis

Aggregate runs and generate plots + summary.md.

(D) 4:10~5:00 report

Generate report.md skeleton, then fill with the summary numbers and key plots references.

Do not refactor.
Do not change defaults.
Do not claim results unless we ran them and have saved outputs.

# Takehome Assessment – Seonglae Cho

Date: 2026-02-XX  
Runtime: ~4.5 hours  
Environment: RTX 3090, CUDA XX.X