Seonglae Cho- The essence of an LLM loop isn’t “autonomy”; it’s “verifiable iteration.”
- It’s not that the agent is smart and will eventually solve it if you let it run.
- You need a mechanical way to judge success/failure at every iteration.
- In a good loop, the exit condition matters more than the prompt.
- You need a clear stop signal like “tests pass,” “compile succeeds,” “metric improves,” or “zero exit code.”
- Without an exit condition, a loop is closer to gambling than engineering.
- Separate what the agent decides from what the code must enforce.
- LLM: ambiguous judgment, code generation, diagnosis.
- Deterministic code: execution, verification, logging, retry caps, rollback, diff checks.
- An agent loop is ultimately a software architecture problem.
- Loops where an agent evaluates itself are risky.
- It can keep rationalizing within the same flawed logic.
- It’s safer to separate the task agent and the evaluator agent.
- The biggest risk isn’t cost; it’s humans losing understanding.
- The faster an agent produces code, the less people know “why it ended up this way.”
- When a production failure happens later, debugging becomes very difficult.
- Planning loop → Execution loop → feed the results back into planning. Separating deciding what to build from actually building it was reportedly the single biggest efficiency gain.
- Humans don't review all the code; they only approve chokepoints like product direction, scope changes, and infra/data model decisions.
- The principle is: when agent output is bad, don't fix it by hand — fix the harness/rules and re-run. Improve the machine, not the output.
- Wherever possible, don't leave verification to the LLM; enforce it with deterministic checks like lint, type systems, static analysis, and executable specs.
- In the planning stage, multiple agents repeatedly critique the spec, applying "shift-left validation" to catch as many problems as possible before implementation.
- Implementation runs in parallel according to a ticket dependency graph → separate agent review/E2E → integration tests → deploy → prod smoke tests, all automated. Failures and fixes flow back into the planning loop.
