The Drain

By Yianna Kokalas & Claude

How this paper was written. This is a working paper from a production side project: software built for the Magic: The Gathering community, with real users, built and operated by one engineer (an enterprise engineer by day) together with Claude, Anthropic’s AI model. Every mechanism claim was fact-checked against the running code before publication; the fact-check itself found eight places where our own documentation had drifted, and they are published in section 8 rather than quietly fixed. Both names on the byline mean it.

Position: headless autonomy is built on trust, and trust is built in increments. This system began at zero: every action interactive, a human in every loop. Skills proved repeatable, repeatable skills composed into workflows, and each time verification caught up with a new class of mistake, trust was extended one notch further: first to implement, then to be reviewed by agents, then to merge, then to deploy. The gates in this paper are not the opposite of that trust; they are its load-bearing structure. Each one encodes a specific incident where extended trust broke, and the repair that made re-extending it safe. The pipeline’s core discipline follows from the same arc: an agent’s claim of success is never itself the basis for trust; every claim is re-verified against ground truth (GitHub, the test suite, the deploy) before state advances. This paper walks the machine gate by gate, names the incident behind each one, reports the measured numbers, and closes with the part that keeps us honest: eight places where our own documentation had drifted behind the code, found while fact-checking this paper.

Every mechanism claim below was verified against the code at HEAD on 2026-08-26. The file:line citations throughout reference our private orchestrator code, so a reader cannot follow them; they are retained anyway, as evidence that each claim was checked against a specific line of running code rather than recalled from memory.

1. What the drain is

One person approves work; the drain does the rest. A ticket is a prose spec in tickets/<slug>.md (see the first paper in this series, Files as Prose, Store as Truth, for why prose and state live where they do). When the human promotes a ticket, the drain dispatches it as a fresh, headless agent session:

claude -p "/implement <slug>" --permission-mode bypassPermissions \
       --model <pinned> --max-budget-usd 20 --no-session-persistence

That session implements the change in its own git worktree, gets reviewed by two more agents, survives a fix loop, merges its own PR, and hands back a verdict. The drain then deploys, verifies, and marks the ticket done, or files a ticket about why it could not. One dispatch at a time, median 22.7 minutes, roughly $9 of model spend each (internal feasibility study, 2026-08-16, 60-dispatch sample).

The rest of this paper is about everything wrapped around that one command, because the command is the easy part. It is also the top rung of a ladder: none of this was trusted on day one. These same skills ran interactively, with a human watching, before the drain was allowed to run them headless; the gates below are what incidents taught us to require before each next rung of autonomy was safe to climb. (Dating each rung to the evidence that justified it is its own paper, planned in this series as The Trust Ladder.)

2. The ready queue: six gates, every one required

A ticket must pass every gate below to dispatch. The gates run in two layers. Layer 1 is one SQL query inside the store: four checks whose answers are columns the store already holds. Layer 2 is Ruby code for the three checks whose answers live outside the database: it runs git to detect code changes, reads other tickets to resolve dependencies, and parses the ticket body for hold dates. Layer 2 can only remove tickets, never reorder them (ready_queue.rb:33-35); ordering is decided once, in the store query: priority first, alphabetical slug as the tiebreak, so the queue is repeatable across runs (ticket_store.rb:755-763) and “why did B run before A” always has exactly one of two answers: A was removed by a named gate, or A sorts after B.

                    every ticket in the store
                               |
                               v
  +---------------------------------------------------------+
  | LAYER 1: four checks, answered by the database alone    |
  | (one SQL query over columns the store already holds)    |
  |                                                         |
  |  is status "todo"?                        no --> OUT    |
  |  has the human promoted it                              |
  |   (the execution-ready label)?            no --> OUT    |
  |  has the human opted in to auto-merge?    no --> OUT    |
  |   (promotion grants both together; the SQL              |
  |    check on auto_merge stays a fail-closed              |
  |    backstop, e.g. an opt-out flipped after promoting)   |
  |  is it marked interactive                               |
  |   (work that needs a human present)?      yes --> OUT   |
  +----------------------------+----------------------------+
                               |
                               |  survivors exit in FINAL order:
                               |  by priority (critical first),
                               |  ties broken alphabetically
                               v
  +---------------------------------------------------------+
  | LAYER 2: three checks, answered outside the database    |
  | (Ruby code: runs git, reads other tickets, parses       |
  | the ticket body). May REMOVE tickets, never REORDER.    |
  |                                                         |
  |  FRESHNESS GATE                                         |
  |   was the ticket audited in the last 14 days,           |
  |   did that audit conclude "still worth doing",          |
  |   and is the code it referenced unchanged since?        |
  |   any answer no, or no audit stamp at all --> OUT       |
  |                                                         |
  |  SCHEDULED-HOLD GATE                                    |
  |   does the ticket say "do not implement                 |
  |   before <date>", with that date still ahead? --> OUT   |
  |                                                         |
  |  DEPENDENCY GATE                                        |
  |   is every ticket listed in blocked_by finished         |
  |   (done or cancelled)?                                  |
  |   any still open, or listed but not found --> OUT       |
  +----------------------------+----------------------------+
                               |
                               v
        tickets stranded mid-run by a crash go first,
       then this queue dispatches ONE ticket at a time

Two details show the texture of the thing. The freshness gate has exactly one carve-out: SHA drift caused entirely by squash-merges of this ticket’s own cleared blockers admits instead of blocking, resolved by exact filename only, never a glob (ready_queue.rb:497-509). And the forward-date hold accepts both a due + label pair and a literal DO NOT IMPLEMENT BEFORE YYYY-MM-DD line in the body, bound deliberately to the caller’s clock rather than the database’s (ready_queue.rb:267-290).

The queue also runs recovery first: tickets stranded in-progress or verifying by a crashed run re-enter through the verification gate, never through a fresh /implement (bin/drain:859-926). Support-loop tickets are excluded from recovery because their verifying means “waiting on the customer,” not “deploy interrupted” (bin/drain:132-142).

3. Dispatch discipline: the incidents behind the plumbing

Each mechanism here has a date and a scar attached.

• The model is pinned. On 2026-08-19 a bare claude -p inherited the operator’s interactive /model selection and silently ran the whole drain on the most expensive model available. Dispatch now always passes --model (default sonnet), and the planning pipeline pins its phases the same way (bin/drain:1210-1223).
• Kills target the process group. A dispatched session’s grandchild once merged a PR eight minutes after the drain had timed the session out and stamped the ticket blocked. Dispatches now spawn with pgroup: true and the hard-kill TERMs then KILLs the whole group (bin/drain:1458-1462, 2286-2339).
• The wall-clock ceiling is enforced, and its history is a lesson in tuning. Before enforcement, 14 of 14 sampled timeouts overshot the ceiling (median +259s, max +13,636s). The ceiling then went 1800s → 2700s → 3600s, the last bump after 2700s hard-killed a healthy $17 dispatch in the middle of its third fix cycle (bin/drain:177-194, 1425-1429).
• Two locks, deliberately split. A process-lifetime lock (.drain.lock) makes the drain a singleton; a round-scoped lock (.implement-lock) is released between watch rounds. They used to be one lock, until an idle watcher held it for 2h24m and starved the hygiene sweep 100% of the time (bin/drain:304-363).
• Transient API errors get one deterministic retry. After three dispatches in one day died on API Error: 500 and all escalated to a human, 5xx/overloaded now gets exactly one classifier-free retry with a 60s backoff (bin/drain:123-130).
• Budgets are caps with a history. The implement cap went $10 → $15 → $20; the $15 bump came after a review legitimately widened an implementation mid-run and the dispatch died of budget exhaustion after pushing the PR (bin/drain:144-166).
• The dispatched session is told who it is. A system-prompt preamble identifies the session as the drain’s own dispatch, because on 2026-08-18 three dispatches found their own process with ps, concluded another implement was running, and self-aborted having done zero work (bin/drain:1264-1283).

Every dispatch streams a forensic JSONL log, in which two specific pathologies are detected live: an AskUserQuestion call (a headless session asking a question nobody will answer) and output-token truncation (bin/drain:84-87, 1519-1528).

4. The implement pipeline

Inside the dispatch, /implement runs a fixed pipeline: preflight, implementation agent in its own worktree, two parallel reviewers, a bounded fix loop, then auto-merge eligibility.

• Preflight is binding. Clean tree, on main, or the ticket blocks before any agent spawns. Once preflight passes, re-deriving collision state with ps is banned; the 2026-08-18 self-collision incident is why (SKILL.md:48-132, 126).
• Reviewers are chosen per risk, not per habit. The critical reviewer runs on the strongest available model interactively but drops a tier under drain dispatch (cost and latency); the security reviewer is pinned to a model that does not false-refuse on auth-adjacent code, in all modes; the fix agent runs a tier lighter still (SKILL.md:261-283).
• The fix loop is bounded and re-reviews from scratch. At most two cycles; each fix re-runs both reviewers fresh; at the cap the verdicts alone decide. PASS-with-warnings is a clean exit, an operator ruling made after a green PR sat unmerged for a day over advisory nits (SKILL.md:337-391).
• Merged means read-only. A merge-state gate runs before every fix cycle; if the PR merged mid-loop, remaining findings become a follow-up ticket, never a commit to a merged branch (SKILL.md:339-348).
• Auto-merge requires everything. Store-level auto_merge: true, validation PASS, tree-integrity PASS (fails closed if absent), and both reviews PASS (SKILL.md:393-398).
• /implement never writes done. It stops at in-review. Only the drain, holding the deploy evidence, advances in-review → done (SKILL.md:472).

5. Trust, but verify

The drain’s deepest design rule: an agent’s self-report is a claim, not a fact, and trust attaches to evidence rather than to claims. The pipeline re-derives every consequential claim from ground truth, which is exactly what makes it safe to keep extending the agents’ autonomy: verification is the mechanism trust rides on.

• A dispatch that dies without a parseable verdict but whose PR is confirmed merged on GitHub is done, not blocked; an infra flake must never bury finished work (bin/drain:1336-1344).
• A timeout kill with a merged PR is done with an honest “merged but unverified” note (bin/drain:1346-1371).
• A session self-reporting auto_merged: yes is never believed on its own; merge state is confirmed against GitHub before the verification gate runs (bin/drain:4426-4474).
• The skipped-deploy path (repos with no deploy gate) still demands probe-confirmed PR merge before done. Origin: on 2026-08-14 a retry self-reported success with its PR still open, and the drain stamped done fourteen seconds later (bin/drain:2794-2861).
• The inverse failure is also guarded: a merged PR once read as unmerged because of GitHub index lag, so the probe retries and requires two consecutive clean empties before declaring anything unmerged (bin/drain:3786-3834).

Failures that survive these short-circuits go to a local model for classification, and the same discipline applies to the classifier itself: the deterministic recovery table is authoritative over the model’s retry suggestion, and only two categories (both “your clone is behind”) are retryable at all; the other eighteen escalate to a human (classifier.rb:23-65).

6. After the merge: verify, heal, or file a ticket

Merging is not done. The drain moves the ticket to verifying and runs the repo’s deploy, which carries its own gate ordering (for the backend: Brakeman → Rubocop → Minitest → Playwright smoke), so a failure signature also proves which gates never ran (bin/drain:2944-2953).

On a red deploy with a live worktree, a fixer agent gets at most three attempts at $10 each, with the attempt count persisted in the store so a crashed drain cannot reset the ceiling; worst case a ticket costs $20 + 3 × $10 = $50 and stops (bin/drain:2055-2106). On a red deploy after the worktree is gone, no fixer runs; classification is deterministic by failure signature, and regressions are attributed only by changed-file overlap with the merged diff, so an unrelated flake never blames the merged PR (bin/drain:2925-2972, 3568-3595).

When the drain cannot finish a ticket’s story, it files a ticket about it. Six cohorts, each with a deliberate routing decision (drain-followups skill; writers in bin/drain):

Every auto-filed ticket is registered in the store at write time; before that rule, three follow-ups existed as files with no store row and “silently did not exist” (bin/drain:2043-2051).

7. The numbers, honestly

From the measured discoveries (2026-08-16 sample of 60 dispatch logs; 2026-08-23 window of 55 dispatches):

• Median dispatch 22.7 minutes; mean cost ~$9.10; $2,171.82 total model spend across the first 427 dispatch logs.
• Completion rate 66% in the August sample; done-rate 43.6% in the harder 08-23 window, where the top failure clusters were dispatch collisions (22.6%), timeouts (19.4%), and transient API errors (12.9%).
• Telemetry volume: 6,570 rows in an 11-day window, 99.16% of them heartbeats.
• Deploy verification is nearly free. A natural suspicion is that per-ticket deploys drive the dispatch time; measured against the store’s event log (30 days, 256 done tickets, fixture-validated), the deploy phase is a median 3.75 minutes, occurs on only 56 of 256 tickets, and is 0.3% of summed wall-clock. The implement dispatch itself (tests, two reviews, fix cycles) is the ~29-minute phase. Meanwhile median first-touch-to-done is 88 minutes: most of a ticket’s life is spent between phases (serial dispatch, gaps before the verification gate, recovery waits), which is where any future throughput work belongs.
• The bottleneck is not execution. Of 368 todo tickets at measurement time, only 5 passed every gate. The constraint is the human approval gate and model quota, not ticket supply and not host placement (the internal feasibility study).

That last number is the honest headline: the drain can execute more than its operator can responsibly approve. The gates are doing exactly what they were built to do, and the system’s throughput limit is, by design, a person.

8. What we believed vs what the code does

This paper was written under a rule: every mechanism claim gets verified against the code before it gets printed. That fact-check found eight places where our own documentation, memory notes, or prior discoveries had drifted behind the code. Recorded here both as corrections and as evidence for the rule.

1. We believed unresolvable blocked_by refs failed OPEN (a memory note says so). The code fails CLOSED: a blocker with no store row and no file blocks the ticket (ready_queue.rb:338-340).
2. The 24x7 feasibility discovery’s quota arithmetic assumed dispatches inherit an Opus-class model; dispatch has pinned --model (default sonnet) since 08-19, so that arithmetic no longer describes the dispatch path (bin/drain:1220-1223).
3. Telemetry’s model label defaults to an Opus-class string while the dispatched model defaults to sonnet; with the env var unset, telemetry mislabels every dispatch. A real bug, found by this audit, now ticketed (bin/drain:2167 vs 1220-1223).
4. The fixer-agent skill description still says $5 per attempt; the default is $10 (half the $20 implement cap), and a stale worst-case comment underquotes the real $50 ceiling (bin/drain:166, 1827-1830).
5. The project’s instructions file still describes hygiene stamps living in frontmatter; since the store-only cutover the sweep stamps the store and the gate reads it there, so frontmatter copies will disagree (ready_queue.rb:20-32).
6. The feasibility discovery’s 2700s ceiling was already stale by publication + one week: 3600s since 08-19 (bin/drain:185-194).
7. The 08-23 triage’s “classifier only knows 8 environment-drift categories” was fixed by its own suggested ticket: the full 18-category taxonomy shipped, all deliberately non-retryable (classifier.rb:46-65).
8. Minor: a legacy sort key implies execution-ready affects ordering; on the live path ordering is purely the store query, and the key is a no-op (ready_queue.rb:612-618).

The pattern in the drift is consistent: the code gets fixed at incident speed, the prose gets fixed at documentation speed, and the gap between the two is measured in weeks. That is precisely why the fail-closed gates read the code-adjacent store rather than prose, and why this series verifies against code, not against our own notes.

9. Open problems

• Escalation labels are still too coarse in practice: 93.5% of non-done outcomes in the 08-23 window wore the unknown label; the deterministic envelope-first labeling shipped since, but the window’s data shows how blind a busy pipeline can be about its own failures.
• fix_cycle_count was 0 on 55 of 55 telemetry rows in that window: either fix cycles are rarer than believed or the counter has a wiring gap. Unresolved.
• The promote gate is the throughput ceiling (5 of 368 tickets dispatchable). Whether to widen it is a judgment about trust, not a technical gap; so far the answer is no.
• Quota economics: steady-state ~6h/day of drain fits the measured ticket-generation rate; 24/7 remains uneconomical on current plan quotas even with the cheaper pinned dispatch model. Re-measure before believing either number again (see section 8, items 2 and 6).

10. References

• The companion fact sheet (internal): every claim above with its full citation, compiled by reading the orchestrator source end to end before this paper was written.
• Internal measurement artifacts: the 24/7 feasibility study (2026-08-16) and the dispatch-outcome triage (2026-08-23), both quoted with their sample sizes in section 7.
• Files as Prose, Store as Truth: the substrate the drain runs on, and paper 1 of this series.
• The machine itself: the orchestrator, ready-queue, classifiers, and implement skill, private, at HEAD 2026-08-26.

This is a paper in the Yianna & Claude Whitepapers series: working papers on running a production side project at the output of a team, with a fleet of AI agents, alongside a full-time engineering job, written from the system’s own measurements and incident reports. Evidence citations reference our internal decision records and postmortems; where a claim could not be traced to a measurement, the text says so.