The Drain

Published 2026-08-30 · Revised 2026-09-13 · Revision history

Technical verification: 2026-08-26; scope and evidence are described in the article.

By Yianna Kokalas & Claude

How this paper was written. This is a living document from a production side project: software built for the Magic: The Gathering community, built and operated by only Claude and me. Every mechanism claim was fact-checked against the current system before any publication.

Position: autonomy is built on trust and that trust was earned over time. This system began at zero: every action interactive, a human at every step. Over time, skills proved repeatable and reliable. Then those skills chained together to make workflows. As those workflows grew, new problems surfaced. We learned what to watch for and what to check before the system could move forward. Those lessons became checks built into our workflows. Being able to see what happened and verify the results with logging gave us the confidence for more autonomy.

1. What the drain is

One person approves work; the drain does the rest. A ticket is a prose spec: a Markdown file describing the problem, the intended change, and what done looks like. The file lives in the vault at tickets/<slug>.md (see the first paper in this series, Files as Prose, Store as Truth, for why prose and state live where they do).

# Keep collection search after editing a card

## Objective
Keep the current search when a collector saves a card edit.

## In Scope
Preserve the search text and matching card list after save.

## Out of Scope
Changing search behavior or saving filters between visits.

## Implementation Shape
Refresh the edited card without resetting the search state.

## Acceptance Criteria
- [ ] Saving a card edit keeps the current search text.
- [ ] The matching list shows the updated card details.
- [ ] Clearing the search still shows the full collection.

## Test Plan
Search, edit a matching card, save, and verify the search remains.

A ticket must be marked execution-ready and have auto_merge: true to enter the drain. Tickets labeled interactive stay out of this queue; those run in a session with a human. Once the remaining readiness checks pass, the drain dispatches the ticket as a fresh, headless agent session:

claude -p "/implement <slug>" --permission-mode bypassPermissions \
       --model <pinned> --max-budget-usd 20 --no-session-persistence

That session implements the change in its own git worktree, gets reviewed by two more agents, works through any required fixes, and merges its PR when the checks pass. It hands back a verdict: a structured report of the outcome. After confirming the merge against GitHub, the drain deploys, verifies, and marks the ticket done, or files a ticket about why it could not.

---DISPATCH RESULT---
ticket: keep-collection-search
status: SUCCESS
auto_merged: "yes"
pr_url: <pull-request-url>
implement_fix_cycles: 1
implement_review_verdict: "critical:PASS security:PASS"
---END---

One dispatch at a time, median 22.7 minutes, roughly $9 of model spend each.

These workflows first ran with a human watching. We gave the system more responsibility as we learned how to catch mistakes and verify its work. The sections below explain the checks that made that possible and the incidents behind them.

2. The ready queue: six gates, every one required

A ticket must pass every gate below to dispatch. The gates run in two layers. Layer 1 is one SQL query inside the store: four checks whose answers are columns the store already holds. Layer 2 is Ruby code for the three checks whose answers live outside the database: it runs git to detect code changes, reads other tickets to resolve dependencies, and parses the ticket body for hold dates. Layer 2 filters out tickets that fail the remaining checks. Tickets that pass keep the order established by the database query. The query sorts by priority first, with alphabetical slug as the tiebreak, so the queue is repeatable across runs and “why did B run before A” always has exactly one of two answers: A was filtered out by a named gate, or A sorts after B.

                    every ticket in the store
                               |
                               v
  +---------------------------------------------------------+
  | LAYER 1: four checks, answered by the database alone    |
  | (one SQL query over columns the store already holds)    |
  |                                                         |
  |  is status "todo"?                        no --> OUT    |
  |  has the human promoted it                              |
  |   (the execution-ready label)?            no --> OUT    |
  |  has the human opted in to auto-merge?    no --> OUT    |
  |   (promotion grants both together; the SQL              |
  |    check on auto_merge stays a fail-closed              |
  |    backstop, e.g. an opt-out flipped after promoting)   |
  |  is it marked interactive                               |
  |   (work that needs a human present)?      yes --> OUT   |
  +----------------------------+----------------------------+
                               |
                               |  survivors exit in FINAL order:
                               |  by priority (critical first),
                               |  ties broken alphabetically
                               v
  +---------------------------------------------------------+
  | LAYER 2: three checks, answered outside the database    |
  | (Ruby code: runs git, reads other tickets, parses       |
  | the ticket body). Filters tickets; preserves order.     |
  |                                                         |
  |  FRESHNESS GATE                                         |
  |   was the ticket audited in the last 14 days,           |
  |   did that audit conclude "still worth doing",          |
  |   and is the code it referenced unchanged since?        |
  |   any answer no, or no audit stamp at all --> OUT       |
  |                                                         |
  |  SCHEDULED-HOLD GATE                                    |
  |   does the ticket say "do not implement                 |
  |   before <date>", with that date still ahead? --> OUT   |
  |                                                         |
  |  DEPENDENCY GATE                                        |
  |   is every ticket listed in blocked_by finished         |
  |   (done or cancelled)?                                  |
  |   any still open, or listed but not found --> OUT       |
  +----------------------------+----------------------------+
                               |
                               v
        tickets stranded mid-run by a crash go first,
       then this queue dispatches ONE ticket at a time

Two details show the texture of the thing. The freshness gate has exactly one carve-out: SHA drift caused entirely by squash-merges of this ticket’s own cleared blockers admits instead of blocking, resolved by exact filename only, never a glob. And the forward-date hold accepts both a due + label pair and a literal DO NOT IMPLEMENT BEFORE YYYY-MM-DD line in the body, bound deliberately to the caller’s clock rather than the database’s.

The queue also runs recovery first: tickets stranded in-progress or verifying by a crashed run re-enter through the verification gate, never through a fresh /implement. Support-loop tickets are excluded from recovery because their verifying means “waiting on the customer,” not “deploy interrupted”.

3. Dispatch discipline: the incidents behind the plumbing

Each mechanism here has a date and a scar attached.

• The model is pinned. On 2026-08-19 a bare claude -p inherited the operator’s interactive /model selection and silently ran the whole drain on the most expensive model available. Dispatch now always passes --model (default sonnet), and the planning pipeline pins its phases the same way.
• Kills target the process group. A dispatched session’s grandchild once merged a PR eight minutes after the drain had timed the session out and stamped the ticket blocked. Dispatches now spawn with pgroup: true and the hard-kill TERMs then KILLs the whole group.
• The wall-clock ceiling is enforced, and its history is a lesson in tuning. Before enforcement, 14 of 14 sampled timeouts overshot the ceiling (median +259s, max +13,636s). The ceiling then went 1800s → 2700s → 3600s, the last bump after 2700s hard-killed a healthy $17 dispatch in the middle of its third fix cycle.
• Two locks, deliberately split. A process-lifetime lock (.drain.lock) makes the drain a singleton; a round-scoped lock (.implement-lock) is released between watch rounds. They used to be one lock, until an idle watcher held it for 2h24m and starved the hygiene sweep 100% of the time.
• Transient API errors get one deterministic retry. After three dispatches in one day died on API Error: 500 and all escalated to a human, 5xx/overloaded now gets exactly one classifier-free retry with a 60s backoff.
• Budgets are caps with a history. The implement cap went $10 → $15 → $20; the $15 bump came after a review legitimately widened an implementation mid-run and the dispatch died of budget exhaustion after pushing the PR.
• The dispatched session is told who it is. A system-prompt preamble identifies the session as the drain’s own dispatch, because on 2026-08-18 three dispatches found their own process with ps, concluded another implement was running, and self-aborted having done zero work.

Every dispatch streams a forensic JSONL log, in which two specific pathologies are detected live: an AskUserQuestion call (a headless session asking a question nobody will answer) and output-token truncation.

4. The implement pipeline

Inside the dispatch, /implement runs a fixed pipeline: preflight, implementation agent in its own worktree, two parallel reviewers, a bounded fix loop, then auto-merge eligibility.

• Preflight is binding. Clean tree, on main, or the ticket blocks before any agent spawns. Once preflight passes, re-deriving collision state with ps is banned; the 2026-08-18 self-collision incident is why.
• Models are assigned by role and execution mode. The critical reviewer uses Fable interactively and Opus under drain dispatch. The security reviewer uses Opus in both modes, and the fix agent uses Sonnet. These choices include assumptions we have not yet validated through comparisons. Our later security-review experiment did not support the original false-refusal rationale.

• The fix loop is bounded and re-reviews from scratch. At most two cycles; each fix re-runs both reviewers fresh; at the cap the verdicts alone decide. PASS-with-warnings is a clean exit, an operator ruling made after a green PR sat unmerged for a day over advisory nits.
• Merged means read-only. A merge-state gate runs before every fix cycle; if the PR merged mid-loop, remaining findings become a follow-up ticket, never a commit to a merged branch.

• Auto-merge requires everything. Store-level auto_merge: true, validation PASS, tree-integrity PASS (fails closed if absent), and both reviews PASS.
• /implement never writes done. It stops at in-review. Only the drain, holding the deploy evidence, advances in-review → done.

5. Trust, but verify

The drain’s deepest design rule: an agent’s self-report is a claim, not a fact, and trust attaches to evidence rather than to claims. The pipeline re-derives every consequential claim from ground truth, which is exactly what makes it safe to keep extending the agents’ autonomy: verification is the mechanism trust rides on.

• A dispatch that dies without a parseable verdict but whose PR is confirmed merged on GitHub is done, not blocked; an infra flake must never bury finished work.
• A timeout kill with a merged PR is done with an honest “merged but unverified” note.
• A session self-reporting auto_merged: yes is never believed on its own; merge state is confirmed against GitHub before the verification gate runs.
• The skipped-deploy path (repos with no deploy gate) still demands probe-confirmed PR merge before done. Origin: on 2026-08-14 a retry self-reported success with its PR still open, and the drain stamped done fourteen seconds later.
• The inverse failure is also guarded: a merged PR once read as unmerged because of GitHub index lag, so the probe retries and requires two consecutive clean empties before declaring anything unmerged.

When a failed run doesn’t match a known error pattern, a local model helps identify the likely cause. The ticket then waits for a human to review it. The model helps with diagnosis; it cannot authorize another attempt.

6. After the merge: verify, heal, or file a ticket

Merging is not done. The drain moves the ticket to verifying and runs the repo’s deploy, which carries its own gate ordering (for the backend: Brakeman → Rubocop → Minitest → Playwright smoke), so a failure signature also proves which gates never ran.

On a red deploy with a live worktree, a fixer agent gets at most three attempts at $10 each, with the attempt count persisted in the store so a crashed drain cannot reset the ceiling; worst case a ticket costs $20 + 3 × $10 = $50 and stops. On a red deploy after the worktree is gone, no fixer runs; classification is deterministic by failure signature, and regressions are attributed only by changed-file overlap with the merged diff, so an unrelated flake never blames the merged PR.

When the drain cannot finish a ticket’s story, it files a ticket about it. Six cohorts, each with a deliberate routing decision:

Every auto-filed ticket is registered in the store at write time; before that rule, three follow-ups existed as files with no store row and “silently did not exist”.

7. The numbers, honestly

From the measured discoveries (2026-08-16 sample of 60 dispatch logs; 2026-08-23 window of 55 dispatches):

• Median dispatch 22.7 minutes; mean cost ~$9.10; $2,171.82 total model spend across the first 427 dispatch logs.
• Completion rate 66% in the August sample; done-rate 43.6% in the harder 08-23 window, where the top failure clusters were dispatch collisions (22.6%), timeouts (19.4%), and transient API errors (12.9%).
• Telemetry volume: 6,570 rows in an 11-day window, 99.16% of them heartbeats.
• Deploy verification is nearly free. A natural suspicion is that per-ticket deploys drive the dispatch time; measured against the store’s event log (30 days, 256 done tickets, fixture-validated), the deploy phase is a median 3.75 minutes, occurs on only 56 of 256 tickets, and is 0.3% of summed wall-clock. The implement dispatch itself (tests, two reviews, fix cycles) is the ~29-minute phase. Meanwhile median first-touch-to-done is 88 minutes: most of a ticket’s life is spent between phases (serial dispatch, gaps before the verification gate, recovery waits), which is where any future throughput work belongs.
• The bottleneck is not execution. Of 368 todo tickets at measurement time, only 5 passed every gate. The constraint is the human approval gate and model quota, not ticket supply and not host placement (the internal feasibility study).

That last number is the honest headline: the drain can execute more than its operator can responsibly approve. The gates are doing exactly what they were built to do, and the system’s throughput limit is, by design, a person.

8. What we believed vs what the code does

This paper was written under a rule: every mechanism claim gets verified against the code before it gets printed. That fact-check found eight places where our own documentation, memory notes, or prior discoveries had drifted behind the code. Recorded here both as corrections and as evidence for the rule.

1. We believed unresolvable blocked_by refs failed OPEN (a memory note says so). The code fails CLOSED: a blocker with no store row and no file blocks the ticket.
2. The 24x7 feasibility discovery’s quota arithmetic assumed dispatches inherit an Opus-class model; dispatch has pinned --model (default sonnet) since 08-19, so that arithmetic no longer describes the dispatch path.
3. Telemetry’s model label defaults to an Opus-class string while the dispatched model defaults to sonnet; with the env var unset, telemetry mislabels every dispatch. A real bug, found by this audit, now ticketed.
4. The fixer-agent skill description still says $5 per attempt; the default is $10 (half the $20 implement cap), and a stale worst-case comment underquotes the real $50 ceiling.
5. The project’s instructions file still describes hygiene stamps living in frontmatter; since the store-only cutover the sweep stamps the store and the gate reads it there, so frontmatter copies will disagree.
6. The feasibility discovery’s 2700s ceiling was already stale by publication + one week: 3600s since 08-19.
7. The 08-23 triage’s “classifier only knows 8 environment-drift categories” was fixed by its own suggested ticket: the taxonomy expanded to 18 categories. The newly added categories did not authorize retries.
8. Minor: a legacy sort key implies execution-ready affects ordering; on the live path ordering is purely the store query, and the key is a no-op.

The pattern in the drift is consistent: the code gets fixed at incident speed, the prose gets fixed at documentation speed, and the gap between the two is measured in weeks. That is precisely why the fail-closed gates read the code-adjacent store rather than prose, and why this series verifies against code, not against our own notes.

9. Open problems

• Escalation labels are still too coarse in practice: 93.5% of non-done outcomes in the 08-23 window wore the unknown label; the deterministic envelope-first labeling shipped since, but the window’s data shows how blind a busy pipeline can be about its own failures.
• The promote gate is the throughput ceiling (5 of 368 tickets dispatchable). Whether to widen it is a judgment about trust, not a technical gap; so far the answer is no.
• Quota economics: steady-state ~6h/day of drain fits the measured ticket-generation rate; 24/7 remains uneconomical on current plan quotas even with the cheaper pinned dispatch model. Re-measure before believing either number again (see section 8, items 2 and 6).

The earlier question about zero fix cycles has since been resolved. The August 30 follow-up found that fix_cycle_count measures repairs in the deployment stage. The counter for /implement’s review-and-fix loop was added after the August 23 sample. Those zeros cannot tell us how often reviewers requested fixes. The follow-up paper explains the measurement.

10. References

• The companion fact sheet (internal): every claim above with its full citation, compiled by reading the orchestrator source end to end before this paper was written.
• Internal measurement artifacts: the 24/7 feasibility study (2026-08-16) and the dispatch-outcome triage (2026-08-23), both quoted with their sample sizes in section 7.
• Files as Prose, Store as Truth: the substrate the drain runs on, and paper 1 of this series.
• The machine itself: the orchestrator, ready-queue, classifiers, and implement skill, private, at HEAD 2026-08-26.

This is a paper in the Yianna & Claude Whitepapers series: working papers on running a production side project at the output of a team, with a fleet of AI agents, alongside a full-time engineering job, written from the system’s own measurements and incident reports. The supporting evidence is retained in our internal decision records and postmortems; where a claim could not be traced to a measurement, the text says so.

Revision history

2026-09-13 · Editorial

Clarified the original collaboration with Claude and ongoing editing with Astra. Yianna retains final editorial and publishing decisions.

2026-09-13 · Editorial, Correction, System update

Reworked the introduction and explanations in the author’s voice; added optional ticket, verdict, and source examples, clearer link and code styling, and visible implementation notes. Corrected the reviewer-model rationale, classifier category accounting, and interpretation of the fix-cycle counter. Updated the classifier’s retry authority and introduced this dated revision history. Historical measurements keep their original dates; this revision checks the affected claims and does not renew the whole-paper audit.

• Correction · Important: The earlier account presented reviewer model choices and refusal behavior as established facts. The assignments are configured by role and execution mode, but their comparative benefit has not been established. The August 30, 2026 experiment observed no Fable refusals across 12 auth-related diffs supplied as text; it did not test the full tool-using reviewer workflow or establish that either model never refuses. Affected claim

• Correction · Routine: The original account counted two retry categories plus eighteen others. The August 26, 2026 implementation had eighteen categories in total, including the two eligible for retries. Section 8 also incorrectly described the whole taxonomy as non-retryable; that restriction applied to the newly added categories. Both descriptions are corrected.

• System update · Important: The failure classifier now supplies a diagnosis for human review and cannot authorize a retry. September 8–9, 2026 changes removed the earlier path in which its classification could select one of two allowed recovery actions. Affected claim

• Correction · Important: The earlier account treated 55 zero values as possible evidence of infrequent review fixes or a broken counter. The August 30, 2026 follow-up established that the field measures repairs during deployment. The implementation review-and-fix counter was added after the earlier sample, so those zeros cannot establish reviewer activity. This resolves the question previously marked unresolved. Affected claim