Published 2026-09-01 · Revised 2026-09-13 · Revision history
Technical verification: 2026-08-30; scope and evidence are described in the article.
By Yianna Kokalas & Claude
How this paper was written. This is a working paper from a production side project: software built for the Magic: The Gathering community, with real users, built and operated by one engineer (an enterprise engineer by day) together with Claude, Anthropic’s AI model. Every mechanism claim was fact-checked against the running code before publication. Checking the code and skill definitions revealed nine places where our documentation had drifted, and two claims we had repeated were challenged by measurement before this went out. The consequential findings appear beside the claims they qualify, with a visible record of discrepancies after the argument.
Position: the working unit of serious agentic engineering is not one strong agent with a long leash. It is a supervisor delegating to role-scoped subagents, each with its own context window, its own permissions, and a model chosen for the risk of its specific job. The supervisor holds the plan, evaluates the subagents’ results, and decides what happens next; the subagents do the work. This is the same trust arc the previous paper in this series walked (The Drain): autonomy is extended only as far as verification allows, and role boundaries are what make verification tractable. A reviewer with a fresh context does not inherit the implementer’s conversation, though it can still make the same mistake. A fix agent that may not merge must return its work for review.
Fresh contexts do not prevent shared mistakes, and role instructions are not necessarily access controls. The implementation guide permits an in-review transition on a documented push-failure path. The supervisor handles merge eligibility; the drain marks the ticket done after deployment verification. The prose and diagram now distinguish these responsibilities.
View revision →Corrected · 2026-09-13
Fresh contexts do not prevent shared mistakes, and role instructions are not necessarily access controls. The implementation guide permits an in-review transition on a documented push-failure path. The supervisor handles merge eligibility; the drain marks the ticket done after deployment verification. The prose and diagram now distinguish these responsibilities.
View revision →To assess the system, we checked its code and skill definitions, reviewed the records behind its model choices, and examined two separate empirical checks: a reviewer-log analysis and a security-refusal test. These activities address different parts of the same question:
Do role-scoped subagents on deliberately different models beat one strong agent, and is each of our model assignments still evidence-justified, or a habit nobody re-measured?
The honest answer, argued in section 6: the role-scoping is incident-justified and we would rebuild it; the model assignments mostly are not. The source review found many assignments without comparative evidence. The measurement-backed choices it identified were local classifiers; other choices rested on decisions, incidents, or precedent. Section 5 gives the reported inventory and its counting limitations. A failed swarm also challenged one delegation pattern, and the skill records that incident in its own body. The public account focuses on the assignments that explain the architecture and its limits. The paper’s response to the unevidenced rows is not backfilled justification: section 6.1 uses the security-refusal test to show how a model-choice rationale can be tested, and what its result can support. Two empirical checks were completed before publication. One killed a number from the previous paper in this series. The other left the repeated security-refusal rationale unsupported: 0 refusals in 12 selected diffs, for thirteen dollars. That test measured engagement, not review quality or the behavior of the full tool-using reviewer.
The model assignments, relative prices, ticket status, and uses of “now” or “current” below describe the system as documented on August 30, 2026, unless another date is given. They are not a current model catalog or a new verification of the running system. Fable, Opus, Sonnet, and Haiku are the model names used in those records; descriptions such as “strongest” state the assignment rationale, not a measured ranking on these tasks.
1. The shape of the unit
The system is a planning vault: a git repository of Markdown ticket specs, plus skills (versioned prompt programs a session can invoke) and a headless dispatcher called the drain that turns approved tickets into merged pull requests with no human in the loop. A “supervisor” here is the one session that owns a run: it reads the ticket, spawns every worker agent itself, scores their results, and decides whether the run can advance. A “subagent” is a worker spawned for exactly one role, with a fresh context containing only what the role needs.
Three properties define the unit, and none of them is stylistic:
- The supervisor spawns everything. The harness blocks subagents from spawning subagents, so delegation is one level deep by construction. . This is a constraint we chose to lean into rather than fight: a flat topology means there is exactly one place where subagent results are evaluated and exactly one agent whose claims must be re-verified against ground truth.
- Roles have explicit boundaries. Read-only investigator tools can restrict writes;
other boundaries are instructions in the skill, not separate access controls. The
implementation agent writes code and leaves merging to the supervisor. Its guide
permits an
in-reviewtransition on a documented push-failure path; the drain owns the finaldonetransition after deployment verification. The fix agent applies review findings. When an agent needs to work without disturbing the supervisor’s checkout, it uses a separate Git worktree. . - Models are assigned per role, per risk. The strongest models go where judgment is scarcest (adversarial review), cheaper models go where the work is bounded (mechanical synthesis, batch classification), and one entire task family runs on local open-weight models where the economics demand it. Whether each assignment is still justified is the evidence question in section 5.
The trust-ladder frame carries over from the previous paper: none of this delegation was trusted on day one. Skills ran interactively with a human watching before any supervisor was allowed to run them over subagents, and subagent output is treated the way the drain treats dispatch output: as a claim, verified against ground truth (git log, GitHub PR state, the test suite) before anything advances. The specific rule “confirm via git, not via the agent’s say-so” was earned: named background subagents were observed going idle and never returning result blocks, so the flagship pipeline spawns its agents unnamed and synchronous and verifies their work products directly. .
2. Planning and investigation skills
The planning workflow starts with optional scoping, then required exploration, ticket creation, and review. Debugging is a separate investigation path. The tags describe changes to application code or ticket specs; read-only investigators may still produce reports and other planning artifacts.
- /prompting — read-only; optional for larger scopes. Scopes an idea into
investigation prompts for
/plan-explore. . - /plan-explore — read-only; required. Investigators gather evidence; an Analyst and Critic challenge the discovery draft before the lead decides. .
- /plan-create — write: ticket specs. Creators draft tickets, then separate reviewers and fixers check them. . Structural lint runs only when its executable is present. . Human approval is required before execution. .
- /plan-review-swarm — read-only. Five specialist perspectives supplement the sequential review, each limited to three findings. .
- /debug-swarm — read-only. Investigates unbounded bugs through independent evidence gathering; unsupported conclusions remain hypotheses. . Bounded bugs stay inline: a July 20 swarm returned no results from four investigators. This incident motivated the default; it was not a controlled comparison. .
The historical planning skill ran structural lint only when the executable was present. The earlier account omitted that exception.
View revision →Corrected · 2026-09-13
The historical planning skill ran structural lint only when the executable was present. The earlier account omitted that exception.
View revision →3. The flagship: the /implement pipeline
/implement executes one ticket end to end: implementation, two independent reviews, a
bounded fix loop, then merge eligibility. It is the pipeline the drain dispatches
headlessly, which means every role boundary in it must hold with nobody watching.
The diagram shows the August 30 configuration. On September 13, 2026, we changed the implementation-agent assignment from inheritance to an explicit Sonnet pin.
supervisor (thin dispatcher)
reads ticket, spawns all agents, scores verdicts,
checks merge eligibility
|
+------------------+
|
v
IMPLEMENTATION AGENT
fresh context, builds its own
git worktree, implements the
ticket, runs the tests
model: inherits the dispatcher
may NOT merge or mark done
|
| work product verified via git, not via self-report
v
+---------------------------+---------------------------+
| |
v v
CRITICAL REVIEWER SECURITY REVIEWER
correctness, design, tests auth, injection, secrets
model: Fable interactively, model: Opus in ALL modes,
Opus under headless drain deliberately never Fable
(cost); on refusal or death, (recorded refusal rationale
re-spawned on Opus; a missing unsupported by the 12-diff
review is NEVER scored PASS test; see section 6.1)
| |
+---------------------------+---------------------------+
|
both PASS? | findings?
v
FIX AGENT (model: sonnet)
applies review findings only;
merge-state gate runs first so a fix
never lands on an already-merged PR
|
v
BOTH REVIEWERS RE-RUN, FRESH CONTEXTS
(max 2 fix cycles, then verdicts decide)
|
v
merge eligibility: human opt-in flag AND
validation PASS AND tree-integrity PASS
(missing = fail closed) AND both reviews PASS
|
v
gh pr merge --auto --squash
(the pipeline never writes "done";
the drain does, after deploy verification)
The texture is in the gates, and each gate has a scar:
- Fresh re-reviews. After a fix cycle, both reviewers re-run with fresh contexts rather than being asked “is it fixed now?” A reviewer that carries the prior conversation has already committed to a framing; a fresh one has not. .
- The merge-state gate. Before every fix cycle the supervisor checks actual PR state, because in June 2026 a fix was committed to a branch whose PR had already merged. Findings against a merged PR become a follow-up ticket, never a commit. .
- PASS-with-warnings is PASS. An operator ruling dated 2026-08-19, made after a green PR sat unmerged for a day over advisory nits. Only an explicit FAIL blocks. .
- A missing review is never a pass. If the critical reviewer’s model refuses or dies, the same prompt re-runs on Opus; the absence of a verdict is treated as the most suspicious outcome, not the cleanest. .
- The self-collision ban. Once preflight passes, re-deriving “is another implement
running” from process listings is forbidden: on 2026-08-18 three headless dispatches
found their own process with
ps, concluded a collision, and self-aborted with zero work done. .
The two reviewers cover different questions: correctness, design, and tests in one seat; security in the other. That division does not establish that two reviewers beat one. The recorded reason for keeping Fable out of the security seat was a claimed false-refusal risk on authentication and credential code. . No dated refusal artifact supported it. On August 30, Fable reviewed all 12 selected authentication-related diffs without refusing. The test supplied diffs as text and disabled tools; it did not exercise the complete reviewer workflow or validate its findings. Cost remained a reason to retain Opus, but the test did not establish the better reviewer model. Section 6.1 gives the scope and result.
4. Models chosen per risk: how assignment actually works
Model assignment in this system is distributed across many choices, made in skill definitions, agent frontmatter, and dispatcher code. The mechanisms worth naming:
- The dispatch pin. Every headless dispatch passes an explicit
--model(default sonnet). This exists because on 2026-08-19 a bare dispatch inherited the operator’s interactive model selection and silently ran the drain on the most expensive model available, at roughly twice Opus cost, until the round errored out. The planning pipeline pins its phase dispatches identically. . The pin is evidence-backed. The choice of sonnet specifically is precedent (“every prior healthy dispatch had run on sonnet”), not measurement.
- The implementation-agent pin. On August 30, the agent inherited the supervisor’s model. On September 13, 2026, we explicitly pinned it to Sonnet in interactive and headless runs. This removes a configuration dependency; comparative implementation quality remains unmeasured. .
- The mode gate. Fable was expensive, so we configured drain runs to use Opus for critical review while we tried it as a lower-cost alternative. Interactive reviews kept Fable. We had not yet measured the difference in review quality. .
- The security exception. The strongest model is excluded from the security seat in all modes, as described in section 3.
- The local family. The backlog-hygiene classifiers (freshness, dedup, dispatch triage) run on local open-weight models via Ollama, pinned in a code registry with the benchmark table reproduced beside the pin. . This is the only corner of the system where model choice was measured properly: a 2026-06-01 benchmark of six models on 20 hand-labeled tickets, and a 2026-08-21 re-benchmark on a 70-entry hand-verified corpus. It is also where measurement delivered the most consequential result: on August 21, the deployed freshness classifier missed all 40 non-fresh tickets in the 70-ticket corpus. Valid output was not enough: a classifier that labels everything fresh cannot perform the intended filtering. Check the failures the classifier must detect, not just whether its output is valid.
Verify model attribution before comparing results. In the August 30 source,
dispatch defaulted to Sonnet while telemetry defaulted to an Opus label when
DRAIN_DISPATCH_MODEL was unset. Grouping cost or outcomes by that label would
misattribute runs. .
The implementation agent’s inheritance was established separately by its spawn
instruction; the telemetry mismatch alone does not prove which model ran.
Assignments can also conflict before execution: the user-scenario agent declared Opus while its exploration caller requested Sonnet. . The source establishes the conflict, not runtime precedence. The inventory below reports the caller’s Sonnet assignment for exploration; it does not resolve that agent’s executed model.
5. The model-assignment inventory
The roles below explain the central tradeoff: delegation creates reviewable boundaries, but a model assignment needs evidence of its own. These are selected roles from the August 30 model-assignment inventory, not the complete operational inventory. The September 13 implementation-agent pin is described in section 4.
| Role | Model at the time of review | Evidence for the choice |
|---|---|---|
| Implementation | Inherits the dispatcher; Sonnet under the drain | The dispatch pin followed a cost incident. Inheritance and the choice of Sonnet were not benchmarked. |
| Critical review | Fable interactively; Opus under the drain | Opus was a lower-cost trial. Review quality had not been compared. |
| Security review | Opus | The recorded Fable-refusal rationale was unsupported by the 12-diff test; finding quality remained untested. |
| Fixes | Sonnet | Bounded work was the rationale; no comparative measurement. |
| Ticket creation and review | Opus | No recorded comparative evidence. |
| Exploration and plan-review investigators | Sonnet | Recorded reasoning, without comparative measurement. Exploration also used a separate Analyst and Critic. |
| Local backlog classifiers | Gemma and Qwen variants | Benchmarked on labeled data. Measurement exposed a major failure in the deployed freshness classifier. |
The August 30 internal inventory reported 33 assignment slots: 18 without recorded evidence and four backed by measurements, all involving local models. We cannot reproduce those exact totals from its grouped rows without making assumptions about how assignments were separated and evidence categories were counted. They remain a historical tally, not verified counts. The supported observation is narrower: several frontier-model assignments lacked comparative tests, while the local classifiers had been benchmarked. A recorded rationale or incident can explain a choice without establishing that it is the best model for the role.
The earlier article presented 33 assignments, 18 without evidence, and four measured choices as settled counts. Those historical totals cannot be reproduced from the grouped inventory without unstated counting and classification assumptions. They are now attributed and qualified rather than presented as verified totals.
View revision →Corrected · 2026-09-13
The earlier article presented 33 assignments, 18 without evidence, and four measured choices as settled counts. Those historical totals cannot be reproduced from the grouped inventory without unstated counting and classification assumptions. They are now attributed and qualified rather than presented as verified totals.
View revision →The pattern is worth stating plainly: the cheaper the model, the better its evidence. The local classifiers were benchmarked twice on hand-labeled corpora because they were suspected of being weak. The frontier-model assignments were never benchmarked because they were assumed to be strong. Suspicion produced measurement; confidence produced habit. One caveat on the word: “habit” in the evidence review describes the state of the record, not necessarily the state of the reasoning. Section 6 returns to the distinction.
6. What the evidence supports
Use role boundaries to make consequential steps inspectable. Fresh-context re-reviews, checking PR state before a fix, one writer per file, read-only investigators, and never counting a missing review as PASS address observed failures. They give the supervisor specific things to verify before work advances. That is the supported reason to adopt these safeguards. We have not compared the same tickets with a single strong agent, so neither better output nor a net benefit from parallel delegation has been established.
The failed swarm and separate inline investigation are incident evidence, not a controlled comparison. They support specific safeguards but do not establish that delegation produces better output than one strong agent.
View revision →Corrected · 2026-09-13
The failed swarm and separate inline investigation are incident evidence, not a controlled comparison. They support specific safeguards but do not establish that delegation produces better output than one strong agent.
View revision →Preserve what happened before the final PASS. Our August 30, 2026 analysis found a completed fix cycle in 26.8% of 347 dispatches. A final PASS can conceal an earlier failure and repair. . For a similar system, record initial findings, attempted repairs, and re-review outcomes separately. Then validate the findings before using those records to claim reviewer quality: warnings can trigger fixes, failed repairs can go uncounted, and this study did not measure false positives.
The reviewer study measured completed fix activity, not confirmed bugs or comparative reviewer quality. The revision corrects the verdict and cost-sample descriptions and removes causal and value claims the study did not establish.
View revision →Corrected · 2026-09-13
The reviewer study measured completed fix activity, not confirmed bugs or comparative reviewer quality. The revision corrects the verdict and cost-sample descriptions and removes causal and value claims the study did not establish.
View revision →6.1 Test the reason for a model choice
A model assignment becomes testable when its rationale predicts something observable. Our security-reviewer policy excluded Fable partly because of a claimed refusal risk on authentication code. On August 30, we tested that claim on 12 selected authentication-related diffs from two repositories: Fable engaged with all 12 without refusing.
That result left the refusal rationale unsupported in the tested setting. It did not select the better security reviewer: the test supplied inline diffs without tools, covered one day, and did not validate the findings. The practical lesson is to match the decision to the measurement. A refusal test can challenge an exclusion rule; choosing between reviewers requires comparing independently validated findings on the same inputs, alongside cost and consequential misses. That comparison remains open.
The 12-diff refusal test measured engagement with inline text on one day. It did not test July behavior, validate the findings, compare reviewer quality, or reproduce the full tool-using workflow.
View revision →Corrected · 2026-09-13
The 12-diff refusal test measured engagement with inline text on one day. It did not test July behavior, validate the findings, compare reviewer quality, or reproduce the full tool-using workflow.
View revision →7. Open problems
- Comparative reviewer quality remains unmeasured. The August 30 log analysis corrected the previous paper’s zero-fix claim, which used the wrong counter, and established repair activity. It did not establish which reviewer finds more consequential bugs. Initial findings, repair outcomes, and independent validation are prerequisites for that comparison.
- A rationale can outlive its evidence by two months and be repeated in three places without anyone checking. The false-refusal claim was recorded on 2026-07-01, copied into two skill definitions and a memory note, and cited as settled in the fact-gather for this paper. Thirteen dollars left it unsupported in the tested setting. The open problem is not that one claim was wrong; it is that nothing in the system flags a rationale that has never been tested. Recording a testable rationale is a manual safeguard, not a guarantee that it will be checked.
- Something moved the fix-cycle rate five-fold at the end of July, and we do not know what. 14.9% of dispatches ran a fix cycle before 2026-08-01; 77.3% after, with the 2-cycle cap now the modal outcome. Reviewer models were constant across the boundary. The step localizes to roughly 07-29 through 08-01, a window holding both a pipeline hardening batch (including the stalled-reviewer enforcement gate) and the start of a harder feature batch, and the four dispatches that could separate those explanations cannot carry an attribution. Until this is understood, no fix-cycle trend is interpretable as reviewer behavior alone.
- The freshness classifier’s future is an open decision: move the task to a frontier model, retire automated stamping, or accept the no-op as a cheap TTL refresher. The 2026-08-21 decision records the question; nothing has answered it.
- Fan-out ergonomics remain the weakest harness surface. The 0-of-4 idle failure, the stall alarms, the unnamed-synchronous workaround: all compensate for background agents that fail quiet. The system’s topology choices are partly downstream of that reliability gap, and would deserve another check against harness behavior if it closes. This paper’s own fact-gathering and refusal-corpus agents both went idle after finishing their work and before writing it up; the supervisor re-derived the results from their artifacts. The pattern the paper describes happened to the paper.
Discrepancies found while checking the paper
These findings from the August 30, 2026 source review are a public record to revisit, not a history of revisions to this article. Follow-up actions below are reminders; the current resolution of historical tickets has not been reverified. The September 13 implementation-model pin is a separately verified change. Filing a ticket alone does not establish that a discrepancy was resolved.
- Dispatch and telemetry used different model defaults. With the environment variable unset, dispatch selected Sonnet while telemetry labeled it Opus, invalidating model-based attribution of costs or outcomes. Align the defaults and check affected reports. .
- The two-skill Fable reservation was stale. A comment and memory note limited Fable to two skills, while assignments also included implementation review, exploration critique, copy review, and support. The historical review recorded a memory correction; the remaining descriptions need checking against the actual assignments. .
- The scenario agent and its caller requested different models. The agent declared Opus; the exploration caller requested Sonnet. This proves a source conflict, not which model executed. Resolve the intended assignment and verify runtime selection. .
- The agent index listed two missing agents. Both belonged to retired integrations. Remove or mark obsolete entries so the index does not direct work to unavailable agents; the historical review filed a documentation follow-up.
- Exploration instructions referenced retired sources. They requested a deprecated wiki and told synthesis to consult a tracker another step said to skip. Remove those paths or replace them with supported sources. .
- The documentation skill’s model requirement was only prose. It named “Opus 4.6,” identified as stale in the historical review, without enforcing the selection. Choose an explicit assignment or describe it as guidance rather than a runtime guarantee. .
- Freshness-filter documentation overstated the measured behavior. The deployed classifier missed all 40 non-fresh tickets in the 70-ticket benchmark described in section 4. Correct the filter’s description and revisit whether to replace it, retire automated classification, or retain only time-based refresh.
- The debug-swarm description advertised a path its body restricted. The headline offered four investigators; after a run returned zero of four results, the instructions favored inline work for bounded bugs. Align the description with that default so callers choose the appropriate path. .
- The implementation agent inherited its supervisor’s model. The spawn omitted a model argument, making interactive selection depend on the parent. On September 13, 2026, we pinned it to Sonnet in both modes (section 4). That resolves the configuration dependency; comparative implementation quality remains unmeasured. .
8. References
- Original research notes, August 30, 2026 (internal): workflow descriptions, model assignments, discrepancies, and source citations gathered before drafting. Later checks and corrections are documented separately.
- Internal measurement notes: the reviewer catch-rate measurement (2026-08-30, 347 dispatches from forensic logs, fixture-validated) and the Fable security-refusal corpus test (2026-08-30, 12 diffs, every output verified from raw JSON), both quoted with their sample sizes and limits in section 6.
- Internal decision records: the two local-model benchmarks (2026-06-01 and 2026-08-21) behind the only measured rows in the inventory.
- The Drain: the dispatch layer under this paper, and the trust-ladder frame it borrows.
- Files as Prose, Store as Truth: the substrate both run on.
- The machine itself: the implement, plan-create, plan-explore, debug-swarm, and plan-review-swarm skills, the agent definitions, the dispatcher, the pipeline runner, and the model registry, private, at HEAD 2026-08-30.
This is a paper in the Yianna & Claude Whitepapers series: working papers on running a production side project at the output of a team, with a fleet of AI agents, alongside a full-time engineering job, written from the system’s own measurements and incident reports. Evidence citations reference our internal decision records and postmortems; where a claim could not be traced to a measurement, the text says so.
Revision history
2026-09-13 · Editorial
Clarified the original collaboration with Claude and ongoing editing with Astra. Yianna retains final editorial and publishing decisions.
2026-09-13 · Editorial, Correction, System update
Reorganized the workflow and evidence discussion, added inspectable historical excerpts, and kept all nine discrepancies visible for follow-up. Corrected overstatements about role boundaries, inventory counts, and what the studies establish. Recorded the explicit Sonnet implementation-agent pin and introduced publication history.
- Correction · Important: Fresh contexts do not prevent shared mistakes, and role instructions are not necessarily access controls. The implementation guide permits an in-review transition on a documented push-failure path. The supervisor handles merge eligibility; the drain marks the ticket done after deployment verification. The prose and diagram now distinguish these responsibilities. Affected claim
- Correction · Important: The historical planning skill ran structural lint only when the executable was present. The earlier account omitted that exception. Affected claim
- Correction · Routine: Clarified that the planning lead approves discoveries and spawns creators to write additional tickets, rather than writing every ticket itself.
- Correction · Important: The failed swarm and separate inline investigation are incident evidence, not a controlled comparison. They support specific safeguards but do not establish that delegation produces better output than one strong agent. Affected claim
- Correction · Important: The reviewer study measured completed fix activity, not confirmed bugs or comparative reviewer quality. The revision corrects the verdict and cost-sample descriptions and removes causal and value claims the study did not establish. Affected claim
- Correction · Important: The 12-diff refusal test measured engagement with inline text on one day. It did not test July behavior, validate the findings, compare reviewer quality, or reproduce the full tool-using workflow. Affected claim
- Correction · Important: The earlier article presented 33 assignments, 18 without evidence, and four measured choices as settled counts. Those historical totals cannot be reproduced from the grouped inventory without unstated counting and classification assumptions. They are now attributed and qualified rather than presented as verified totals. Affected claim
- Correction · Routine: Removed unsupported duration and seat-count details; qualified model attribution where declarations conflict; clarified that interactive tiering depends on the parent model and that Opus critical review was a lower-cost trial without measured quality differences. The internal research notes are now described as original research, not a complete current fact sheet.
- System update · Important: On September 13, 2026, the implementation agent was explicitly pinned to Sonnet in interactive and headless runs. Previously it inherited the supervisor’s model. This removes a configuration dependency; comparative implementation quality remains unmeasured. Affected claim