Skip to content

Stand on Harbor's substrate; touchstone's value is the decision layer

ADR 0015 borrowed Harbor at the sandbox layer and explicitly rejected adopting "Harbor's task/agent/verifier framework." A fuller audit shows that line was drawn one notch too low — and that drawing it correctly is also the answer to "is touchstone just a Harbor wrapper?"

Status

accepted — extends ADR 0015 (widens the borrow-line from the sandbox to sandbox + agent / trajectory + runner); revises the ACP-as-cross-agent-unifier ambition of ADR 0003 / 0010 (ATIF becomes the cross-agent observation substrate; ACP is demoted to one optional Interaction transport).

Context

Harbor is the Terminal-Bench team's framework "for evaluating and improving agents," not merely a sandbox library. Audited end to end, it already ships — uniformly across the very agents touchstone targets — three things touchstone currently re-implements:

  • ATIF (Agent Trajectory Interchange Format) — a normalized JSON trajectory carrying messages, tool calls (tool_call_id / function_name / arguments), token usage, cost_usd, logprobs, model_name, and ISO-8601 timing — auto-emitted by Claude Code, Codex, Gemini CLI, OpenHands, Mini-SWE-Agent, and Terminus-2.
  • Pre-integrated agents — that same fleet, already wired to run in Harbor's sandboxes.
  • A job runner — parallel across 32–100 containers, per-trial persistence, a results hierarchy, and a web viewer that compares across agent/model combinations.

Each maps onto code touchstone maintains itself: the per-agent trajectory adapters (claude-stream, and the ACP adapter that CONTEXT.md itself calls "the aspiration… partial and uneven across providers"), the harness fleet, and runner.py's ThreadPool / resume / store. The duplication is heaviest where touchstone is weakest — cross-agent trajectory capture — and the ACP unifier is trying to solve a problem (one normalized trajectory across many agents) that ATIF already solves.

Two things ATIF does not carry, and they are not incidental — they are touchstone's: a canonical Tool-Kind taxonomy (ATIF normalizes a call's shape, not its meaning) and permission/approval events (ATIF has no concept of mediated mid-run interaction).

This ADR therefore also answers the strategic risk: if touchstone borrows Harbor's sandbox, agents, trajectories, and runner, what is left that is touchstone — or is it "a Harbor wrapper and nothing else"? The answer below is the load-bearing part of the decision.

Decision

Widen the borrow-line to three layers; keep the seam at the fourth. Harbor becomes touchstone's substrate — sandboxes (ADR 0015, done), agents + trajectories (ATIF), and run execution — behind touchstone's own contracts. Touchstone's complexity budget is then spent entirely above that line, on the decision layer.

  1. Trace = a projection over ATIF, not a parallel capture. Where a Harbor agent runs, touchstone ingests its ATIF and maps it onto the Trace schema, enriching it with the two things ATIF lacks: Tool-Kind normalization and permission events. The Trace stays touchstone's external contract (graders read it); ATIF becomes one Substrate source feeding it. Stop finishing ACP as the cross-agent unifier — ATIF holds that role.

  2. Capability-split harnesses. For observation (Tracing-only / output-only) Cells, borrow Harbor's agents and consume their ATIF instead of maintaining claude-code / claude-stream / ACP. Keep the openai in-process adapter as the Interaction path — Harbor's agents are autonomous and ATIF cannot represent mediated permission/approval, so controlled mid-run interaction must be touchstone's own loop. Clean line: observation → Harbor; interaction → touchstone.

  3. Run-as-Harbor-job (wrap the runner). Let Harbor own the undifferentiated orchestration it already does — parallel-across-containers, per-trial persistence, resumability. Touchstone's runner keeps only what is touchstone-shaped: matrix expansion (models paired per harness), observation-capability negotiation, the grading pass, and cross-run comparison. Each cell's result.json stays the source of truth, now derived from the Harbor job's outputs.

  4. The seam stays at the decision layer. Everything below — where work runs, which agent emits the trajectory, how trials are scheduled — is a borrowed implementation detail behind a touchstone contract (Case, Trace, Grader/Score, Regression). Everything at and above the decision layer is touchstone's and is never delegated.

Why this is not "a Harbor wrapper" — where touchstone's value is irreducible

A wrapper exposes someone else's abstractions with a thin convenience layer. Touchstone inverts the dependency: Harbor is a swappable backend behind touchstone's contracts, and the test of that is simple — swap Harbor out and touchstone still means something. It does, because the value lives in five places Harbor structurally does not go:

  1. A different question. Harbor answers "how good is this agent on this benchmark" (absolute, public, agent-improvement, an RL reward). Touchstone answers "for my recurring usecases, which model should I ship" (relative, personal, a decision). That reframes the whole stack: the axis that varies is the model with the harness held constant, and the output is a recommendation with confidence, not a scalar reward.

  2. Trustworthy judging of subjective work (the moat). The usecases that matter most — "is this rewrite better," "is this answer well-judged" — have no test.sh. touchstone's model_judge is audited: its agreement with gold labels (Cohen's κ / Krippendorff's α, ADR 0013) tells you whether to trust its verdicts. Harbor's verifier is a programmatic reward; it has no judge, and therefore no notion of judge credibility. Graded confidence in subjective evaluation is the part of "best for me" Harbor cannot reach.

  3. Regression as a statistical verdict over time (ADR 0011). A paired-bootstrap CI over shared cases, pass^k flips, pinned baselines — "did the new model actually get worse for my tasks, or is that noise?" Harbor's viewer compares runs; it does not gate on a sound regression test. touchstone is a standing model-watch over your own Case library as new models ship.

  4. Behavioral evaluation under controlled interaction (ADR 0001). A held-fixed Responder + permission policies let touchstone grade how a model behaves mid-run — does it ask the right clarifying question, respect a permission boundary — not just its final artifact. ATIF doesn't even record these events; this surface is invisible to Harbor.

  5. The Case as a portable personal eval. Task + sources + your agent artifacts (skills / MCP / commands) + weighted, gated graders + a pass threshold, bundled once and replayed against every future model. Harbor tasks are public-benchmark-shaped; Cases are your-workflow-shaped, with your artifacts as part of what's under test, and a scoring model richer than reward.json's mean/final.

In short: Harbor is an agent benchmarking-and-training substrate; touchstone is a personal model-selection decision system that stands on it. Borrowing more of Harbor is what frees touchstone to deepen the decision layer — it is the opposite of becoming a wrapper.

Considered options

  • Hold ADR 0015's line (sandbox only); finish ACP as the cross-agent unifier. Keeps re-deriving ATIF by hand across providers — the partial/uneven status quo — for no differentiation. Rejected.
  • Adopt Harbor as the framework (Cases become Harbor tasks, graders become reward.json). Discards judge credibility, regression CIs, interaction, and weighted grading — the entire decision layer. This is the wrapper failure mode, reached by surrendering rather than borrowing. Rejected.
  • Widen the borrow-line to agents / trajectories / runner; keep the decision-layer seam (chosen). Deletes touchstone's worst-maintained code, stands the Trace on a solved substrate, and concentrates every remaining line on the differentiators.

Consequences

  • The ACP adapter is demoted from "future unifier" to one optional Interaction transport; ATIF is the cross-agent observation substrate. CONTEXT.md's Adapter / Substrate entries and the README Harness bullet are updated to match; ADR 0003 / 0010 framing is superseded on this point.
  • Trace gains an ATIF ingest path; Tool-Kind normalization and permission-event synthesis become the Substrate's enrichment responsibility rather than per-adapter capture.
  • The runner gains a Harbor-job execution mode; each cell's result.json stays the source of truth, derived from the job outputs. The matrix, capability negotiation, grading, and comparison are unchanged.
  • The [harbor] extra grows from "remote sandboxes" to "sandboxes + agents + run execution"; the import stays optional and lazy — non-Harbor Cells (notably the in-process openai Interaction path) pull nothing.
  • README / landing positioning is sharpened to state the identity explicitly (a decision system on a borrowed substrate), so the differentiators are legible and the wrapper reading is foreclosed.

Future work — deepening the decision layer (new value)

These lean into the identity, and are things Harbor structurally will not do:

  • Pareto model selection. "Best for me" is multi-objective — quality × $ × latency. The efficiency grader + trace cost already hold the data; emit a frontier and a recommendation ("X is 95% of Y's quality at 30% of the cost on your cases").
  • A recommendation report, not a leaderboard. The deliverable is the decision: pick X for these usecases, here's the evidence and the confidence interval.
  • Continuous model-watch. New model drops → re-run the baseline → tell me whether to switch, with the regression CI as the gate.
  • Surface judge calibration as a product signal. "Your judge agrees with you κ=0.7 here — trust it; κ=0.3 there — go look." Meta-evaluation of your own eval, turned into a first-class output.