Execute a Run as a Harbor job and derive each Cell from the job's per-trial output tree¶

ADR 0017 (item 3, "Run-as-Harbor-job — wrap the runner") set the direction: let Harbor own the undifferentiated orchestration — parallel-across-containers, per-trial persistence, resumability — and keep only the touchstone-shaped work above it. This ADR makes that concrete: it pins how a finished Harbor job's output tree is turned back into touchstone Cells so the existing grading pass (build_graders → grade → combine_scores) runs unchanged. It is recorded as a design decision validated by a spike (src/touchstone/harbor_job.py, not wired into the runner).

Status¶

Proposed — spike only; NOT scheduled. Refines ADR 0017 item 3; depends on the ADR 0017 ATIF→Trace projection for the trajectory step; does not change the Case, Trace, Grader/Score, or Regression contracts. The reference code (src/touchstone/harbor_job.py) is an unwired spike: nothing in runner.py imports it (harbor_job.WIRED_INTO_RUNNER is False), the runner integration described below is not scheduled, and it must not be relied on in production.

Fencing note (2026-07-02). Per the project audit (remediation #21, finding A5), the chosen resolution for the ADR-vs-code gap is to fence this module, not to finish the runner integration. The two previously-live TODOs are now explicit, clearly-labeled spike gaps rather than silent placeholders: the multi-metric-reward "primary" pick stays None by design (open Q5), and duration_s is None — an honest "unknown" — instead of a fake 0.0 (open Q4). The module docstring and the WIRED_INTO_RUNNER = False flag state the spike status in code. Landing the full runner integration (which would resolve open Q4/Q5 for real) remains future work.

Implementation note (2026-06-30). The trajectory step is now wired: harbor_job's staging calls project_trajectory_into_cell, which projects the located ATIF trajectory through the shared atif projector into cell_dir/trace.jsonl. The projector now enforces the schema_version gate this ADR calls for (it validates the ATIF version and raises on an unsupported one); the staging path catches that and degrades to "no Trace" rather than aborting. The spike's original "stages the trajectory path and references the projection" stance (below) is superseded by this on the import question only — the rest of the spike (reward-as-metric, fail-closed reads, graders own scoring) is unchanged. The path remains a spike: it is still not wired into the runner.

Context¶

Harbor's job runner already does, uniformly across the agent fleet, the orchestration touchstone's runner.py re-implements by hand: it fans a (task × trial) matrix across 32–100 containers, runs each agent in its sandbox, runs the task's verifier, and persists a results hierarchy (jobs/<job>/<trial>/{config.json,result.json,agent/,verifier/}) that resumes and that a web viewer reads. Touchstone today drives that loop itself in _run_cell (prepare sandbox → run harness → grade → write result.json).

ADR 0017 decided touchstone should stand on that runner rather than duplicate it — but a Harbor job answers Harbor's question ("how good is this agent on this benchmark", a reward.json scalar), not touchstone's ("for my usecases, which model should I ship", a graded recommendation). The verifier reward is a programmatic pass/fail; touchstone's value is its graders owning the score (ADR 0017). So ingesting a Harbor job cannot mean "adopt its reward as the result" — it must mean "stage the job's raw materials into touchstone's per-Cell conventions and let touchstone's graders re-grade them."

Two seams in the job tree are external and untrusted (ADR's fail-closed invariant): the ATIF trajectory.json (Harbor ships Pydantic ATIF models with a schema_version) and the verifier artifacts (reward.json / reward.txt / reward-details.json / ctrf.json). They must parse defensively and fail closed on shapes touchstone does not understand.

Decision¶

Execute a Run as a Harbor job, then derive each Cell's artifacts from the job's per-trial output tree. Harbor owns parallelism, per-trial execution, and resume; touchstone keeps matrix expansion (models paired per harness), observation-capability negotiation, the grading pass, and cross-run comparison. The derivation writes the same per-Cell files _run_cell writes into the Cell dir, then touchstone computes score.json / result.json exactly as it does for a native run. Each Cell's result.json stays the source of truth, now derived from the job outputs.

Harbor output → touchstone Cell mapping¶

Harbor artifact (per trial)	touchstone Cell artifact	Role
`agent/trajectory.json` (ATIF)	`cell_dir/trace.jsonl`	Projected to the Trace via the ADR 0017 ATIF projection (Tool-Kind + permission enrichment)
ATIF last `source=='agent'` step `message`	`cell_dir/output.txt`	The agent's final output text the `output`/regex/llm-judge graders read
`verifier/{reward.json,reward.txt}`	staged verbatim → `cell_dir/harbor/` and surfaced as a metric	RAW MATERIAL only — a grader may read it; never the overall score (ADR 0017)
`verifier/{reward-details.json,ctrf.json,test-stdout.txt,test-stderr.txt}`	staged verbatim → `cell_dir/harbor/`	Optional inputs a `command`/`files` grader may be pointed at
ATIF `final_metrics` + per-step `metrics`; trial `result.json` durations	`cell_dir/metrics.json`	tokens (input_tokens/output_tokens/total, native key convention) → `tokens`; `total_cost_usd` → `cost_usd`; agent-exec duration → `duration_s`; reward echoed as a metric
trial `config.json` (task id, agent, model, trial idx) + task prompt	`cell_dir/spec.json`	`{case, harness, model, trial, prompt}` — the same file `_run_cell` writes

What touchstone still computes itself, unchanged, after staging:

cell_dir/score.json {overall_score, overall_passed, scores[]} ← build_graders(case) → g.grade(result) over the staged output.txt / trace.jsonl / metrics.json, then combine_scores(scores, case.expect.pass_threshold). Graders own scoring.
result.json via store.set_status(DONE, …): overall_score (round 4), overall_passed, scores, metrics, warnings, error=None.
Identity (id/case/harness/model/trial) comes from the Cell coords (the manifest), not from Harbor; skipped_reason stays touchstone-only (reachability/availability).

The verifier reward is a metric, not the score¶

This is the load-bearing line. Harbor's reward.json / reward.txt is staged verbatim as raw material and echoed into metrics.json (e.g. harbor_reward) so the report can show it, but it is never mapped to overall_score / overall_passed. Those are produced only by touchstone's graders + combine_scores (ADR 0017). A Case that wants the reward can add a command/files grader pointed at the staged cell_dir/harbor/reward.json; the reward then enters scoring through a grader the Case author chose, on the same footing as any other grader.

The trajectory becomes a Trace via the ADR 0017 projection¶

agent/trajectory.json is parsed against the ATIF schema_version (fail closed on unknown/older versions) and projected onto the Trace schema by the ADR 0017 ATIF→Trace step — step.source → event role, tool_calls[] → tool_call events with Tool-Kind normalization, observation.results[] → tool_result, reasoning_content → thought. The spike deliberately does not import that projector (it lives on the other track); it stages the trajectory path and the reward/metrics, and references the projection as the trajectory→trace.jsonl step.

Considered options¶

Adopt Harbor's reward.json as the Cell result. Discards the grader/judge/regression decision layer — the ADR 0017 wrapper failure mode. Rejected.
Keep touchstone's runner; bolt Harbor parallelism into it. Re-implements the very orchestration ADR 0017 chose to borrow, for no differentiation. Rejected.
Run as a Harbor job and re-grade the output tree (chosen). Harbor owns orchestration; the derivation stages its artifacts into the Cell-dir conventions so the existing grading pass runs unchanged. The score stays touchstone's; the reward is one optional input.

Consequences¶

The runner gains a Harbor-job derivation path (harbor_job.py) beside HarborExecutor. They are distinct: HarborExecutor borrows only Harbor's sandbox (the agent loop stays touchstone's); this path ingests a job Harbor already ran (its own agent + verifier) and re-grades it.
Staging reuses _run_cell's file conventions (spec.json, output.txt, metrics.json, trace.jsonl) so score.json / result.json are computed by code that does not know Harbor exists. No existing module, the runner, or the graders change to support this path.
The external seams (ATIF trajectory, verifier reward/ctrf) parse defensively and fail closed.
The [harbor] extra is unaffected — the importer reads a filesystem tree and needs no live Harbor.

Open questions¶

Auth / secret passthrough. How API keys and verifier secrets cross into the job at execution time (the derivation reads a finished tree, but launching the job is out of scope here). Likely reuses ADR 0015's by-name env_passthrough.
Trials mapping. How a Harbor trial index maps onto a touchstone Trial, and how a Harbor task id maps onto a touchstone Case (a synthetic harbor:<agent> harness label is one option). Without a matching Case there are no graders + no pass_threshold to apply — there is nothing to grade.
Where output.txt comes from when ATIF is absent. The canonical source is the last source=='agent' ATIF step; if an agent emits no final text step (or no trajectory), the derivation must decide a fallback (e.g. /logs/artifacts/) or leave output.txt empty.
Trial result.json schema. Harbor does not formally document the trial-level result.json field names (reward, env-setup vs agent-exec vs verification durations, token totals, error). Until a real tree is inspected, timing/metrics are derived from the documented ATIF final_metrics and per-step metrics.
Multi-metric rewards. reward.json may carry several named metrics; the staged file keeps them verbatim, but how multiple keys are surfaced as metrics (and which a grader reads) is undecided.