added 260907 version of worker toolkit
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: detector-fact-check-rubric-claims
|
||||
description: |
|
||||
Self-check every load-bearing factual claim in your
|
||||
holistic rubric against the source repo at the commit declared
|
||||
in `task.toml`. Catches stale citations, dead-code-as-load-bearing
|
||||
assertions, schema-constraint claims that don't hold, behaviors
|
||||
mis-attributed to a file or line, and rubric self-contradictions. Each
|
||||
claim is checked on two axes: is it TRUE against the workspace, and — for
|
||||
facts your rubric grades the response for knowing or finding — is it
|
||||
REACHABLE from what the test agent is given (the prompt, the snapshot
|
||||
session, and the workspace)? A true fact the agent has no way to learn is
|
||||
a fairness defect, not a knowledge test. The most common worker mistakes:
|
||||
citing files or lines from memory and letting the rubric drift from what
|
||||
the code actually shows, and gating the score on privileged context
|
||||
(provider behavior, policy thresholds) that lives only in your head.
|
||||
allowed-tools: Bash, Read, Write
|
||||
---
|
||||
|
||||
# Fact-check rubric claims
|
||||
|
||||
This skill checks every factual claim in your holistic rubric (the file
|
||||
`bash scripts/guidance-target.sh <slug>` resolves)
|
||||
that the rubric's score depends on, against the actual source repo at the
|
||||
commit your `task.toml` declares — and, for facts the rubric grades the
|
||||
response for knowing or finding, whether the test agent could actually
|
||||
reach them from the package it is given.
|
||||
|
||||
Read these before starting:
|
||||
|
||||
1. `.claude/skills/_detector-worker-shell.md` — where to write the report and how to handle re-runs. Detectors with structured payloads (this one's `claims` array) embed them in the same frontmatter block as `detector`/`verdict`/`confidence`.
|
||||
2. `.claude/skills/detector-fact-check-rubric-claims/core.md` — what counts as a load-bearing factual claim, the per-claim verdict enums, the top-level reduction, the frontmatter/body schema.
|
||||
|
||||
## How to do it
|
||||
|
||||
Work through the rubric one claim at a time:
|
||||
|
||||
1. **Read the rubric.** Open the guidance file `bash scripts/guidance-target.sh <slug>` resolves. Identify every load-bearing factual claim (file paths, line ranges, function names, schema constraints, runtime behaviors that the rubric says are load-bearing for some scoring criterion). Assign each claim an id (`c01`, `c02`, …). Also mark which claims **gate scoring on knowledge** — facts the response is graded for knowing or finding, as opposed to background that only justifies the rubric to the grader (see `core.md`, "The second axis").
|
||||
2. **Make sure the patched workspace exists.** Your rubric describes what the test agent sees, and the test agent sees `git archive <commit>` plus `environment/workspace.patch` applied — the workspace at `harbor-tasks/<slug>/environment/workspace/`. The directory is gitignored; if it's missing, run `scripts/build-workspace.sh <slug>` to rebuild it. Reading the bare `git show <commit>:<path>` instead would miss any files your snapshot session added/modified/deleted, producing false fails on every patched file.
|
||||
3. **For each claim, verify against the patched workspace.** Open `harbor-tasks/<slug>/environment/workspace/<path>` and compare what the rubric claims against what's actually there. For symbol-existence / call-site / dead-code checks, grep the workspace tree (`rg '<symbol>' harbor-tasks/<slug>/environment/workspace/`). Mark `pass` / `unclear` / `partial` / `fail` per `core.md`'s verdict definitions.
|
||||
4. **For each scoring-gate claim, run the reachability check.** Where your rubric grades the response for *knowing or finding* a fact (external-provider behavior, business context, a policy threshold, a canonical root cause), trace where in the package the test agent could learn it — the prompt, the snapshot session, or the patched workspace — per `core.md`'s "The second axis" ladder. Hard-to-find is reachable; nowhere-in-the-package means the claim's `note` leads with an `unreachable:` marker plus the searches you ran. A claim can be true and still unreachable — that's exactly the defect this step catches.
|
||||
5. **Compose the report.** Embed all per-claim records in the YAML frontmatter alongside `detector` / `verdict` / `confidence`. The body is a short provenance summary (workspace path, commit, count of claims checked); the substance is in the inline claims array. If any claim is unreachable, name those claims in the body.
|
||||
6. **Write per `_detector-worker-shell.md`** to `harbor-tasks/<slug>/detectors/detector-fact-check-rubric-claims.md`.
|
||||
|
||||
The top-level `verdict` reduces from the per-claim verdicts using the rules in `core.md` — `fail` if any load-bearing claim is `fail`; `not-applicable` if there are no claims or every claim is `unclear`; `partial` if any load-bearing claim is `partial` / `unclear`, or any claim at all is `fail`, or any claim's note leads with `unreachable:`; else `pass` (non-load-bearing `partial`/`unclear` drift doesn't change the color — the per-claim list still shows it).
|
||||
|
||||
## Acting on the verdict
|
||||
|
||||
- **`pass`** — every load-bearing claim survived verification and every scoring-gate fact is reachable (any remaining drift is non-load-bearing and listed per-claim). Good. Move on.
|
||||
- **`partial`** — a load-bearing claim has real drift or couldn't be verified, OR a non-load-bearing claim is outright false, OR a fact your rubric grades the response for knowing isn't reachable from the package. Look at the per-claim list in the report: fix any incorrect citations, even non-load-bearing ones, since they make the rubric harder to trust. For an `unreachable:` claim the fix is one of two moves: put the fact in the materials (state it in the prompt, plant a reachable signal in the repo), or stop gating the score on it (grade the overclaim — the agent asserting what the evidence can't support — instead of the hidden answer).
|
||||
- **`fail`** — at least one load-bearing claim doesn't survive verification at the declared commit. The fix is to either (a) rewrite the rubric so the load-bearing claim matches the source, or (b) change `task.toml`'s `commit` to one where the claim holds. Re-run this skill after.
|
||||
- **`not-applicable`** — the rubric is empty / template, or `task.toml` is missing repo/commit. Write the rubric first (and confirm the commit), then come back.
|
||||
|
||||
## Single-session approach
|
||||
|
||||
This worker version does the whole thing in one Claude session (read rubric → extract claims → verify each → write the markdown). Take time on each claim — read the source, quote the relevant lines, write a specific `note`. Skimming claims wholesale is the failure mode this skill exists to prevent in your own rubric.
|
||||
@@ -0,0 +1,201 @@
|
||||
# Fact-check-rubric-claims detector — core
|
||||
|
||||
This file is the canonical, context-neutral content for the
|
||||
detector-fact-check-rubric-claims detector. It defines what counts as a load-bearing
|
||||
factual claim, the per-claim verdict enums, how the top-level verdict
|
||||
reduces, the structured claims payload schema, and the output frontmatter
|
||||
shape. It's read in two contexts — the base repo's review pipeline (which
|
||||
fans the work out across multiple subagents) and the worker toolkit's
|
||||
self-check (which does it sequentially in one session) — so nothing here
|
||||
should mandate a specific orchestration shape; the wrapping `SKILL.md`
|
||||
tells you that.
|
||||
|
||||
## What this detector is for
|
||||
|
||||
The rubric (the resolved grader-guidance file — see Inputs) is hand-authored by the worker. Workers routinely cite specific file paths, line ranges, function names, schema constraints, and concrete behaviors as the load-bearing evidence for an issue's score. **Many of those citations don't survive verification at the commit declared in `task.toml`** — the cited line says something different, the function is dead code, the schema column has no FK, the behavior described is one the worker imagined and then over-fit into a rubric.
|
||||
|
||||
When the rubric is factually wrong, the entire score signal becomes unreliable: agents lose points for not naming an issue that isn't actually in the code, or get full credit for restating an inaccuracy. Fact-checking is mechanical (read source, compare strings/lines/types) but tedious and per-claim independent — each claim is verified against one source location, with no cross-claim contamination.
|
||||
|
||||
Truth is not the only way a claim breaks the score signal. Each load-bearing claim is checked on **two axes**: is it **true** against the workspace at the declared commit, and — when the rubric grades the response for *knowing or finding* the fact — is it **reachable** from the package the test agent is given (the prompt, the snapshot session, and the patched workspace)? The axes are independent. A claim can be **true and unreachable** — the provider's retention window may be exactly as the rubric says; the defect is that the agent was never given it, so the task grades guessing the author's private knowledge (a fairness problem). Or **false and reachable** — the workspace contradicts the rubric (a truth problem). A rubric is entitled to privileged context for the *grader's* benefit; it is not entitled to gate scoring on the agent asserting a fact that exists only in that privileged context. See "The second axis" below.
|
||||
|
||||
The reader looks at each per-claim verdict individually; the queue / report shows "N/M pass" so they can scan the column at a glance. **There is no opaque aggregate verdict that drives action** — the value is the per-claim list. The detector entry's top-level `verdict` field is just a derived color for the queue cell.
|
||||
|
||||
## Verdict enums
|
||||
|
||||
**Per-claim verdict** (`verdict` field — the part a reader actually acts on):
|
||||
|
||||
- `pass` — the rubric's assertion is clearly correct against the source.
|
||||
- `unclear` — genuinely ambiguous. Either (a) the source needed to check the claim isn't available (commit missing from every local clone, file lives outside the repo), or (b) the rubric's claim is itself too vague or malformed to evaluate ("the codebase has a complex transfer flow" with no specific assertion; a citation that doesn't pin down what's being asserted). Not a synonym for "tedious to check".
|
||||
- `partial` — directionally correct but has real issues (line numbers off by a few, citation names the right file but adjacent function, schema type generalization that still preserves the rubric's point). **Bar for `partial` is real material drift, not pedantry**: if the rubric says `PUT` and the source says `PATCH` but the verb doesn't change what the rubric is asserting, that's `pass`, not `partial`. Reserve `partial` for drift a careful reader would care about.
|
||||
- `fail` — totally false. The cited file doesn't exist in the patched workspace, the line says something different, the function is dead code, the schema constraint isn't there.
|
||||
|
||||
**Per-claim impact** (`loadBearing` field — pairs with the verdict):
|
||||
|
||||
- `loadBearing: true` — the truth or falsity of this claim matters for the broader point the grader guidance is making. Example: rubric says "the worker agent is supposed to identify that the XYZ subsystem has an ABC endpoint" and that endpoint doesn't exist — the grader's whole assertion is ruined.
|
||||
- `loadBearing: false` — the claim may be false, but its falseness doesn't undermine the validity of what the grader guidance is asserting on. Example: rubric says an endpoint is `PUT` when it's actually `PATCH`, but the verb doesn't change anything about how the worker agent is being evaluated.
|
||||
|
||||
A `fail` on a `loadBearing: true` claim is the loud signal this detector exists to surface. A `fail` on a `loadBearing: false` claim is rubric-craft drift worth flagging but not blocking.
|
||||
|
||||
**Per-claim reachability** (encoded in the `note` field — no separate enum): for claims that gate scoring on knowledge (marked `gatesScoring: true` at extraction; see "The second axis" below), the checker also records where in the package the agent could learn the fact. When the honest answer is "nowhere," the note **leads with an `unreachable:` marker** followed by the searches that establish absence; when the fact is reachable, the note records the discovery path (the disclosing prompt/session line, the workspace path, or the common-knowledge call). Hard-to-find is reachable — the marker is strictly for nowhere-in-the-package. Reachability never changes the per-claim truth verdict: a true-but-unreachable claim stays `pass` on the truth axis; the fairness defect lives in the marker and drives the top-level reduction.
|
||||
|
||||
The `note` and the verdict must agree. A `pass` whose note documents that the source contradicts the rubric on a load-bearing point is malformed — if the evidence disagrees with the claim, so must the verdict.
|
||||
|
||||
The same rule holds *across* claims: if the evidence recorded for one claim refutes another claim's `pass` (one claim's source quote shows a surviving stub while a sibling claim passes an "only trace erased" assertion), the claim set is internally inconsistent. Reconcile the verdicts before the report ships — whichever production path is in use, someone reads the assembled array end-to-end before saving it.
|
||||
|
||||
**Top-level detector verdict** (queue cell color only — derived from the per-claim list):
|
||||
|
||||
- `not-applicable` — no claims to check (rubric empty / template), OR every claim is `unclear`.
|
||||
- `fail` — at least one load-bearing claim is `fail`.
|
||||
- `partial` — none of the above, AND at least one load-bearing claim has issues (`partial` or `unclear`), or any claim is `fail`, or any claim's note leads with the `unreachable:` marker. An unreachable scoring-gate claim caps the verdict at `partial` even when its truth verdict is `pass` — a fact the agent has no way to reach breaks the score signal just like a false one.
|
||||
- `pass` — every load-bearing claim is `pass`, no claim is `fail`, and no claim carries the `unreachable:` marker. Non-load-bearing `partial`/`unclear` claims don't change the color — the per-claim cards still show the drift.
|
||||
|
||||
The reduction is checked in order. The reduction is a simple computation over the claims array — there's no judgment call to make (the `unreachable:` scan is a mechanical check for the leading marker in each note). The reader doesn't act on the top-level verdict directly; they look at the per-claim list. The color tracks the graded-signal question — "is the rubric's substance sound?" — so surface drift on non-load-bearing claims stays visible in the cards without coloring the cell. This puts real weight on the `loadBearing` flag: a substantive claim mislabeled `loadBearing: false` now keeps a genuine defect out of the queue color, which is why the extractor defaults to load-bearing when in doubt and the checker's downgrade carve-out is limited to surface-citation drift.
|
||||
|
||||
## Inputs
|
||||
|
||||
- The grader guidance — the rubric. The primary input. Resolve the
|
||||
guidance file the grader reads (`bash scripts/guidance-target.sh <slug>`
|
||||
prints its path, `tests/grader-guidance-consolidated.md` — the worker shell's
|
||||
guidance-target resolution) and fact-check the file it names, never
|
||||
another document.
|
||||
- `harbor-tasks/<slug>/instruction.md` — the prompt the agent received.
|
||||
- `harbor-tasks/<slug>/task.toml` — to confirm `repo` and `commit` are set.
|
||||
|
||||
For per-claim verification, **the canonical source is the patched workspace at `harbor-tasks/<slug>/environment/workspace/`**, not `git show <commit>:<path>` against the baseline commit. The test agent sees `git archive <commit>` followed by `environment/workspace.patch` applied — when the patch adds, modifies, or deletes files, the workspace differs from the bare commit. The rubric describes the workspace state (what the test agent reads), so fact-checking must too. Reading the baseline alone produces false `fail` verdicts on every file the patch creates, and false `pass` verdicts on every file the patch modifies.
|
||||
|
||||
The workspace is gitignored. If `harbor-tasks/<slug>/environment/workspace/` is missing, build it with `bash scripts/build-workspace.sh <slug>` before checking claims (in a repo checkout, `harbor-tasks/raccoon-shared/build-workspace.sh <slug> <repo from task.toml> <commit from task.toml>`). The build is idempotent (it `rm -rf`s the workspace before re-exporting), takes seconds, and applies any `workspace.patch` it finds.
|
||||
|
||||
Read patterns:
|
||||
|
||||
- File content / line citation / schema / behavior claims: read the file under `harbor-tasks/<slug>/environment/workspace/<path>` directly (no `git -C` indirection — the workspace has no git history).
|
||||
- Symbol-existence / call-site / dead-code claims: grep the workspace tree (e.g., `rg '<symbol>' harbor-tasks/<slug>/environment/workspace/`).
|
||||
- Claims the rubric attributes to the prompt, ticket, or snapshot ("the user says they are available to answer questions", a quote attributed to the ticket): read `harbor-tasks/<slug>/instruction.md` and the snapshot session at `harbor-tasks/<slug>/environment/session.jsonl`. Those artifacts — not the workspace — are the source of truth for what the user or ticket said.
|
||||
- The reachability axis reads `instruction.md` and the snapshot session for *any* scoring-gate claim, not just `prompt`-type ones — a fact disclosed in an earlier session turn is reachable even when the claim itself is about external behavior. Never record an `unreachable:` marker without reading the session.
|
||||
- Genuine historical claims (rare — "this symbol was deleted in commit X") still need the source repo: the toolkit's `repo/` checkout, or `repos/<RepoName>/repo` in a repo checkout. Use `git -C <source repo> log -S '<symbol>' <commit>` for that subset only; most rubric claims are about the present state of the workspace.
|
||||
|
||||
If the workspace can't be built (no submodule checkout, no clone with the declared commit, build-workspace.sh fails) and the source repo is also unavailable, return `unclear` (sub-case: source unavailable) for any claim citing files in that repo.
|
||||
|
||||
## What counts as a load-bearing factual claim
|
||||
|
||||
A factual claim is **load-bearing** when the truth or falsity of the claim matters for the broader point the grader guidance is making. If the claim is wrong, the grader's score signal is wrong. Concrete shape:
|
||||
|
||||
- The rubric says "the worker agent is supposed to identify that the XYZ subsystem has an ABC endpoint." If `ABC` doesn't exist on `XYZ`, the grader's whole assertion is ruined — **load-bearing**.
|
||||
- The rubric says an endpoint is `PUT` when it's actually `PATCH`, but the verb doesn't change anything about how the worker agent is being evaluated — **not load-bearing**. The claim is false, but the grader's substance still stands.
|
||||
|
||||
**Surface citation drift is not load-bearing.** When the rubric quotes a code block, names a line range, or otherwise points at a piece of source, the *substance* of what it's pointing at is what's load-bearing — not the exact citation surface. If the substance matches the source and only the surface drifts (off-by-a-few line numbers; wrapper-syntax difference like `Foo.new(...)` vs `params.merge(...)` when the keyword args inside are identical and serve the same role), mark `loadBearing: false`. The verdict can still be `partial` for the surface drift; the not-load-bearing flag tells the reader "this is rubric-craft polish, not a graded-signal defect."
|
||||
|
||||
Other calibration:
|
||||
|
||||
- *"`foo.ts:42-58` returns `null` when X happens"* — load-bearing if the issue's heavy penalty or A+ tier is "agent identifies that `foo.ts:42-58` returns null when X." Not load-bearing if it's mentioned as ambient context for a different claim.
|
||||
- *"The codebase polls at 10–30s intervals"* — load-bearing if the rubric scores agents on understanding polling cadence; not load-bearing if mentioned as a tangential nice-to-know.
|
||||
- *"`Organization.requestEmails: String @default("")`"* — load-bearing if the rubric grades agents on identifying that this is a free-form string with no FK to Member records.
|
||||
|
||||
When in doubt about the **substantive point**, mark the claim load-bearing. False positives there are cheap; false negatives miss the defect this detector exists to catch. But surface-citation drift on an otherwise-correct claim is the named exception above — those go `loadBearing: false`.
|
||||
|
||||
## What counts as a factual claim (vs. a substantive judgment)
|
||||
|
||||
**Factual** — checkable by reading source. File path X exists / has function Y / has line numbers Z. Function Y returns null on X / gates on Z. Schema column C has type T / has FK / has default D. The codebase uses pattern P at runtime. Symbol S is referenced N times / is dead code. The prompt or ticket contains statement S (checkable against `instruction.md` / the snapshot session rather than the workspace).
|
||||
|
||||
**Substantive judgment** — checkable only by argument. Stays with the human reviewer. Whether a prescribed fix is "the only correct one" or just one defensible option. Whether 10 points is "the right weight" for an issue. Whether a heavy penalty is calibrated correctly. Whether a failure has "real-world impact" or is "process-only consequence."
|
||||
|
||||
The detector explicitly does NOT cover substantive judgments. Those stay with a human reviewer.
|
||||
|
||||
**Uncited claims count too.** A load-bearing ground-truth assertion that carries no `path:line` is still a factual claim — leaving it unchecked because there's nothing cited to open is how a false assertion ships behind a clean pass. The checker locates the evidence itself and verdicts on the substance; the missing citation goes in the `note` as rubric-craft feedback, not a verdict downgrade (a true-but-uncited claim is still `pass`).
|
||||
|
||||
## Verify the assertion, not the citation surface
|
||||
|
||||
The recurring miss shape is a claim whose citation checks out — the file exists, the line roughly says that — while the assertion the rubric builds on it is false. Confirming "the cited line contains the code" is necessary but never sufficient. Four claim shapes need verification beyond the cited lines:
|
||||
|
||||
- **Mechanism claims** ("X fires when Y", "the early return is why Z never runs"). Trace how the cited symbol is actually invoked or wired — grep the call sites, read the caller — before passing. A function can contain exactly the code the rubric quotes and still never fire for the reason the rubric gives; the real gate may live in the caller. The cited line existing is not evidence for a claim about *when or why* it executes. Ordering variants count too: a claim that reordering or changing a step would alter an outcome needs the execution order traced — if the values are computed and locked in before the cited step runs, changing that step can't affect them, however plausible the rubric's story reads.
|
||||
- **Universality claims** ("every", "all", "only", "always", "no way to", "exactly N"). Actively hunt counterexamples across the whole workspace; a single confirming example is not a pass. "Every review creates an audit record" fails if any write path bypasses the audited callback; "there are exactly four creation sites" fails if grep finds a fifth.
|
||||
- **Consequence-chain claims** ("users are spammed", "the data is destroyed with no way to get it back"). Follow the code path end-to-end from the cited defect to the claimed effect — every link. If a link doesn't hold in the workspace (the delivery adapter returns `false` before sending anything; the archive step retains recoverable data), a present-tense consequence claim is a `fail` on a load-bearing claim. Boundary: this is mechanical path-tracing, not impact judgment. Whether the effect that *does* occur matters enough is the human reviewer's substantive call; whether the asserted effect occurs at all is the fact-check question.
|
||||
- **Beyond-the-repo claims** ("every production organization has a record stuck in state X", how a third-party service behaves, what the current version of an external standard requires). The workspace establishes what the code does — not what production data contains, how an external service will respond, or what an external document says today. Verify the repo-side part, then check whether the assertion overreaches it: code showing the normal flow never calls `approve!` supports "the flow never approves," not "every organization has a stuck record." An overreach on a load-bearing claim is `partial` or `fail`, not `pass`. And outside research is not repo verification — a claim whose truth rests on out-of-repo facts can't `pass` on the strength of what you looked up; say what the repo does and doesn't establish. A load-bearing claim that reduces to `unclear` because its source lives outside the repo (provider behavior, standards text) is precisely the population the reachability axis must rule on — an unverifiable source is often also an unreachable one, so run the second-axis trace and record it in the note rather than letting a quiet `unclear` carry the fairness conclusion on its own.
|
||||
|
||||
Two cross-cutting rules apply to every shape. **Verify the predicate, not the nouns**: confirming that the cited symbols, definitions, or files exist is not verifying what the rubric asserts *about* them — that X is the correct or only home for a behavior, that a validation actually covers Y, that a relocated control is unreachable. Name the load-bearing predicate in the claim and check that specifically; what a schema *permits* is likewise not proof that a user-facing workflow actually reaches it. And **the rubric's gloss is itself a claim**: when the rubric characterizes what a symbol represents or how a feature is scoped (a "tenure" field framed as account age when it derives from the employment start date; a "month-to-date" window that's actually user-selectable), check the definition and usage instead of inheriting the framing.
|
||||
|
||||
## The second axis: could the agent reach the fact?
|
||||
|
||||
Truth is checked against the workspace; **reachability** is checked against the package the test agent is given — `instruction.md`, the snapshot session, and the patched workspace. The axis applies only to claims that **gate scoring on knowledge**: facts the response is graded for knowing, finding, or acting on — external-provider behavior, business context, product-policy thresholds, a canonical root cause the answer key requires. The extractor marks these `gatesScoring: true`. Claims that merely justify the rubric to the grader — why a failure matters, background a response never has to state to score well — are legitimately privileged; reachability doesn't apply to them.
|
||||
|
||||
For each scoring-gate claim, trace where the agent could learn the fact, in this order:
|
||||
|
||||
1. **Disclosed** — stated in `instruction.md` or the snapshot session. Reachable; quote the disclosing line.
|
||||
2. **Derivable** — present in the patched workspace: grep the symbols, read the cited files, comments, docs, migrations, configuration — as the agent would see them. **Hard-to-find is reachable**: a fact buried in an unglamorous file, discoverable only by tracing a call graph or reading a migration, is fair game — difficulty of discovery is headroom, not unfairness. Record the path.
|
||||
3. **Common knowledge** — stable, uncontested, not version-sensitive, not proprietary: what ~all competent engineers assert without network access. A **narrow gate** with named disqualifiers: provider-specific behavior, proprietary status semantics, version-sensitive standards text, product-policy choices, and quantitative business thresholds never qualify.
|
||||
4. Nothing hits — the fact is **unreachable**. Before recording the marker, confirm the score actually requires asserting the fact: if the rubric credits, at the top tier, a response that surfaces the uncertainty, states its assumption, and scopes its claims *without* asserting the fact, the claim doesn't gate scoring after all — say so in the note and skip the marker.
|
||||
|
||||
An unreachable call must be backed by the searches actually run — the greps against the patched workspace, the files and session turns read. "The rubric doesn't cite a source for it" is not a search. And check the *patched* workspace in both directions: a `workspace.patch` can strip the comment that carried the constraint (the bare commit would wrongly say reachable) or plant the disclosure that makes it fair (the bare commit would wrongly say unreachable).
|
||||
|
||||
Whether an *expectation* built on a reachable fact is a fair ask is not this axis — facts have a true/false value the agent could in principle look up; preferences and judgment calls don't, and they stay with the human reviewer. When an unreachable scoring gate is found, the body's remediation note is always the same pair: put the fact in the materials (state it in the prompt, plant a reachable signal in the repo), or stop gating the score on it (grade the epistemic behavior — the overclaim, the unscoped assertion — instead of the hidden ground truth).
|
||||
|
||||
## Common defect classes
|
||||
|
||||
When categorizing each claim, use one of:
|
||||
|
||||
- `citation` — file path / line citation. Stale or invented citations show up here.
|
||||
- `dead-code` — grader treats a symbol as central without showing call sites.
|
||||
- `contradiction` — rubric self-contradicts (says X then cites a file that shows Y).
|
||||
- `schema` — DB / type-schema constraint claim.
|
||||
- `behavior` — function/method runtime behavior claim (returns null on X, gates on Y).
|
||||
- `prompt` — rubric attributes a statement to the prompt / ticket / snapshot ("the user says they are available to answer questions") — checked against `instruction.md` and the snapshot session, not the workspace.
|
||||
|
||||
This classification is used internally during checking; the saved
|
||||
per-claim record (see schema below) does NOT carry the `claimType` field.
|
||||
|
||||
## Failure modes to handle
|
||||
|
||||
- **Workspace not built and source repo unavailable.** `harbor-tasks/<slug>/environment/workspace/` is missing AND the build script (`scripts/build-workspace.sh` in the toolkit; `harbor-tasks/raccoon-shared/build-workspace.sh` in a repo checkout) can't build it (no source checkout at the toolkit's `repo/` or the repo checkout's `repos/<RepoName>/repo`, and no other local clone with the declared commit). Per-claim verdict for any claim whose cited file lives in that workspace: `unclear` (sub-case: source unavailable). If every claim is `unclear`, the top-level verdict is `not-applicable`. Note the build failure in the body's "Source" line.
|
||||
- **Workspace missing but buildable.** `environment/workspace/` is absent but the source repo and `workspace.patch` are present. Build the workspace before fact-checking — don't return `unclear`, you have everything you need.
|
||||
- **Rubric is empty / template.** Extract step emits `[]`. The save step records `claims: []` and `verdict: not-applicable`.
|
||||
- **`task.toml` missing or unreadable.** Treat as `not-applicable` with an explanatory note in the body.
|
||||
|
||||
## Frontmatter and body schema
|
||||
|
||||
The detector report is YAML frontmatter (with the structured `claims` array inline) followed by a markdown body. Both contexts produce the same shape; only the *production path* differs (the wrapping `SKILL.md` tells you how — single sequential session vs. parallel subagent fan-out).
|
||||
|
||||
**Frontmatter** — exactly these top-level keys:
|
||||
|
||||
```yaml
|
||||
---
|
||||
detector: detector-fact-check-rubric-claims
|
||||
verdict: pass | partial | fail | not-applicable
|
||||
confidence: HIGH | MEDIUM | LOW
|
||||
claims:
|
||||
- id: c01
|
||||
verdict: pass | unclear | partial | fail
|
||||
loadBearing: true | false
|
||||
summary: "<one-line claim summary, suitable as card title>"
|
||||
rubricQuote: "<verbatim from the resolved guidance file>"
|
||||
sourceEvidence: "<verbatim from the patched workspace at the cited lines>" | null
|
||||
sourceProvenance: "harbor-tasks/<slug>/environment/workspace/<path> (lines 42-58)"
|
||||
note: "<1-2 sentences explaining the verdict>"
|
||||
- id: c02
|
||||
...
|
||||
---
|
||||
```
|
||||
|
||||
Per-claim field rules:
|
||||
|
||||
- `id`: sequential string assigned by the extractor (`c01`, `c02`, …). Just an identifier — must be unique within the array.
|
||||
- `verdict`: one of `pass` / `unclear` / `partial` / `fail`.
|
||||
- `loadBearing`: whether the truth or falsity of this claim matters for the broader point the grader guidance is making (see "What counts as a load-bearing factual claim" above).
|
||||
- `summary`: one-line claim summary, suitable as a card title.
|
||||
- `rubricQuote`: verbatim quote from the resolved guidance file.
|
||||
- `sourceEvidence`: verbatim quote from the patched workspace at `harbor-tasks/<slug>/environment/workspace/<path>` at the cited lines. For `prompt`-type claims, the quote comes from `instruction.md` / the snapshot session instead. `null` when `verdict: unclear` and the sub-case is "source unavailable" (no source could be read).
|
||||
- `sourceProvenance`: the workspace path + line range the fact-checker read against (e.g., `harbor-tasks/<slug>/environment/workspace/app/foo.rb (lines 42-58)`). For `prompt`-type claims, the artifact read (e.g., `harbor-tasks/<slug>/instruction.md`). For the rare claim that needed git history from the source repo instead, record that command (e.g., `git -C repos/<RepoName>/repo log -S '<symbol>' <commit>`).
|
||||
- `note`: 1–3 sentences explaining the verdict, tying the rubric quote to the source evidence. For `unclear`, the note also names which sub-case fired (source unavailable vs. claim too vague). For scoring-gate claims, the note also carries the reachability finding: a leading `unreachable:` marker plus the searches that establish absence when the fact is nowhere in the package, or the discovery path (disclosing prompt/session line, workspace path, common-knowledge call) when it is reachable.
|
||||
|
||||
`confidence` reflects how confident you are in the per-claim verdicts as a set — `HIGH` when every claim has clear source evidence from a freshly-built patched workspace; `MEDIUM` when a few claims fell back to a separate clone of the source repo or the rubric had ambiguous wording; `LOW` when most claims were `unclear`.
|
||||
|
||||
**Body sections**, in this order:
|
||||
|
||||
```markdown
|
||||
# Fact-check rubric claims: <slug>
|
||||
|
||||
Source: `harbor-tasks/<slug>/environment/workspace/` — `git archive` of `repos/<RepoName>/repo` at commit `<short-sha>` (declared in `task.toml`) with `environment/workspace.patch` applied.
|
||||
Checked <N> claims (<L> load-bearing, <U> unclear, <R> unreachable, …). See per-claim list below.
|
||||
```
|
||||
|
||||
A body that's just the source line + a one-paragraph provenance summary is fine — the substance is in the structured `claims` array. The body is what a human reads if they want to skim; the array is what downstream tooling renders per-claim cards from. One exception: when any claim carries the `unreachable:` marker, the body must say so explicitly — name the unreachable claims (`Unreachable: c03 (the dedup retention window), c07 (the >= 3 paychecks threshold)`) and state the standard remediation pair for this task in a sentence (put the fact in the materials, or stop gating the score on it).
|
||||
|
||||
The Source line states only provenance you actually established. Verify the declared commit resolves (`git rev-parse` in the clone the workspace was built from) before naming it; if it doesn't resolve in any local clone and the workspace came from a fallback, say that instead. A report that asserts verification against a commit that doesn't exist locally rests its confidence on an unestablished fact — that's the same defect class this detector flags in rubrics.
|
||||
Reference in New Issue
Block a user