Worker detector shell

This file is the shared boilerplate for all detector self-check skills under .claude/skills/ in the worker toolkit. Each detector's SKILL.md points at this file plus its own core.md so the output path convention, the directory-creation step, and the overwrite semantics don't duplicate across detectors.

Resolve the guidance target first

A task directory can carry two guidance files: tests/grader-guidance-consolidated.md (the Consolidated Grading Standard) and tests/grader-guidance.md (the legacy standard). Wherever your detector's core.md reads or assesses "the grader guidance", it means the file the grader will actually use. Resolve it before reading anything:

bash scripts/guidance-target.sh <slug>

It prints the guidance file's path and the standard it grades under (consolidated or legacy), using the same rule as the grader itself. Assess that file, and assess it against its own standard's structure:

Consolidated: Task context, optional Business context, Ground truth, one section per criterion in the standard's order, optional Heavy penalties.
Legacy: Task context, Business context, what strong and weak responses look like, Ground truth, optional Supporting evidence, optional Correctness, optional Heavy penalties.

Never flag a document for not following the other standard's structure, and never assess the file the resolver did not name. To assess the legacy file deliberately (for a task being graded with GRADING_STANDARD=legacy), set that variable when running the resolver.

Open the report body with one line naming what you assessed:

Assessed: <resolved-path> (<consolidated|legacy> standard)

Output path

Compose the detector report (YAML frontmatter + markdown body) per the schema in your detector's core.md. The frontmatter must include detector, verdict, and confidence; detectors with structured payloads (detector-fact-check-rubric-claims → claims, detector-run-behaviors → runBehaviors) embed them in the same frontmatter block as the other fields.

Write the report to:

harbor-tasks/<slug>/detectors/<detector-name>.md

<detector-name> is the value of the detector: frontmatter field — e.g. detector-snapshot-leakage, detector-cross-task-reference, detector-meaningful-failure, detector-rubric-clarity, detector-rubric-generality, detector-answer-obviousness, detector-good-response-defined, detector-good-response-exhaustiveness, detector-dimension-misapplication, detector-broken-dev-env, detector-over-hinting, detector-offline-verifiability, detector-credential-leakage, detector-fact-check-rubric-claims, detector-run-behaviors.

Directory + overwrite semantics

Create the detectors/ directory if it doesn't exist (mkdir -p).
Overwrite the file if it already exists from a prior run. Detector skills are meant to be re-runnable — every time you tweak the rubric, the snapshot, or anything else this detector reads, re-run the skill and read the fresh report.

Record what the report assessed

After writing (or rewriting) the report, stamp it with the checksums of the task inputs it assessed:

npx tsx scripts/record-detector-inputs.ts <slug> <detector-name>

This writes harbor-tasks/<slug>/detectors/<detector-name>.inputs.json. submit-task.ts compares those checksums against the task at packaging time and warns when the report predates a prompt or grader-guidance edit — by content, so it stays accurate even when file timestamps get disturbed. An unstamped report falls back to the less reliable timestamp comparison. Re-run the command after every re-run of the detector.

Verdict and body schema live in core.md

Each detector's core.md is the source of truth for its verdict enum and the body section structure. Don't restate them in SKILL.md — read core.md and use the values it specifies.

3.9 KiB Raw Blame History