chore: create two files from instructions - generate... and theFailure

theFailure is my response to the describe the failure requirement.
generateAtomic... is the page in the docs formatted as markdown.
This commit is contained in:
2026-09-22 13:01:38 -04:00
parent f2b8e617d6
commit 3f433c35c9
2 changed files with 122 additions and 31 deletions

View File

@@ -0,0 +1,98 @@
# Generate the atomic rubric and its grades ​
Start here after saving reference runs graded with the final holistic rubric. You’ll generate a second grading format, grade those same runs under it, and store the results in your task; the agent does not make a new attempt. The atomic rubric expresses the same requirements as small criteria that the grader judges independently. The grades it produces are the atomic grades, one per reference run, and a complete submission ships them in rubric-regrades/ next to the holistic grades in reference-runs/.
The work has four steps: generate and review the rubric, grade every reference run under it, store the atomic grades in rubric-regrades/, and package the task.
Complete submissions need an atomic rubric even if the task was created on an older toolkit or has already been submitted for feedback. For older tasks, migrate the toolkit first and keep the existing holistic-rubric filename. The skill reads tests/grader-guidance-consolidated.md directly.
## Generate the files ​
Run /write-atomic-rubric in Claude Code or $write-atomic-rubric in Codex. The skill creates:
tests/atomic-rubric.yaml, containing the criteria; and
tests/grader-context.md, containing the context sections copied from the holistic rubric.
Do not write the atomic rubric from scratch. Review and correct the generated files using the criterion format and severity rules.
## Review the conversion ​
Compare the generated files with the holistic rubric:
- Every required or important requirement, penalty, and non-trigger needs a corresponding criterion.
- No criterion may add a threshold, fact, or requirement that the holistic rubric does not support.
- Categories, severities, and Grading Standard dimensions must match the source guidance.
- Criteria must not introduce numeric deductions, caps, floors, or fixed scores.
- Rewording must not strengthen or weaken a requirement.
- Copy the holistic rubric’s context sections into grader-context.md exactly, without rewriting or omitting anything.
Some repetition is necessary. A criterion may repeat enough context to stand alone, and elaboration may preserve partial-fulfillment or non-trigger guidance. Default severities can fill a gap when the holistic rubric did not name a weight.
## Stage the criteria ​
Atomic grading reads temporary staged files rather than atomic-rubric.yaml directly:
```
npx tsx scripts/stage-atomic-rubric.ts <slug>
```
The command requires grader-context.md and writes:
rubric-criteria.md, the criterion text the grader reads;
rubric-criteria.json, metadata used by the score renderer; and
render-rubric-grade.py, the shared renderer.
Run the staging command again after every atomic-rubric edit.
## Grade every reference run under the atomic rubric ​
From the toolkit root in Authoring, run this once for every reference run:
```
HARBOR_REGRADE_OUT=harbor-jobs/<run> HARBOR_GRADER_MODE=rubric-trinary scripts/harbor-regrade \
harbor-tasks/<slug> \
harbor-tasks/<slug>/reference-runs/<run> \
--verifier-env GRADER_SAMPLES=1
```
Replace <run> with the reference run’s folder name in both places, for example reward-0.62-h4KNEAg, and <slug> with your task folder’s name. <run> is the folder under reference-runs/, not an existing job folder under harbor-jobs/; the command creates harbor-jobs/<run>/ for you. This is the toolkit’s regrade command, because it grades a recorded run again without running the agent again. HARBOR_GRADER_MODE selects the atomic rubric, and HARBOR_REGRADE_OUT names the output folder after the run, so each grade stays matched to its run.
A grade usually takes 15–30 minutes. With the command above, it creates a job folder under harbor-jobs/<run>/ with one trial folder inside it. The trial’s verifier/ folder holds reward.txt, the authoritative reward, along with grade.md and rubric-grade.json, which records each criterion verdict and rationale.
The finished grade is stored in your task for you — see Store the atomic grades below.
Grades can run in parallel, one command per run, but each needs memory. With 4 GB allocated to Docker, run only one or two at once.
## Check that the two grading methods agree ​
Read every criterion verdict and confirm that it describes behavior that occurred. Then compare each atomic reward with the original holistic grade.
- The scores do not need to match exactly. Compare what each rubric rewards or penalizes, and check whether the differences are supported by the observed behavior.
- A difference within 0.15 is a useful rule of thumb, not a hard requirement.
- Runs whose holistic scores differ by about 0.05 may change order because of grader variance.
- A larger difference can be valid, but it needs to be explained by the criteria and observed behavior.
When the results disagree, investigate why. Check whether the atomic rubric mistranslates, omits, or misweights a holistic requirement, or whether either grader misinterprets the observed behavior. Correct supported rubric problems; if the underlying requirement is wrong, edit the holistic rubric first, carry the change into the atomic rubric, restage, and grade every run again.
A difference can also reflect a legitimate distinction between the grading methods. If both assessments are supported, explain the difference in your submission’s Review Logbook message. Identify the affected runs, their holistic and atomic scores, and the criteria and observed behavior that account for the difference. Do not weaken a supported requirement merely to make the numbers agree.
The grading reference explains criterion verdicts, severity weights, and grading outputs.
## Store the atomic grades ​
Your submission carries the atomic grade of every reference run, so a reviewer can compare both grades of each run without grading it again.
This happens for you. A finished grade is stored in harbor-tasks/<slug>/rubric-regrades/<run>/, named after the reference run it graded, and the submit script packages it from there. Nothing to copy, and no name to choose.
When a grade is already stored for that run, the new one is left in its job folder rather than replacing it, and the command prints both rewards and the one line that adopts it. An earlier grade is never overwritten unless you ask. Add --replace to store each new grade as it finishes, which is the usual thing to want after correcting the rubric and staging it again:
```
HARBOR_GRADER_MODE=rubric-trinary scripts/harbor-regrade \
harbor-tasks/<slug> --all --replace \
--verifier-env GRADER_SAMPLES=1
```
Store grades only from the final state of your atomic rubric. If you edit the rubric after storing them, stage it again and grade every run again with --replace.
Atomic grades never belong in reference-runs/. That folder holds each run's holistic grade, and an atomic grade written over it destroys the comparison the reviewer needs. The toolkit refuses to do it.
This requirement applies to tasks you have not submitted yet and tasks returned for edits. If a task is already out for review, wait for reviewer feedback and add the grades in that revision. See Submit for review for the submission and review checks.
## Run the final detectors and restore ​
Run the atomic-rubric detectors: rubric-coverage and rubric-form. Read both reports and correct supported findings.
Remove the temporary staged files before packaging:
```
npx tsx scripts/stage-atomic-rubric.ts <slug> --restore
```