added 260907 version of worker toolkit
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: detector-rubric-coverage
|
||||
description: |
|
||||
Self-check that your atomic rubric fully captures your holistic rubric.
|
||||
Verifies four things. Every load-bearing requirement, penalty, and
|
||||
"do not penalize" rule in the holistic rubric maps to a criterion. No
|
||||
criterion invents a requirement or an answer-key fact the holistic rubric
|
||||
does not support. The holistic rubric's context sections survive in
|
||||
`tests/grader-context.md`. Every heavy penalty that targets the overall
|
||||
score is encoded as a crux criterion, or at `certain_dealbreaker` once two
|
||||
criteria already carry crux. Restructuring is never flagged; only
|
||||
content differences that change scoring are. Reads the holistic rubric,
|
||||
`tests/atomic-rubric.yaml` (or `tests/rubrics.yaml`), and
|
||||
`tests/grader-context.md`. Emits `not-applicable` when the task has no
|
||||
atomic rubric yet.
|
||||
allowed-tools: Bash, Read, Write
|
||||
---
|
||||
|
||||
# Rubric-coverage detector
|
||||
|
||||
This skill checks that your atomic rubric and your holistic rubric express the
|
||||
same task. The atomic rubric restructures the holistic rubric into criteria.
|
||||
It must not lose scoring content, and it must not add scoring content.
|
||||
|
||||
The failure shapes to catch:
|
||||
|
||||
- **A lost requirement or penalty.** The holistic rubric requires something,
|
||||
or penalizes something, and no criterion captures it. A response the
|
||||
holistic rubric would mark down now scores clean.
|
||||
- **A lost "do not penalize" rule.** The holistic rubric protects a behavior,
|
||||
and the criteria drop the protection. The atomic rubric now penalizes what
|
||||
the holistic rubric permits.
|
||||
- **Invented content.** A criterion requires something the holistic rubric
|
||||
never asks for, or states an answer-key fact with no source in the holistic
|
||||
rubric or the context document.
|
||||
- **Lost context.** A ground-truth fact that criteria rely on is missing from
|
||||
both `tests/grader-context.md` and the criteria themselves.
|
||||
- **A crux mismatch.** The holistic rubric applies a heavy penalty against
|
||||
the overall score, and no criterion carries `severity: crux` to encode it.
|
||||
A task carries at most two crux criteria; once two are designated, a
|
||||
further overall-score penalty is correctly encoded at `certain_dealbreaker`.
|
||||
|
||||
Read these before deciding:
|
||||
|
||||
1. `.claude/skills/_detector-worker-shell.md` — where to write the report and how to handle re-runs.
|
||||
2. `.claude/skills/detector-rubric-coverage/core.md` — what counts as a coverage gap versus invented content, the crux-alignment rule, what is deliberately not a finding, verdict definitions, and the body schema.
|
||||
|
||||
Compose the report per the schema in `core.md` and write it per `_detector-worker-shell.md`.
|
||||
|
||||
## Acting on the verdict
|
||||
|
||||
- **`clear`** — the atomic rubric fully captures the holistic rubric. A
|
||||
grader scoring from either form would land in the same place.
|
||||
- **`minor-issues`** — the load-bearing mapping is sound, but some
|
||||
non-load-bearing content drifted. Read the findings and tighten the
|
||||
conversion. There is no need to rebuild the rubric.
|
||||
- **`material-issues`** — a load-bearing requirement, penalty, or protection
|
||||
is missing, a criterion invents content, needed context is gone, or a
|
||||
heavy penalty against the overall score has no criterion encoding it at
|
||||
`crux` (or at `certain_dealbreaker` once two crux criteria exist). Fix the
|
||||
named findings in the atomic rubric. If a finding reveals that the
|
||||
holistic rubric itself needs the change, edit the holistic rubric first
|
||||
and then re-convert, so the two forms stay in agreement. Re-run this
|
||||
skill after editing either file.
|
||||
- **`not-applicable`** — the task has no atomic rubric yet, or no holistic
|
||||
rubric to compare it against. Write the missing rubric first, then come
|
||||
back to this skill.
|
||||
Reference in New Issue
Block a user