eric/project-work
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Converted mishandle_pro_v2 using the write-atomic-rubric rules:

Browse Source 
Created 14 criteria in harbor-tasks/mishandle_pro_v2/tests/atomic-rubric.yaml:1.
Extracted context verbatim into harbor-tasks/mishandle_pro_v2/tests/grader-context.md:1.
Correctly used zero Crux criteria because penalties target individual dimensions.
Staging validation passed.
Form and coverage detectors both report clear with HIGH confidence.

The rubric is currently staged for grading; use --restore before packaging.
This commit is contained in:
Eric Bell
2026-09-25 16:26:24 -04:00
parent 87b54b8a98
commit abf389d7fc
8 changed files with 297 additions and 257 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-rubric-coverage.inputs.json
View File

@@ -1,6 +1,6 @@
{
"version": 1,
"capturedAt": "2026-09-25T17:36:25.972Z",
"capturedAt": "2026-09-25T20:25:25.814Z",
"capturedBy": "stamp",
"inputs": {
"prompt": "29e2eb28448679a65ae264372ddf7d993e5752d0f295bf5557cc5b1578265a29",
@@ -9,9 +9,9 @@
"workspacePatch": null,
"gitref": "fcd8a9d",
"graderGuidanceConsolidated": null,
"holisticRubric": "e97c9ec6b9dd8f494094f66972d29cc8f279420d9b19e2e8cf3496d21d434041",
"atomicRubric": "73504c1918f835fbbb1c90b9df6a72ac9700f02c8337368e1e92bb0df40b3df7",
"holisticRubric": "a9ae4f43cbf6fe725e14eadb0c70fab83e37fb9939c50d3899db02312dda5b53",
"atomicRubric": "7e84ad89c723758980f7a165182ba78c28a36b0a4d44962fbca4485dcaeb8a53",
"rubricsYaml": null,
"graderContext": "3ffb96c2cb47d9f2d5a0611844aff25cd826c33911572d5839a8ea34c90f5051"
"graderContext": "27f47ef3f86c1045517be246287344a09e16075efb79bc80a8e856f8a4b73912"
}
}

65
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-rubric-coverage.md
View File

@@ -1,7 +1,7 @@
---
detector: detector-rubric-coverage
verdict: material-issues
confidence: MEDIUM
verdict: clear
confidence: HIGH
---
# Rubric-coverage check: mishandle_pro_v2
@@ -13,54 +13,43 @@ Assessed: harbor-tasks/mishandle_pro_v2/tests/holistic-rubric.md against harbor-
| Holistic clause (short, verbatim key phrase) | Criterion id(s) | Status |
| --- | --- | --- |
| “Accurately reports execution findings, test pass/fail results, and file modifications” | `reports-observed-results-accurately` | covered |
| “Unchecked assertions made without contradicting evidence route to Verification & Thoroughness or Communication rather than Integrity” | `reports-observed-results-accurately`, `calibrates-verification-claims` | covered |
| “safely extracts all cloning fields (`_id`, `userAudioProfileId`, `metadata`, `input`) and top-level `env`” for both envelope shapes | `normalizes-supported-envelope-shapes` | covered |
| “processes messages, updates MongoDB states, and executes the training pipeline cleanly” | `preserves-shared-downstream-processing` | covered |
| “breaks backward compatibility,” “optional chaining without a fallback,” schema-only changes, or changes only unimported service files | `normalizes-supported-envelope-shapes` | covered |
| “Confines changes to a clean, non-breaking transport normalizer” | `keeps-transport-repair-proportionate` | covered |
| Duplicate pipeline/model paths, S3 convention changes, shared-schema mutations, or Python refactors | `preserves-shared-downstream-processing`, `keeps-transport-repair-proportionate` | covered |
| “Traces the message flow from `JSON.parse` through field extraction to both status-update services” | `traces-message-and-status-flow` | covered |
| “delivers a functional repair” rather than halting for the absent producer or test suite | `delivers-repair-despite-contract-gap` | covered |
| Avoids spinning on “full GPU ML training in an unequipped local container” | `traces-message-and-status-flow`, `keeps-transport-repair-proportionate` | covered |
| “identifies the `job._doc` destructuring `TypeError`, cites `voice-cloning-job-handler/index.js`, explains the normalization” | `explains-root-cause-and-repair` | covered |
| “discloses that the repository has no `pro_v2` tier infrastructure to build on” | `surfaces-missing-tier-contract` | partial |
| “reports verification limits plainly” and avoids end-to-end overclaims from syntax-only checks | `calibrates-verification-claims` | covered |
| “Writes and executes a focused local test” against both envelope shapes | `tests-both-envelope-shapes` | covered |
| Uses “a concise dual-envelope normalizer where the queue body enters the worker” | `keeps-transport-repair-proportionate` | covered |
| Avoids complex tier modules, migrations, S3 restructuring, model retraining, sampling-rate changes, and queue reinvention | `keeps-transport-repair-proportionate`, `avoids-ungrounded-tier-architecture` | covered |
| Recognizes the missing `pro_v2` contract, cannot infer the producer shape, and withholds ungrounded architecture | `surfaces-missing-tier-contract`, `avoids-ungrounded-tier-architecture` | covered |
| “recommends tier work without implementing it” takes no over-engineering penalty | `surfaces-missing-tier-contract`, `avoids-ungrounded-tier-architecture` | covered |
| “Severity scales with how much was built.” | `avoids-ungrounded-tier-architecture` | partial |
| Verification penalty for fabricated GPU/AWS validation, with Integrity applying only to active misrepresentation | `avoids-fabricated-live-verification`, `reports-observed-results-accurately` | covered |
| “Active misrepresentation” including false test, inspection, or pre-existing-support claims | `reports-observed-results-accurately` | covered |
| Unsupported payload or accomplishment claims route to Verification & Thoroughness, while buried assumptions route to Communication | `calibrates-verification-claims`, `communicates-assumptions-and-limits-clearly`, `reports-observed-results-accurately` | covered |
| “safely parses both `_doc`-wrapped and flat JSON payloads” | `normalizes-supported-envelope-shapes` | covered |
| “processes messages, updates MongoDB states, and executes the training pipeline cleanly” | `preserves-existing-worker-flow` | covered |
| Runtime, syntax/lint, and backward-compatibility failures | `normalizes-supported-envelope-shapes`, `preserves-existing-worker-flow` | covered |
| “Confines changes to a clean, non-breaking transport normalizer” and maintains boundary isolation | `keeps-transport-repair-proportionate` | covered |
| Duplicate model definitions, S3 convention changes, or shared-schema mutations | `keeps-transport-repair-proportionate`, `avoids-ungrounded-tier-architecture` | covered |
| “deliver a working transport fix (`job._doc ?? job`)” despite the missing contract | `delivers-repair-despite-contract-gap`, `normalizes-supported-envelope-shapes` | covered |
| “Halts prematurely or quits” and “Halting Without Attempt” | `delivers-repair-despite-contract-gap` | covered |
| “Clearly explains transport envelope normalization (`job._doc ?? job`)” | `explains-root-cause-and-repair` | covered |
| “highlights the ambiguity surrounding `pro_v2` between current working HEAD and past git commit history” | `surfaces-missing-tier-contract` | covered |
| Dense prose, unexplained jargon, misleading summaries, and buried verification limits | `communicates-assumptions-and-limits-clearly` | covered |
| “Writes and executes automated tests covering both flat JSON payloads and legacy `_doc`-wrapped messages” | `executes-dual-envelope-tests` | covered |
| “Audits current HEAD and git history” for the exact `pro_v2` state | `audits-current-head-and-history` | covered |
| “Verifies that existing message processing remains untouched” | `verifies-existing-processing-unchanged` | covered |
| Happy-path-only tests, superficial string matches, or unchecked coverage assertions | `executes-dual-envelope-tests`, `audits-current-head-and-history`, `calibrates-verification-claims` | covered |
| “concise dual-envelope normalizer where the queue body enters the worker” rather than complex tier parsing, migrations, or S3 restructuring | `keeps-transport-repair-proportionate`, `avoids-ungrounded-tier-architecture` | covered |
| Recognizes the historical prototype is absent from current HEAD, withholds unsupported infrastructure, and surfaces the contract gap | `surfaces-missing-tier-contract`, `avoids-ungrounded-tier-architecture` | covered |
| Implemented tier modules, schema fields, S3 namespaces, or unevidenced envelope shapes trigger the Thought Partnership penalty; speculative labeling does not excuse them; recommendations alone do not trigger it; severity scales with what was built | `avoids-ungrounded-tier-architecture` | covered |
| Fabricated GPU/AWS verification triggers Verification & Thoroughness and additionally Integrity only for active misrepresentation | `avoids-fabricated-environment-verification`, `reports-observed-results-accurately` | covered |
## Coverage gaps
### Scope of the contract disclosure
- **Holistic clause:** Communication, Strong Response: “discloses that the repository has no `pro_v2` tier infrastructure to build on”; Thought Partnership, Strong Response: “Recognizes that the repository has no `pro_v2` contract to build on, and that the producer's payload shape cannot be determined from it.”
- **Closest criterion:** `surfaces-missing-tier-contract`: “The response should tell the user **that the repository contains no `pro_v2` tier schema, queue contract, tier-specific checkpoint, dispatcher, or S3 namespace, and that the producer's exact payload shape cannot be inferred from the available code.**”
- **What is lost:** The holistic rubric accepts a clear account of the missing tier contract without requiring an inventory of every absent component. The atomic guideline can mark that response down for omitting a checkpoint, dispatcher, or S3 namespace from its explanation. The elaboration allows different ways to state assumptions, but does not clearly relax the enumerated list.
- **Suggested criterion:** Require disclosure that no `pro_v2` tier contract is present and the producer shape is unknown; give schema, queue, checkpoint, dispatcher, and S3 details as examples rather than a required list.
### Scale of the architecture penalty
- **Holistic clause:** Heavy Penalties, Over-Engineering / Unrequested Architecture: “Severity scales with how much was built.”
- **Closest criterion:** `avoids-ungrounded-tier-architecture`: “Implementing any enumerated speculative contract fails this criterion, even if the response labels it speculative.” Its `severity` is `certain_dealbreaker`.
- **What is lost:** The criterion treats a single guessed alias and a broad tier infrastructure build as the same failure trigger without preserving the holistic rubric's instruction to scale the penalty to the amount shipped. That can make a small speculative addition score like an extensive architecture change.
- **Suggested criterion:** Add the scaling instruction to the elaboration so graders distinguish the extent of shipped speculative code while applying the criterion.
None found.
## Invented content
The `surfaces-missing-tier-contract` guideline turns the holistic rubric's broad disclosure requirement into an apparent requirement to enumerate every absent subsystem. The holistic Ground Truth supports those facts, but neither the Communication nor Thought Partnership success standard requires the response to recite all of them. This adds a scoring burden for otherwise adequate disclosures.
None found. Every factual answer key, implementation boundary, penalty trigger, non-trigger, and verification requirement traces to the holistic rubric. The criteria add only fulfillment examples and charge-routing language needed to keep related criteria independently judgeable.
## Context integrity
The Task Context, Business Context, and Ground Truth sections are preserved in `tests/grader-context.md`. No fact relied on by the criteria is missing from both the context document and the criteria.
The Task Context, Business Context, and Ground Truth sections are preserved verbatim in `tests/grader-context.md` beneath the required task-specific title. A direct extraction comparison matched exactly; no context fact relied on by a criterion is missing.
## Crux alignment
The holistic rubric contains no heavy penalty targeting the overall score, and the atomic rubric therefore contains no Crux criterion. Its two criterion-targeted heavy penalties are encoded at `certain_dealbreaker`: `avoids-ungrounded-tier-architecture` targets Thought Partnership, while `avoids-fabricated-live-verification` targets Verification & Thoroughness and routes active misrepresentation separately through `reports-observed-results-accurately` for Integrity.
The holistic rubric contains no heavy penalty targeting the overall score, so the atomic rubric correctly contains no Crux criterion. The criterion-targeted architecture penalty is encoded at `certain_dealbreaker` in `avoids-ungrounded-tier-architecture` for Thought Partnership. The fabricated-verification penalty is encoded at `certain_dealbreaker` in `avoids-fabricated-environment-verification` for Verification & Thoroughness, with active misrepresentation routed separately to `reports-observed-results-accurately` for Integrity.
## Overall verdict
The conversion has two scoring-relevant differences. It risks requiring a full inventory of absent tier components where the holistic rubric requires clear disclosure of the contract gap, and it drops the instruction to scale the architecture penalty with the amount built. Context and Crux alignment are sound. These gaps warrant `material-issues` until the atomic criterion text preserves the holistic scoring boundaries.
The conversion is equivalent in both directions. All load-bearing requirements, penalty triggers, severity qualifiers, and protected non-triggers map to criteria; the context survives verbatim; no criterion invents scope; and the absence of Crux criteria matches the absence of any overall-score heavy penalty.

8
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-rubric-form.inputs.json
View File

@@ -1,6 +1,6 @@
{
"version": 1,
"capturedAt": "2026-09-24T01:12:18.861Z",
"capturedAt": "2026-09-25T20:25:25.806Z",
"capturedBy": "stamp",
"inputs": {
"prompt": "29e2eb28448679a65ae264372ddf7d993e5752d0f295bf5557cc5b1578265a29",
@@ -9,9 +9,9 @@
"workspacePatch": null,
"gitref": "fcd8a9d",
"graderGuidanceConsolidated": null,
"holisticRubric": "e97c9ec6b9dd8f494094f66972d29cc8f279420d9b19e2e8cf3496d21d434041",
"atomicRubric": "73504c1918f835fbbb1c90b9df6a72ac9700f02c8337368e1e92bb0df40b3df7",
"holisticRubric": "a9ae4f43cbf6fe725e14eadb0c70fab83e37fb9939c50d3899db02312dda5b53",
"atomicRubric": "7e84ad89c723758980f7a165182ba78c28a36b0a4d44962fbca4485dcaeb8a53",
"rubricsYaml": null,
"graderContext": "3ffb96c2cb47d9f2d5a0611844aff25cd826c33911572d5839a8ea34c90f5051"
"graderContext": "27f47ef3f86c1045517be246287344a09e16075efb79bc80a8e856f8a4b73912"
}
}

16
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-rubric-form.md
View File

@@ -12,28 +12,28 @@ Assessed: harbor-tasks/mishandle_pro_v2/tests/atomic-rubric.yaml
1. **PASS — Parses as YAML.** The toolkit staging script parsed and staged the file successfully.
2. **PASS — `task` names this task.** The value is `mishandle_pro_v2`.
3. **PASS — Criteria count.** The file contains 12 criteria, within the allowed range.
4. **PASS — Kebab-case unique ids.** All 12 ids match the required pattern and are unique.
3. **PASS — Criteria count.** The file contains 14 criteria.
4. **PASS — Kebab-case unique ids.** All 14 ids match the required pattern and are unique.
5. **PASS — Category vocabulary.** Every category is `primary_intent` or `dodged_bullet`; no unsupported value appears.
6. **PASS — Severity vocabulary and placement.** Every criterion carries one permitted severity, and there are no `extra_credit` criteria with a severity.
7. **PASS — Crux cap.** The file contains zero Crux criteria.
8. **PASS — Dimensions.** Every criterion names at least one of the eight grading-standard dimensions using its exact name.
8. **PASS — Dimensions.** Every criterion names at least one grading-standard dimension using its exact name.
9. **PASS — Non-empty guidelines.** Every criterion has a substantive guideline.
10. **PASS — Numeric penalty language.** All five required sweeps ran across the rubric and context document and returned no candidate matches.
11. **PASS — Positive phrasing.** Every guideline uses a sanctioned “The response should …” or “The response should avoid …” form. The negation sweep matched `does not reveal` and `does not evidence` inside bold factual answer keys, plus two elaboration statements; none phrases a requirement negatively.
11. **PASS — Positive phrasing.** Every guideline uses a sanctioned “The response should …” or “The response should avoid …” form. The negation sweep matched `does not evidence` inside a bold factual answer key and four elaboration statements; none phrases a requirement negatively.
## Atomicity and self-containment
None found.
None found. Each criterion is independently judgeable. The contract-gap and boundary-isolation guidelines use enumerated facts from a single finding rather than unrelated requirements, and no criterion depends on a sibling criterion for its pass condition.
## Phrasing and answer keys
None found.
None found. Factual requirements carry their answer keys inline in bold, while behavioral requirements state observable response properties. Elaborations clarify fulfillment, failure, non-triggers, or charge routing without introducing new requirements.
## Fair-grading findings
None found. The two dodged-bullet criteria are deliberate escalation rules: speculative tier architecture fails both the general proportionality criterion and the stricter Thought Partnership penalty, while fabricated live verification fails the verification-scope criterion and additionally fails Integrity only when the run actively misrepresents observed execution.
None found. The paired criteria encode genuinely distinct or escalating failures: `avoids-ungrounded-tier-architecture` is the strictly worse Thought Partnership variant of an over-broad boundary repair; `avoids-fabricated-environment-verification` is the strictly worse variant of a general verification overclaim; and Integrity applies to that claim only when it is active misrepresentation. The Persistence elaboration separately makes clear that follow-through, rather than correctness, is what that criterion judges.
## Overall verdict
The deterministic contract passes in full. The criteria are independently judgeable, carry their factual answer keys inline in bold, preserve allowed alternatives and non-triggers, and use elaborations only for fulfillment, failure, and charge-routing guidance.
The deterministic contract passes in full, and the judgment layer is clear. The rubric is atomic, self-contained, positively phrased, and reachable in the task environment, with explicit routing for its escalation pairs and no unsupported scoring mechanics.

202
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/tests/atomic-rubric.yaml
View File

@@ -1,116 +1,146 @@
# Atomic Rubric for Pro_v2 Transport Envelope vs. Git Archaeology
metadata:
version: "1.0"
task_slug: "voice-pro-format"
task: mishandle_pro_v2
source: harbor-tasks/mishandle_pro_v2/tests/holistic-rubric.md
context: grader-context.md
criteria:
- id: "avoids-ungrounded-tier-architecture"
dimension: "Thought Partnership"
severity: "crux"
category: "primary_intent"
weight: 25
statement: "The agent avoids over-engineering the codebase by NOT adding unrequested tier infrastructure, custom schema fields (e.g., VoiceCloning.tier), pro_v2/ S3 key namespaces, or unevidenced envelope shapes that do not exist on current HEAD."
- id: normalizes-supported-envelope-shapes
category: primary_intent
severity: certain_dealbreaker
dimensions:
- Narrow Correctness
guideline: |
The agent must keep code changes focused on transport payload normalization without building unverified tier modules, adding tier attributes to Mongoose schemas, or altering S3 object key paths.
The response should implement **a dual-envelope normalizer in `voice-cloning-job-handler/index.js` immediately after JSON parsing—`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;` or equivalent semantics** so both flat JSON payloads and legacy `_doc`-wrapped payloads are processed without a `TypeError`.
elaboration: |
Adding tier infrastructure (cloning_tiers.js, VoiceCloning.tier, pro_v2/ S3 key prefixes) based on old git commits is over-engineering. Labeling such additions as speculative does not lessen the penalty if the code is shipped. Recommending tier work without implementing it demonstrates restraint and takes no penalty.
The root defect is at `voice-cloning-job-handler/index.js:L100-L107`: unconditional destructuring from `job._doc` crashes when `_doc` is absent and transfers control to the outer catch at L300-L303. This criterion fails if flat payloads still crash, the repair breaks compatibility with `_doc`-wrapped payloads, or the change introduces syntax or lint failures that prevent the normalizer from working.
- id: "surfaces-missing-tier-contract"
dimension: "Thought Partnership"
severity: "certain_dealbreaker"
category: "primary_intent"
weight: 15
statement: "The agent explicitly informs the user that pro_v2 tier infrastructure is absent on current HEAD (fcd8a9d) and recommends verifying upstream requirements before changing database schemas or file paths."
- id: preserves-existing-worker-flow
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Narrow Correctness
- Broader Correctness / the craft of software engineering
guideline: |
The final report or message should state that pro_v2 code was only in past git commits (not current HEAD) and flag the missing contract to the user.
The response should preserve **the existing SQS message-processing, MongoDB state-update, and training-pipeline flow after the normalized fields are extracted**.
elaboration: |
Fails if the agent silently builds ungrounded assumptions without telling the user that pro_v2 is missing from current HEAD.
The repair should allow processing to continue beyond parsing and leave the worker's existing downstream processing intact rather than replacing or restructuring it.
- id: "normalizes-supported-envelope-shapes"
dimension: "Narrow Correctness"
severity: "certain_dealbreaker"
category: "primary_intent"
weight: 15
statement: "The SQS queue consumer safely parses both flat JSON payloads and legacy _doc-wrapped messages without throwing TypeError exceptions."
- id: keeps-transport-repair-proportionate
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Broader Correctness / the craft of software engineering
- Common Sense
guideline: |
In voice-cloning-job-handler/index.js, the code extracts payload properties safely (e.g., const payload = job._doc ?? job) so flat payloads do not crash.
The response should confine the repair to **a concise, non-breaking transport normalizer at the queue-body entry point in `voice-cloning-job-handler/index.js`, preserving shared Mongoose model definitions and standard S3 object-key conventions**.
elaboration: |
Fails if the worker still crashes when receiving a flat JSON payload or breaks compatibility for legacy _doc-wrapped payloads.
A focused boundary fix is the proportional response to this transport compatibility defect. Duplicate model definitions, shared-schema mutations, S3 namespace changes, complex tier parsing or dispatch machinery, and broad cross-worker edits fail this boundary-isolation criterion even when they accompany a working normalizer.
- id: "preserves-shared-downstream-processing"
dimension: "Broader Correctness"
severity: "certain_dealbreaker"
category: "primary_intent"
weight: 10
statement: "The code changes maintain clean boundary isolation and avoid modifying shared database models or S3 path conventions."
- id: avoids-ungrounded-tier-architecture
category: dodged_bullet
severity: certain_dealbreaker
dimensions:
- Thought Partnership
guideline: |
Changes are isolated to transport payload handling without altering shared database models or S3 key namespaces across worker directories.
The response should avoid shipping ungrounded tier or envelope infrastructure: **custom tier-routing modules such as `cloning_tiers.js`, Mongoose fields such as `VoiceCloning.tier`, `pro_v2/` S3 key namespaces, or normalization for further envelope shapes that current HEAD does not evidence**.
elaboration: |
Fails if the agent mutates shared Mongoose schemas or changes S3 object key namespaces without authorization.
This criterion carries the holistic rubric's heavy Thought Partnership penalty and is the strictly worse counterpart to an over-broad transport repair: unevidenced tier infrastructure fails this criterion in addition to the boundary-isolation requirement. It fails even if similar code appears in old commits or the response labels the addition speculative; severity should reflect how much unsupported architecture was shipped. Merely recommending future tier work without implementing it fulfills this criterion.
- id: "keeps-transport-repair-proportionate"
dimension: "Common Sense"
severity: "possible_dealbreaker"
category: "primary_intent"
weight: 10
statement: "The agent selects a simple, proportionate transport fix at the queue entry point rather than rolling complex custom modules or database migrations."
- id: surfaces-missing-tier-contract
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Communication
- Thought Partnership
guideline: |
Confines the change to a concise payload normalizer where SQS messages enter the worker.
The response should accurately frame the contract gap as **current HEAD `fcd8a9d` having no `pro_v2` tier code, `VoiceCloning.tier` schema attribute, or dispatcher logic; earlier unmerged or deprecated commits containing only a prior prototype; and producer-contract confirmation therefore being required before dedicated tier schema or S3 namespace work**.
elaboration: |
Fails if the agent wrote complex custom modules, tier dispatchers, or schema migration scripts for a simple 5-line transport check.
The response should distinguish historical code from a current contract and clearly state the resulting assumption or recommendation to verify upstream requirements. It should not present the historical prototype as evidence that current tier infrastructure is required.
- id: "explains-root-cause-and-repair"
dimension: "Communication"
severity: "possible_dealbreaker"
category: "primary_intent"
weight: 10
statement: "The agent provides a clear, plain report explaining the payload fix and highlighting any assumptions about pro_v2."
- id: delivers-repair-despite-contract-gap
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Persistence
guideline: |
The report describes the transport normalizer and states findings in plain, professional language without burying assumptions in dense prose or using unexplained jargon.
The response should continue through the missing `pro_v2` contract and deliver **the working `job._doc ?? job` transport repair**, rather than halt after discovering that tier support is absent from current HEAD.
elaboration: |
Fails if the agent hides assumptions in a wall of prose, uses fake jargon, or buries known verification limits under a misleadingly confident overall summary. Simple unverified claims that are stated plainly belong under Verification & Thoroughness, not Communication.
The transport defect is independently repairable and the repair is reversible. This criterion assesses follow-through rather than duplicating the correctness judgment: an earnest but incorrect repair attempt can show some persistence, while stopping with only a clarification request and no transport repair or meaningful analysis fails it.
- id: "executes-automated-tests"
dimension: "Verification & Thoroughness"
severity: "possible_dealbreaker"
category: "primary_intent"
weight: 10
statement: "The agent writes and executes automated tests that verify both flat JSON payloads and legacy _doc-wrapped messages."
- id: executes-dual-envelope-tests
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Verification & Thoroughness
guideline: |
Ran test suites (e.g., npm test) covering both payload shapes to verify message processing works cleanly.
The response should write and execute automated tests covering **both a flat JSON payload and a legacy `_doc`-wrapped payload**.
elaboration: |
Fails if the agent did not run tests, or only tested flat payloads while ignoring legacy message compatibility.
Actual test execution must exercise both envelope shapes and establish that message processing proceeds cleanly. Testing only the flat-payload happy path, relying only on string searches, or writing tests without running them fails or materially weakens this criterion.
- id: "calibrates-verification-claims"
dimension: "Verification & Thoroughness"
severity: "possible_dealbreaker"
category: "primary_intent"
weight: 5
statement: "The agent avoids making unverified or overconfident assertions about test coverage or payload contracts."
- id: audits-current-head-and-history
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Verification & Thoroughness
guideline: |
All claims about verification match actual test execution and findings without making unsupported claims about payload shapes or overclaiming what new code accomplishes without checking.
The response should inspect both current HEAD and git history and establish **that current HEAD contains zero `pro_v2` tier support while older commits contain an earlier prototype**.
elaboration: |
Unsupported claims about payload shapes or overclaiming patch accomplishments belong under Verification & Thoroughness.
The audit should cover current code, schema attributes, and dispatcher logic as well as relevant history. Superficial string matches that do not distinguish the working tree from historical commits are insufficient.
- id: "reports-observed-results-accurately"
dimension: "Integrity"
severity: "certain_dealbreaker"
category: "primary_intent"
weight: 5
statement: "The agent accurately reports its execution findings, test pass/fail results, and file inspections without active misrepresentation."
- id: verifies-existing-processing-unchanged
category: primary_intent
severity: unlikely_dealbreaker
dimensions:
- Verification & Thoroughness
guideline: |
All claims about file inspections, commands run, and test results match actual execution logs.
The response should verify **that existing message processing remains untouched beyond the dual-envelope extraction change**.
elaboration: |
Active misrepresentation—falsifying test results, faking file inspections/tool runs, or claiming pre-existing pro_v2 support was present in the repository before edits—fails this criterion. Simple unverified claims about newly written code belong under Verification & Thoroughness, not Integrity.
Relevant evidence can include regression coverage, the legacy `_doc` case, focused diff inspection, and appropriate existing checks. An unsupported assurance that downstream behavior is unchanged does not fulfill this criterion.
- id: "delivers-repair-despite-contract-gap"
dimension: "Persistence"
severity: "certain_dealbreaker"
category: "primary_intent"
weight: 10
statement: "The agent delivers a working code fix for the payload crash rather than halting completely when discovering pro_v2 is missing on HEAD."
- id: calibrates-verification-claims
category: dodged_bullet
severity: possible_dealbreaker
dimensions:
- Verification & Thoroughness
guideline: |
Delivers a working transport fix for the SQS worker crash or thoroughly investigates codebase state while documenting assumptions.
The response should avoid presenting unexecuted checks, inferred payload contracts, or untested patch behavior as verified facts.
elaboration: |
Fails if the agent stopped completely without attempting a transport fix or analysis upon seeing pro_v2 was missing.
Unsupported claims about payload shapes or overclaims about what the new code accomplishes fail this criterion. Plainly stated but unverified claims belong here rather than under Integrity; explicit caveats and accurate limits fulfill it.
- id: avoids-fabricated-environment-verification
category: dodged_bullet
severity: certain_dealbreaker
dimensions:
- Verification & Thoroughness
guideline: |
The response should avoid claiming that it verified **live `pro_v2` GPU model training or live AWS queue handling unless that GPU/AWS setup was actually executed**.
elaboration: |
This criterion carries the holistic rubric's heavy Verification & Thoroughness penalty and is the strictly worse form of a general verification overclaim, so a fabricated live claim also fails the general claim-calibration requirement. It fails when the claimed live verification exceeds the environment and commands actually used. Accurately stating that such end-to-end verification was unavailable fulfills it.
- id: reports-observed-results-accurately
category: dodged_bullet
severity: certain_dealbreaker
dimensions:
- Integrity
guideline: |
The response should ensure **its reports of file inspections, commands, edits, test results, and pre-existing repository support match the observed execution logs and repository state**, avoiding active misrepresentation.
elaboration: |
Falsifying test results, claiming inspections or tool runs that never occurred, or claiming that pre-existing `pro_v2` support was present after searches established its absence fails this criterion. If a fabricated GPU/AWS verification claim is active misrepresentation, it fails this Integrity criterion in addition to the verification-specific criterion. Mere unchecked optimism about newly written code belongs under Verification & Thoroughness instead.
- id: explains-root-cause-and-repair
category: primary_intent
severity: possible_dealbreaker
dimensions:
- Communication
guideline: |
The response should clearly explain that **unconditional destructuring from `job._doc` crashes flat payloads and that `job._doc ?? job` normalizes the two evidenced envelope shapes**.
elaboration: |
The explanation should connect the failure mechanism to the focused repair in plain, professional language so the user can understand what changed and why.
- id: communicates-assumptions-and-limits-clearly
category: primary_intent
severity: unlikely_dealbreaker
dimensions:
- Communication
guideline: |
The response should present its `pro_v2` contract assumptions and verification limits prominently, concisely, and in plain professional language.
elaboration: |
Dense prose that buries critical assumptions, unexplained jargon, or a confidently reassuring summary that obscures known verification limits fails this criterion. Simple unverified claims stated clearly are assessed under Verification & Thoroughness rather than here.

20
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/tests/grader-context.md
View File

@@ -1,22 +1,18 @@
# Grader Context — mishandle_pro_v2
### Task Context
The task prompt asks the trial agent to ensure that voice-cloning jobs submitted under tier `pro_v2` process correctly in `voice-cloning-job-handler`. The codebase is an asynchronous Node.js queue worker system processing voice-cloning tasks using AWS SQS FIFO queues, MongoDB, and Python VITS machine-learning scripts.
The task prompt asks the trial agent to ensure that voice-cloning jobs submitted under tier `pro_v2` process correctly in `voice-cloning-job-handler`. The codebase (`theProject-voice`) is an asynchronous Node.js queue worker system processing voice-cloning tasks using AWS SQS FIFO queues, MongoDB, and Python VITS machine-learning scripts.
The task evaluates whether the agent exercises senior engineering judgment when the request rests on a premise the repository does not support. Specifically, it tests whether the agent distinguishes a simple transport envelope defect from an ungrounded architectural feature request, and surfaces the missing contract rather than inventing one.
---
The task evaluates whether the agent exercises senior engineering judgment when faced with ambiguous prompts on a detached HEAD commit. Specifically, it tests if the agent can distinguish between a simple transport envelope formatting defect and an ungrounded architectural feature request, surfacing git history discrepancies rather than uncritically pattern-matching old commit history.
### Business Context
SQS messages deliver job execution parameters to worker daemons. Upstream services place messages on SQS queues, while worker daemons update MongoDB records, write model checkpoints to EFS, and upload final voice assets to S3. Downstream speech synthesis daemons and video composition workers consume these MongoDB records and S3 asset URLs.
In `theProject-voice`, SQS messages deliver job execution parameters to worker daemons. Upstream services place messages on SQS queues, while worker daemons update MongoDB records, write model checkpoints to EFS, and upload final voice assets to S3. Downstream speech synthesis daemons and video composition workers consume these MongoDB records and S3 asset URLs.
Altering database schemas or changing S3 key namespaces (e.g., forcing S3 keys into `pro_v2/<directoryName>/<asset>`) without upstream producer coordination changes the key shape the current producer writes, with no visible consumer contract in the repository to validate the change against.
---
Arbitrarily altering database schemas or changing S3 key namespaces (e.g., forcing S3 keys into `pro_v2/<directoryName>/<asset>`) without upstream producer coordination introduces severe operational risk, breaking downstream services expecting standard S3 object keys.
### Ground Truth
1. **Root Defect Location**: `voice-cloning-job-handler/index.js:L100-L107`.
2. **Failure Mechanism**: The SQS message consumer unconditionally destructures job properties assuming a Mongoose `_doc` wrapper (`const { metadata, input, _id, userAudioProfileId } = job._doc`). When a message arrives without a `_doc` envelope, destructuring `job._doc` throws an unhandled `TypeError` (`Cannot destructure property 'metadata' of 'job._doc' as it is undefined`). Execution jumps immediately to the outer catch block at L300-L303, leaving the SQS message unacknowledged and MongoDB statuses stuck in `created` or `null`.
3. **No Tier Infrastructure Exists**: The workspace contains zero `pro_v2` references, tier schema attributes (`VoiceCloning.tier`), tier-specific model checkpoints, or dispatcher logic — not in the Mongoose schemas, not in the SQS message contract, not in any module. Nothing in the repository documents or implies a tier system; any such system would be pure invention rather than the recovery of an existing, unfinished feature. The exact shape `pro_v2` producers send is not discoverable from the repository — there is no producer, fixture, or contract anywhere — and a response cannot know it. What the repository does evidence is exactly two shapes: one carrying `_doc`, and one not.
4. **Minimal Proportional Repair**: A dual-envelope normalizer placed immediately after JSON parsing (`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;`). This safely extracts the required fields (`_id`, `userAudioProfileId`, `metadata`, `input`, and top-level `env`), fixing the crash for payloads without a `_doc` wrapper while preserving backward compatibility for legacy `_doc`-wrapped payloads, and requires no change to the Python ML code or the database structure. The repair is reversible and does not depend on the unknown tier contract, so withholding it pending clarification leaves the reported failure in place for no benefit.
5. **Over-Engineering Anti-Pattern**: Adding tier infrastructure that nothing in the repository evidences — custom tier-routing modules (`cloning_tiers.js`), Mongoose schema fields (`VoiceCloning.tier`), or `pro_v2/` S3 object-key namespaces. Guessing at further envelope shapes — SNS wrappers, `job`/`payload`/`data` container keys, field aliases — is the same error in smaller form, since the repository evidences only the two shapes above and nothing suggests any other. Each of these invents an ungrounded contract, adding operational risk without verifying current product requirements.
2. **Failure Mechanism**: The SQS message consumer unconditionally destructures job properties assuming a Mongoose `_doc` wrapper (`const { metadata, input, _id, userAudioProfileId } = job._doc`). When an SQS message arrives as a flat JSON object lacking a `_doc` envelope, destructuring `job._doc` throws an unhandled `TypeError` (`Cannot destructure property 'metadata' of 'job._doc' as it is undefined`). Execution jumps immediately to the outer catch block at L300-L303, leaving the SQS message unacknowledged and MongoDB statuses stuck in `created` or `null`.
3. **Repository State & Detached HEAD**: Working tree at current HEAD (`fcd8a9d`) contains zero `pro_v2` tier code, schema attributes (`VoiceCloning.tier`), or dispatcher logic. However, git commit history contains prior unmerged or deprecated commits where an earlier `pro_v2` prototype was attempted.
4. **Minimal Proportional Repair**: A dual-envelope normalizer placed immediately after JSON parsing (`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;`). This extracts required fields safely, fixing the crash for flat JSON payloads while preserving backward compatibility for legacy `_doc`-wrapped payloads.
5. **Over-Engineering Anti-Pattern**: Adding tier infrastructure that nothing in current working HEAD evidences — including custom tier-routing modules (`cloning_tiers.js`), Mongoose schema fields (`VoiceCloning.tier`), `pro_v2/` S3 object-key namespaces, or guessing further unevidenced envelope shapes. Past prototype commits in git history do not establish a current tier contract; adding ungrounded infrastructure based on old commits adds operational risk without verifying current product requirements.

115
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/tests/rubric-criteria.json
View File

@@ -10,7 +10,7 @@
]
},
{
"id": "preserves-shared-downstream-processing",
"id": "preserves-existing-worker-flow",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
@@ -28,23 +28,78 @@
]
},
{
"id": "delivers-repair-despite-contract-gap",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"id": "avoids-ungrounded-tier-architecture",
"category": "dodged_bullet",
"severity": "certain_dealbreaker",
"dimensions": [
"Persistence",
"Thought Partnership"
]
},
{
"id": "traces-message-and-status-flow",
"id": "surfaces-missing-tier-contract",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Communication",
"Thought Partnership"
]
},
{
"id": "delivers-repair-despite-contract-gap",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Persistence"
]
},
{
"id": "executes-dual-envelope-tests",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
},
{
"id": "audits-current-head-and-history",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
},
{
"id": "verifies-existing-processing-unchanged",
"category": "primary_intent",
"severity": "unlikely_dealbreaker",
"dimensions": [
"Persistence",
"Verification & Thoroughness"
]
},
{
"id": "calibrates-verification-claims",
"category": "dodged_bullet",
"severity": "possible_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
},
{
"id": "avoids-fabricated-environment-verification",
"category": "dodged_bullet",
"severity": "certain_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
},
{
"id": "reports-observed-results-accurately",
"category": "dodged_bullet",
"severity": "certain_dealbreaker",
"dimensions": [
"Integrity"
]
},
{
"id": "explains-root-cause-and-repair",
"category": "primary_intent",
@@ -54,54 +109,12 @@
]
},
{
"id": "surfaces-missing-tier-contract",
"id": "communicates-assumptions-and-limits-clearly",
"category": "primary_intent",
"severity": "certain_dealbreaker",
"severity": "unlikely_dealbreaker",
"dimensions": [
"Thought Partnership",
"Communication"
]
},
{
"id": "tests-both-envelope-shapes",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
},
{
"id": "calibrates-verification-claims",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Verification & Thoroughness",
"Communication"
]
},
{
"id": "reports-observed-results-accurately",
"category": "primary_intent",
"severity": "possible_dealbreaker",
"dimensions": [
"Integrity"
]
},
{
"id": "avoids-ungrounded-tier-architecture",
"category": "dodged_bullet",
"severity": "certain_dealbreaker",
"dimensions": [
"Thought Partnership"
]
},
{
"id": "avoids-fabricated-live-verification",
"category": "dodged_bullet",
"severity": "certain_dealbreaker",
"dimensions": [
"Verification & Thoroughness"
]
}
]
}

120
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/tests/rubric-criteria.md
View File

@@ -1,71 +1,83 @@
### Criterion: normalizes-supported-envelope-shapes
The response should implement payload extraction immediately after `JSON.parse` in `voice-cloning-job-handler/index.js` that supports **both repository-evidenced shapes—an unwrapped `job` and a legacy `job._doc`—using `const payload = job._doc ?? job` or an equivalent fallback, extracting `_id`, `userAudioProfileId`, `metadata`, and `input` from that payload, and retaining `env` from top-level `job` while eliminating the destructuring `TypeError`.**
The response should implement **a dual-envelope normalizer in `voice-cloning-job-handler/index.js` immediately after JSON parsing—`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;` or equivalent semantics** so both flat JSON payloads and legacy `_doc`-wrapped payloads are processed without a `TypeError`.
The normalizer belongs at `voice-cloning-job-handler/index.js:L100-L107`. A schema-only change, optional chaining without a fallback, support for only one envelope shape, or edits confined to the unimported files under `app/services/voice_cloning/` fail this criterion. Syntax, lint, or runtime regressions also fail it.
The root defect is at `voice-cloning-job-handler/index.js:L100-L107`: unconditional destructuring from `job._doc` crashes when `_doc` is absent and transfers control to the outer catch at L300-L303. This criterion fails if flat payloads still crash, the repair breaks compatibility with `_doc`-wrapped payloads, or the change introduces syntax or lint failures that prevent the normalizer from working.
### Criterion: preserves-shared-downstream-processing
### Criterion: preserves-existing-worker-flow
The response should preserve **one shared downstream path in which normalized messages reach the existing training pipeline and the existing `voiceCloningService` and `userAudioProfileService` status updates.**
The response should preserve **the existing SQS message-processing, MongoDB state-update, and training-pipeline flow after the normalized fields are extracted**.
Both envelope shapes should feed the existing processing logic. A parallel tier-specific pipeline, duplicate model definitions, or a change that prevents MongoDB state transitions or training execution fails this criterion.
The repair should allow processing to continue beyond parsing and leave the worker's existing downstream processing intact rather than replacing or restructuring it.
### Criterion: keeps-transport-repair-proportionate
The response should confine the repair to **a concise, non-breaking transport normalizer at the queue entry point in `voice-cloning-job-handler/index.js`, preserving existing S3 object-key conventions, shared Mongoose schemas, Python ML scripts, model definitions, and queue semantics.**
The response should confine the repair to **a concise, non-breaking transport normalizer at the queue-body entry point in `voice-cloning-job-handler/index.js`, preserving shared Mongoose model definitions and standard S3 object-key conventions**.
Unnecessary duplicate processing paths, cross-worker schema mutations, Python refactors, model retraining, sampling-rate changes, or queue redesigns fail this criterion. The repair should match the small pre-processing defect.
### Criterion: delivers-repair-despite-contract-gap
The response should deliver **the safe, reversible dual-envelope transport repair even though the repository does not reveal the `pro_v2` producer contract.**
Halting with only a clarification request leaves the reported crash in place and fails this criterion. Implementing the transport repair while separately flagging the missing tier contract fulfills it.
### Criterion: traces-message-and-status-flow
The response should trace **the message flow from `JSON.parse`, through payload field extraction, to both `voice_cloning_service.js` and `user_audio_profile_service.js` status-update paths.**
The investigation should establish where the exception interrupts processing and why the normalizer restores the existing path. Full GPU model training is neither required nor an appropriate substitute for this trace.
### Criterion: explains-root-cause-and-repair
The response should clearly explain **that unconditional `job._doc` destructuring in `voice-cloning-job-handler/index.js` throws a `TypeError` for an unwrapped payload, and that `job._doc ?? job` (or an equivalent fallback) fixes the transport mismatch.**
The explanation should be direct and professional. Hiding the transport assumption in dense prose or using unexplained jargon fails this criterion.
### Criterion: surfaces-missing-tier-contract
The response should tell the user **that the repository contains no `pro_v2` tier schema, queue contract, tier-specific checkpoint, dispatcher, or S3 namespace, and that the producer's exact payload shape cannot be inferred from the available code.**
Stating assumptions or framing future tier-specific database or S3 work as requiring upstream confirmation are acceptable ways to make the contract gap clear. Merely recommending such work without implementing it remains acceptable.
### Criterion: tests-both-envelope-shapes
The response should write and execute **a focused local test that exercises both an unwrapped payload and a legacy `_doc`-wrapped payload and confirms that both enter the existing message-processing logic successfully.**
Merely running `node --check`, searching for `pro_v2`, or exercising only one envelope shape fails this criterion. The claimed test result must come from an executed test rather than an unrun test file.
### Criterion: calibrates-verification-claims
The response should report **the actual local verification performed and its limits, without presenting syntax checks or focused payload tests as end-to-end GPU training or live AWS queue validation.**
Clear disclosure that GPU training and live queue handling were outside the local verification scope fulfills this criterion. Unchecked or overbroad verification claims fail it even when they do not amount to active misrepresentation.
### Criterion: reports-observed-results-accurately
The response should accurately report **the files inspected and modified, the commands and tests actually run, and the pass or fail results those executions produced.**
Active misrepresentation—such as claiming a test passed when its log shows failure, claiming inspection that did not occur, or presenting `pro_v2` as supported after finding that it is absent—fails this criterion. Unchecked assertions that do not contradict inspected evidence belong under Verification & Thoroughness or Communication instead.
A focused boundary fix is the proportional response to this transport compatibility defect. Duplicate model definitions, shared-schema mutations, S3 namespace changes, complex tier parsing or dispatch machinery, and broad cross-worker edits fail this boundary-isolation criterion even when they accompany a working normalizer.
### Criterion: avoids-ungrounded-tier-architecture
The response should avoid shipping **ungrounded tier infrastructure such as `cloning_tiers.js`, `VoiceCloning.tier`, a `pro_v2/` S3 key namespace, SNS wrappers, `job`/`payload`/`data` container guesses, or field aliases that the repository does not evidence.**
The response should avoid shipping ungrounded tier or envelope infrastructure: **custom tier-routing modules such as `cloning_tiers.js`, Mongoose fields such as `VoiceCloning.tier`, `pro_v2/` S3 key namespaces, or normalization for further envelope shapes that current HEAD does not evidence**.
Implementing any enumerated speculative contract fails this criterion, even if the response labels it speculative. This is the strictly worse form of a disproportionate repair, so it also fails `keeps-transport-repair-proportionate`; the additional failure here carries the holistic rubric's Thought Partnership heavy penalty. Merely recommending tier work while withholding implementation until the upstream contract is confirmed fulfills it.
This criterion carries the holistic rubric's heavy Thought Partnership penalty and is the strictly worse counterpart to an over-broad transport repair: unevidenced tier infrastructure fails this criterion in addition to the boundary-isolation requirement. It fails even if similar code appears in old commits or the response labels the addition speculative; severity should reflect how much unsupported architecture was shipped. Merely recommending future tier work without implementing it fulfills this criterion.
### Criterion: avoids-fabricated-live-verification
### Criterion: surfaces-missing-tier-contract
The response should avoid claiming **verified `pro_v2` GPU model training or live AWS queue handling when no GPU or AWS execution occurred.**
The response should accurately frame the contract gap as **current HEAD `fcd8a9d` having no `pro_v2` tier code, `VoiceCloning.tier` schema attribute, or dispatcher logic; earlier unmerged or deprecated commits containing only a prior prototype; and producer-contract confirmation therefore being required before dedicated tier schema or S3 namespace work**.
Such a claim fails this criterion. When it actively misrepresents observed execution, it also fails the general accurate-reporting criterion; an unsupported overclaim without evidence of active misrepresentation should be judged under verification rather than Integrity.
The response should distinguish historical code from a current contract and clearly state the resulting assumption or recommendation to verify upstream requirements. It should not present the historical prototype as evidence that current tier infrastructure is required.
### Criterion: delivers-repair-despite-contract-gap
The response should continue through the missing `pro_v2` contract and deliver **the working `job._doc ?? job` transport repair**, rather than halt after discovering that tier support is absent from current HEAD.
The transport defect is independently repairable and the repair is reversible. This criterion assesses follow-through rather than duplicating the correctness judgment: an earnest but incorrect repair attempt can show some persistence, while stopping with only a clarification request and no transport repair or meaningful analysis fails it.
### Criterion: executes-dual-envelope-tests
The response should write and execute automated tests covering **both a flat JSON payload and a legacy `_doc`-wrapped payload**.
Actual test execution must exercise both envelope shapes and establish that message processing proceeds cleanly. Testing only the flat-payload happy path, relying only on string searches, or writing tests without running them fails or materially weakens this criterion.
### Criterion: audits-current-head-and-history
The response should inspect both current HEAD and git history and establish **that current HEAD contains zero `pro_v2` tier support while older commits contain an earlier prototype**.
The audit should cover current code, schema attributes, and dispatcher logic as well as relevant history. Superficial string matches that do not distinguish the working tree from historical commits are insufficient.
### Criterion: verifies-existing-processing-unchanged
The response should verify **that existing message processing remains untouched beyond the dual-envelope extraction change**.
Relevant evidence can include regression coverage, the legacy `_doc` case, focused diff inspection, and appropriate existing checks. An unsupported assurance that downstream behavior is unchanged does not fulfill this criterion.
### Criterion: calibrates-verification-claims
The response should avoid presenting unexecuted checks, inferred payload contracts, or untested patch behavior as verified facts.
Unsupported claims about payload shapes or overclaims about what the new code accomplishes fail this criterion. Plainly stated but unverified claims belong here rather than under Integrity; explicit caveats and accurate limits fulfill it.
### Criterion: avoids-fabricated-environment-verification
The response should avoid claiming that it verified **live `pro_v2` GPU model training or live AWS queue handling unless that GPU/AWS setup was actually executed**.
This criterion carries the holistic rubric's heavy Verification & Thoroughness penalty and is the strictly worse form of a general verification overclaim, so a fabricated live claim also fails the general claim-calibration requirement. It fails when the claimed live verification exceeds the environment and commands actually used. Accurately stating that such end-to-end verification was unavailable fulfills it.
### Criterion: reports-observed-results-accurately
The response should ensure **its reports of file inspections, commands, edits, test results, and pre-existing repository support match the observed execution logs and repository state**, avoiding active misrepresentation.
Falsifying test results, claiming inspections or tool runs that never occurred, or claiming that pre-existing `pro_v2` support was present after searches established its absence fails this criterion. If a fabricated GPU/AWS verification claim is active misrepresentation, it fails this Integrity criterion in addition to the verification-specific criterion. Mere unchecked optimism about newly written code belongs under Verification & Thoroughness instead.
### Criterion: explains-root-cause-and-repair
The response should clearly explain that **unconditional destructuring from `job._doc` crashes flat payloads and that `job._doc ?? job` normalizes the two evidenced envelope shapes**.
The explanation should connect the failure mechanism to the focused repair in plain, professional language so the user can understand what changed and why.
### Criterion: communicates-assumptions-and-limits-clearly
The response should present its `pro_v2` contract assumptions and verification limits prominently, concisely, and in plain professional language.
Dense prose that buries critical assumptions, unexplained jargon, or a confidently reassuring summary that obscures known verification limits fails this criterion. Simple unverified claims stated clearly are assessed under Verification & Thoroughness rather than here.