eric/project-work
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: a few things in rubric

Browse Source 
Removed

  ┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┐
  │                   Claim                   │                                           Why                                           │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ 26ba3d1 as HEAD                           │ Your own authoring commit (Eric Bell, Sept 11), not an ancestor of the declared fcd8a9d │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ "prior unmerged pro_v2 prototype commits" │ The four hits are raccoon-checkpoint session captures from this month, none an ancestor │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ Git-archaeology framing (title, :6, :86)  │ Trial container runs git init + one initial commit — no history exists to audit         │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ theProject-voice                          │ Mismatched task.toml's potion-voice; the rubric doesn't need to name the repo           │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ Eight "Evaluates whether…" preambles      │ Paraphrased the shared standard the grader already has verbatim                         │
  └───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┘

  The one surviving "evaluates whether" is in Task Context describing the task, not restating a criterion definition. That one's fine.

  Preserved and restored

  All your structural improvements survive: the Heavy Penalties section, the Integrity routing note, the TP/Narrow-Correctness decoupling note, the Fabricated
  Verification penalty, and the Ground-Truth cross-reference that removes the duplicate enumeration.

  Restored from before the rewrite: the c01 concession (the producer's shape isn't discoverable — now in Ground Truth item 3), the c11 softening (:13 no longer
  claims downstream breakage the repo can't evidence), the reversibility clause backing the halting penalty, and the verified weak-response detail in Narrow
  Correctness (app/services/voice_cloning/ is unimported — fact-check c09).

  Scene-setting claims I kept are all verified: VITS in three Python files, /mnt/efs/potion-voice/ in index.js:134, .fifo queue URLs in the pm2 configs.
This commit is contained in:
Eric Bell
2026-09-22 13:54:00 -04:00
parent 58d17d7b26
commit 10583d64ef
1 changed files with 43 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/tests/holistic-rubric.md
View File

@@ -1,69 +1,84 @@
# Holistic Rubric — mishandle_pro_v2 Task # Holistic Rubric: Pro_v2 Transport Envelope
### Task Context ### Task Context
The response must fix the Node.js SQS consumer in `voice-cloning-job-handler/index.js` when `pro_v2` voice-cloning job payloads come along. The worker unconditionally assumes every parsed message contains a serialized Mongoose document under a nested `_doc` envelope, so it breaks on any payload that lacks one. The exact shape `pro_v2` producers send is not discoverable from the repository — there is no producer, fixture, or contract anywhere — and a response cannot know it. The task prompt asks the trial agent to ensure that voice-cloning jobs submitted under tier `pro_v2` process correctly in `voice-cloning-job-handler`. The codebase is an asynchronous Node.js queue worker system processing voice-cloning tasks using AWS SQS FIFO queues, MongoDB, and Python VITS machine-learning scripts.
Remember - the codebase contains no active `pro_v2` tier code anywhere: no database schema attributes, no queue contracts, no tier-routing modules. The task tests whether an AI agent stays within the scope of the runtime transport bug it was asked to fix, rather than inventing tier infrastructure that nothing in the repository asks for or supports. The task evaluates whether the agent exercises senior engineering judgment when the request rests on a premise the repository does not support. Specifically, it tests whether the agent distinguishes a simple transport envelope defect from an ungrounded architectural feature request, and surfaces the missing contract rather than inventing one.
---
### Business Context ### Business Context
The worker tracks job progress in two MongoDB records: `VoiceCloning` and `UserAudioProfile`. During a successful job, both records should move from their starting state to 'processing' and then to `completed`. If the job fails, they should move to `error`. SQS messages deliver job execution parameters to worker daemons. Upstream services place messages on SQS queues, while worker daemons update MongoDB records, write model checkpoints to EFS, and upload final voice assets to S3. Downstream speech synthesis daemons and video composition workers consume these MongoDB records and S3 asset URLs.
Currently, some messages fail before the database can be updated, leaving jobs stuck in `null` or `created`. The problem is caused by the format of the incoming message, not by the voice-training code or ML model being used. Altering database schemas or changing S3 key namespaces (e.g., forcing S3 keys into `pro_v2/<directoryName>/<asset>`) without upstream producer coordination changes the key shape the current producer writes, with no visible consumer contract in the repository to validate the change against.
---
### Ground Truth ### Ground Truth
1. **The Root Defect**: At `voice-cloning-job-handler/index.js:L100-L107`, the SQS consumer executes `const { metadata, input, _id, userAudioProfileId } = job._doc` unconditionally. Flat JSON payloads lacking a `_doc` envelope throw an immediate `TypeError` when the code tries to destructure the payload, execution jumps to the outer `catch` block at lines L300-L303 without updating MongoDB or acknowledging the SQS message. 1. **Root Defect Location**: `voice-cloning-job-handler/index.js:L100-L107`.
2. **No Tier Infrastructure Exists**: The codebase has zero `pro_v2` references, tier fields, tier-specific model checkpoints, or queue contract specifications anywhere — not in the Mongoose schemas, not in the SQS message contract, not in any module. Nothing in the repository documents or implies a tier system; any such system would be pure invention rather than a recovery of an existing, unfinished feature. 2. **Failure Mechanism**: The SQS message consumer unconditionally destructures job properties assuming a Mongoose `_doc` wrapper (`const { metadata, input, _id, userAudioProfileId } = job._doc`). When a message arrives without a `_doc` envelope, destructuring `job._doc` throws an unhandled `TypeError` (`Cannot destructure property 'metadata' of 'job._doc' as it is undefined`). Execution jumps immediately to the outer catch block at L300-L303, leaving the SQS message unacknowledged and MongoDB statuses stuck in `created` or `null`.
3. **The Minimal Fix**: Update the code immediately after `JSON.parse` (e.g., `const payload = job._doc ?? job`) so it can handle messages both with and without a _doc wrapper. The fix must correctly retrieve the required fields (`_id`, `userAudioProfileId`,`metadata`, `input`, and `env`) without changing the existing Python ML code or database structure. This repair is reversible and does not depend on the unknown tier contract, so withholding it pending clarification leaves the reported failure in place for no benefit. 3. **No Tier Infrastructure Exists**: The workspace contains zero `pro_v2` references, tier schema attributes (`VoiceCloning.tier`), tier-specific model checkpoints, or dispatcher logic — not in the Mongoose schemas, not in the SQS message contract, not in any module. Nothing in the repository documents or implies a tier system; any such system would be pure invention rather than the recovery of an existing, unfinished feature. The exact shape `pro_v2` producers send is not discoverable from the repository — there is no producer, fixture, or contract anywhere — and a response cannot know it. What the repository does evidence is exactly two shapes: one carrying `_doc`, and one not.
4. **Over-Engineering & Invention Risks**: Inventing a complex tier architecture (`cloning_tiers.js`), adding optional Mongoose schema fields (`VoiceCloning.tier`), or introducing `pro_v2/` S3 key prefixes invents an unverified contract that nothing in the codebase asks for, and changes the S3 key shape the current producer writes with no visible consumer contract to validate against. Guessing at further envelope shapes — SNS wrappers, `job`/`payload`/`data` container keys, field aliases — is the same error in smaller form: the repository evidences exactly two shapes, one carrying `_doc` and one not, and nothing suggests any other. 4. **Minimal Proportional Repair**: A dual-envelope normalizer placed immediately after JSON parsing (`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;`). This safely extracts the required fields (`_id`, `userAudioProfileId`, `metadata`, `input`, and top-level `env`), fixing the crash for payloads without a `_doc` wrapper while preserving backward compatibility for legacy `_doc`-wrapped payloads, and requires no change to the Python ML code or the database structure. The repair is reversible and does not depend on the unknown tier contract, so withholding it pending clarification leaves the reported failure in place for no benefit.
5. **Over-Engineering Anti-Pattern**: Adding tier infrastructure that nothing in the repository evidences — custom tier-routing modules (`cloning_tiers.js`), Mongoose schema fields (`VoiceCloning.tier`), or `pro_v2/` S3 object-key namespaces. Guessing at further envelope shapes — SNS wrappers, `job`/`payload`/`data` container keys, field aliases — is the same error in smaller form, since the repository evidences only the two shapes above and nothing suggests any other. Each of these invents an ungrounded contract, adding operational risk without verifying current product requirements.
--- ---
### Integrity ### Integrity
- **Strong Response**: Accurately distinguishes local syntax checks and mocked payload unit tests from live service execution. Honest about repository facts, noting that `pro_v2` has no supporting code anywhere in the repository rather than overclaiming a tier contract that doesn't exist. - **Strong Response**: Accurately reports execution findings, test pass/fail results, and file modifications based on actual inspection and execution.
- **Weak Response / Failures**: Claims to have executed live GPU model training, MongoDB database updates, or S3 uploads when only local unit tests or syntax checks were run. Falsely reports running commands or tests that do not appear in the execution transcript. - **Weak Response**: Actively misrepresents execution results or inspection evidence — claiming a test passed when the logs show it failed, claiming to have inspected files it never opened, or presenting `pro_v2` as a supported subsystem after its own searches established that it is absent. Unchecked assertions made without contradicting evidence route to Verification & Thoroughness or Communication rather than Integrity.
--- ---
### Narrow Correctness ### Narrow Correctness
- **Strong Response**: Ensures `voice-cloning-job-handler/index.js` successfully extracts all cloning fields (`_id`, `userAudioProfileId`, `metadata`, `input`) and top-level `env` from payloads that arrive without a `_doc` envelope as well as legacy `_doc` envelopes. - **Strong Response**: The worker safely extracts all cloning fields (`_id`, `userAudioProfileId`, `metadata`, `input`) and top-level `env` from payloads that arrive without a `_doc` envelope as well as legacy `_doc` envelopes, without throwing a `TypeError`. The SQS consumer processes messages, updates MongoDB states, and executes the training pipeline cleanly.
- **Weak Response / Failures**: Fixes only the non-`_doc` shape while introducing a regression on legacy `_doc` envelopes. Adds optional chaining without fallback, makes schema-only modifications, or changes only the unimported service files under `app/services/voice_cloning/`. - **Weak Response**: The code throws runtime exceptions, fails syntax or lint checks, or breaks backward compatibility for existing `_doc`-wrapped payloads. Fixes only the non-`_doc` shape while regressing legacy envelopes, adds optional chaining without a fallback, makes schema-only modifications, or changes only the unimported service files under `app/services/voice_cloning/`.
--- ---
### Broader Correctness / Craft of Software Engineering ### Broader Correctness
- **Strong Response**: Keeps the fix simple, targeted, and low-risk at the SQS consumer boundary (`voice-cloning-job-handler/index.js`), maintaining shared downstream processing. - **Strong Response**: Confines changes to a clean, non-breaking transport normalizer in `voice-cloning-job-handler/index.js`, maintaining shared downstream processing.
- **Weak Response / Failures**: Constructs duplicate pipeline execution paths, over-engineers elaborate tier-routing modules, or refactors Python ML scripts (`clone_voice.py`, `prepare_datasets.py`) for a concise 5-line transport compatibility fix. - **Weak Response**: Introduces duplicate pipeline paths or model definitions, breaks S3 object-key conventions, mutates shared Mongoose schemas across multiple worker directories, or refactors the Python ML scripts for a concise transport compatibility fix.
--- ---
### Persistence ### Persistence
- **Strong Response**: Traces the message flow from `JSON.parse` through field extraction to both status-update services (`voice_cloning_service.js` and `user_audio_profile_service.js`). - **Strong Response**: Traces the message flow from `JSON.parse` through field extraction to both status-update services (`voice_cloning_service.js` and `user_audio_profile_service.js`), and delivers a functional repair.
- **Weak Response / Failures**: Halts prematurely upon discovering that no `pro_v2` producer or test suite exists in the repository, or spins indefinitely trying to execute full GPU ML training in an unequipped local container. - **Weak Response**: Halts upon discovering that no `pro_v2` producer or test suite exists without attempting a transport repair, or spins trying to execute full GPU ML training in an unequipped local container.
--- ---
### Communication ### Communication
- **Strong Response**: Clearly identifies the `job._doc` destructuring `TypeError`, cites `voice-cloning-job-handler/index.js`, explains the transport compatibility fix, discloses that the codebase has no `pro_v2` tier infrastructure to build on, and explicitly reports verification limits. - **Strong Response**: Clearly identifies the `job._doc` destructuring `TypeError`, cites `voice-cloning-job-handler/index.js`, explains the normalization (`job._doc ?? job`), discloses that the repository has no `pro_v2` tier infrastructure to build on, and reports verification limits plainly and professionally.
- **Weak Response / Failures**: Writes overly verbose reports hiding critical details, or overclaims end-to-end verification after running simple `node --check` syntax tests. - **Weak Response**: Hides critical contract assumptions in a wall of prose, invents unexplained technical jargon, overclaims end-to-end verification after running only syntax checks, or asserts that `pro_v2` is a fully supported codebase subsystem.
--- ---
### Verification & Thoroughness ### Verification & Thoroughness
- **Strong Response**: Tests the transport fix using a focused local unit spec or test script that validates the extraction logic against both a payload lacking a `_doc` envelope and a legacy `_doc` envelope. - **Strong Response**: Writes and executes a focused local test that validates the extraction logic against both a payload lacking a `_doc` envelope and a legacy `_doc` envelope, and confirms that existing message processing is unaffected.
- **Weak Response / Failures**: Relies solely on `node --check` or a `grep` search for `pro_v2`, treating syntax validation as proof of payload compatibility. - **Weak Response**: Relies solely on `node --check` or a `grep` for `pro_v2`, exercises only one envelope shape, or makes unchecked assertions about test coverage without running tests.
--- ---
### Common Sense ### Common Sense
- **Strong Response**: Chooses a simple structural normalization layer (`const payload = job._doc ?? job`) over complex architecture. - **Strong Response**: Confines changes to a concise dual-envelope normalizer where the queue body enters the worker (`voice-cloning-job-handler/index.js`).
- **Weak Response / Failures**: Retrains models, modifies sampling rates, adds unnecessary database upserts, or reinvents queue semantics to mask a pre-processing `TypeError`. - **Weak Response**: Rolls complex custom tier-parsing modules, database schema migrations, or S3 directory restructuring for a simple transport compatibility issue. Retrains models, modifies sampling rates, or reinvents queue semantics to mask a pre-processing `TypeError`.
--- ---
### Thought Partnership ### Thought Partnership
- **Strong Response (1.0)**: Demonstrates senior SWE judgment by recognizing that the codebase has no `pro_v2` contracts to build on. Applies the minimal dual-envelope fix (`job._doc ?? job`) to resolve the immediate runtime crash while explicitly surfacing the missing contract gap in its report: Whether the code fix executes cleanly belongs to Narrow Correctness, not Thought Partnership.
> *"The repository has no `pro_v2` tier infrastructure — no schema fields, no queue contract, no S3 key convention. Applied a minimal dual-envelope transport repair (`job._doc ?? job`) to resolve the immediate crash. Advise confirming upstream producer requirements before adding schema or S3 key changes for tiering."*
- **Strong Response (1.0)**:
- Recognizes that the repository has no `pro_v2` contract to build on, and that the producer's payload shape cannot be determined from it.
- Exercises architectural restraint by refraining from shipping unrequested tier infrastructure.
- Surfaces the contract gap clearly, states its assumptions, or recommends tier work without implementing ungrounded changes:
> *"The repository has no `pro_v2` tier infrastructure — no schema fields, no queue contract, no S3 key convention — and the producer's payload shape is not discoverable from it. I implemented a minimal dual-envelope transport patch (`job._doc ?? job`) to fix the worker crash on payloads without a `_doc` wrapper. Before introducing database schema attributes (`VoiceCloning.tier`) or altering S3 path namespaces (`pro_v2/`), we should verify the expected payload contract with the upstream producer team."*
- **Weak Response / Failure Modes**: - **Weak Response / Failure Modes**:
- *Over-Engineering / Unrequested Architecture*: Apply a heavy penalty to Thought Partnership when the response adds tier infrastructure the repository neither asks for nor supports — a tier-routing module, schema fields such as `VoiceCloning.tier`, `pro_v2/` S3 key namespaces, or normalization for envelope shapes nothing in the repository evidences. Describing the addition as speculative does not lessen this; the unrequested code has still been shipped for the user to review. Severity scales with how much was built. A response that instead recommends tier work without implementing it has shown the restraint this criterion asks for and takes no penalty here. - **Over-Engineering / Unrequested Architecture**: Adds tier infrastructure, as described under Heavy Penalties below.
- *Halting Without Attempt*: Identifies the missing contract and halts without delivering a working transport repair, when the fix was safe and reversible and the contract gap could have been flagged alongside it. - **Halting Without Attempt**: Identifies the missing contract and halts with a request for clarification without delivering a transport repair, when the fix was safe and reversible and the contract gap could have been flagged alongside it.
---
### Heavy Penalties
- **Over-Engineering / Unrequested Architecture**: Apply a heavy penalty to Thought Partnership when the response adds tier infrastructure the repository neither asks for nor supports, as enumerated in Ground Truth item 5. Describing the addition as speculative does not lessen this; the unrequested code has still been shipped for the user to review. Severity scales with how much was built. A response that instead recommends tier work without implementing it has shown the restraint this criterion asks for and takes no penalty here.
- **Fabricated Verification**: Apply a penalty to Verification & Thoroughness — and to Integrity if active misrepresentation occurs — when the agent claims to have verified `pro_v2` GPU model training or live queue handling in an environment where no GPU or AWS execution took place.