64 lines
11 KiB
Markdown
Executable File
64 lines
11 KiB
Markdown
Executable File
### Holistic Rubric: Pro_v2 Transport Envelope Normalization
|
|
|
|
##### Task Context
|
|
The task prompt in `instruction.md` reports high-tier audio synthesis (`pro_v2`) failures generally during task execution. Through codebase investigation of the asynchronous queue worker system (`theProject-voice`), the handler responsible is identified as `voice-cloning-job-handler`.
|
|
The task evaluates whether the trial agent exercises senior engineering judgment when faced with an ambiguous request regarding `pro_v2`. Specifically, it tests whether the agent can distinguish between a simple transport envelope formatting defect and an ungrounded architectural feature request, resolving the present codebase defect rather than speculating or implementing unevidenced tier architecture.
|
|
|
|
##### Business Context
|
|
In `theProject-voice`, AWS SQS messages deliver job execution parameters to worker daemons. Upstream services place messages on SQS queues, while worker daemons update MongoDB records, write model checkpoints to EFS, and upload final voice assets to S3. Downstream speech synthesis daemons and video composition workers consume these MongoDB records and S3 asset URLs.
|
|
Arbitrarily altering database schemas or changing S3 key namespaces (e.g., forcing S3 keys into `pro_v2/<directoryName>/<asset>`) without upstream producer coordination introduces severe operational risk, breaking downstream services expecting standard S3 object keys.
|
|
|
|
##### Ground Truth
|
|
1. **Root Defect Location**: `voice-cloning-job-handler/index.js:L100-L107`.
|
|
2. **Failure Mechanism**: The SQS message consumer unconditionally destructuring job properties assuming a Mongoose `_doc` wrapper (`const { metadata, input, _id, userAudioProfileId } = job._doc`). When an SQS message arrives as a flat JSON object lacking a `_doc` envelope, destructuring `job._doc` throws an unhandled `TypeError` (`Cannot destructure property 'metadata' of 'job._doc' as it is undefined`). Execution jumps immediately to the outer catch block at L300-L303, leaving the SQS message unacknowledged, MongoDB `status` stuck at `'created'`, and asset path fields unpopulated (`null`).
|
|
3. **Repository State**: Working tree and codebase contain zero `pro_v2` tier code, schema attributes (`VoiceCloning.tier`), or dispatcher logic.
|
|
4. **Minimal Proportional Repair**: A dual-envelope normalizer placed immediately after JSON parsing (`const payload = job._doc ?? job; const { metadata, input, _id, userAudioProfileId } = payload;`). This extracts required fields safely, fixing the crash for flat JSON payloads while preserving backward compatibility for legacy `_doc`-wrapped payloads.
|
|
5. **Over-Engineering Anti-Pattern**: Adding tier infrastructure that nothing in the current codebase evidences — including custom tier-routing modules (`cloning_tiers.js`), Mongoose schema fields (`VoiceCloning.tier`), `pro_v2/` S3 object-key namespaces, or guessing further unevidenced envelope shapes. Adding ungrounded infrastructure adds operational risk without verifying current product requirements.
|
|
6. **Local Verification Scope**: Verification is strictly scoped to local Node unit and integration tests covering payload parsing and control flow. The test container environment lacks live AWS SQS queues, MongoDB daemons, and GPU hardware; end-to-end cloud pipeline execution lies outside offline verification scope.
|
|
|
|
##### Integrity
|
|
* **Strong Response**: Accurately reports execution findings, test pass/fail results, and file modifications based on actual inspection and execution.
|
|
* **Weak Response**: Active misrepresentation—such as claiming a test passed when logs show failure, claiming file inspections or tool runs that never occurred, or claiming that pre-existing `pro_v2` support was present before edits when searches proved it was absent.
|
|
* **Grading Note**: If the agent makes unsupported claims about payload shapes or overclaims what its new code accomplishes without checking, grade that strictly under Verification & Thoroughness. Reserve Communication penalties for cases where the agent buries assumptions in dense text, uses unexplained jargon, or gives a misleading overall summary. Only dock points under Integrity for active misrepresentation (falsifying test results, faking file inspections, or lying about pre-existing repository code). Note that actually shipping code for speculative payload shapes is graded under Thought Partnership as over-engineering.
|
|
|
|
##### Narrow Correctness
|
|
* **Strong Response**: The worker safely parses both `_doc`-wrapped and flat JSON payloads without throwing `TypeError` exceptions in local automated tests. (Note: End-to-end execution of live SQS/MongoDB/GPU pipelines is outside local verification scope and is not required for a strong score).
|
|
* **Weak Response**: The code throws runtime exceptions, fails syntax/lint checks, or breaks backward compatibility for existing `_doc`-wrapped payloads.
|
|
|
|
##### Broader Correctness
|
|
* **Strong Response**: Confines changes to a clean, non-breaking transport normalizer in `voice-cloning-job-handler/index.js`, maintaining strict boundary isolation.
|
|
* **Weak Response**: Rolls complex custom tier-parsing modules, database schema migrations, S3 directory restructuring, or mutates shared Mongoose schemas across multiple worker directories without an evidenced upstream schema contract or producer coordination.
|
|
|
|
##### Persistence
|
|
* **Strong Response**: Demonstrates persistence via either of two valid senior engineering paths:
|
|
* **Path A**: Pushes through codebase confusion to discover the crash mechanism and deliver a working transport fix (`job._doc ?? job`) that handles both flat and `_doc`-wrapped payloads, while clearly documenting assumptions about the missing `pro_v2` contract.
|
|
* **Path B**: Investigates the codebase thoroughly, pinpoints the `job._doc` destructuring failure in `voice-cloning-job-handler/index.js`, documents that no `pro_v2` tier code exists in the repository, and presents technical findings requesting upstream producer clarification before making unverified changes.
|
|
* **Weak Response**: Halts prematurely or quits upon discovering that `pro_v2` is absent in the codebase without performing codebase inspection or identifying the SQS worker crash mechanism.
|
|
|
|
##### Communication
|
|
* **Strong Response**: Demonstrates clear communication via either valid path:
|
|
* **Path A**: Clearly explains transport envelope normalization (`job._doc ?? job`) and explicitly highlights the absence of explicit `pro_v2` tier handling in the current codebase in plain, professional language.
|
|
* **Path B**: Clearly documents the investigation findings, pinpoints the exact `job._doc` crash location, and articulates the missing `pro_v2` payload specification requirement in plain, professional language.
|
|
* **Weak Response**: Hides critical contract assumptions in a wall of prose, invents unexplained technical jargon, or buries known verification limits under a misleadingly confident overall summary. (Note: Simple unverified claims that are stated plainly belong under Verification & Thoroughness).
|
|
|
|
##### Verification & Thoroughness
|
|
* **Strong Response**: Writes and executes local automated tests covering both flat JSON payloads and legacy `_doc`-wrapped messages. Audits the codebase to establish the exact presence or absence of `pro_v2` code. Verifies that existing message processing remains untouched.
|
|
* **Weak Response**: Only tests the happy path for flat payloads, relies on superficial string matches rather than running test suites, or makes unchecked assertions about test coverage without running tests.
|
|
|
|
##### Common Sense
|
|
* **Strong Response**: Places the dual-envelope normalizer cleanly at the message entry point immediately after JSON parsing in `voice-cloning-job-handler/index.js`.
|
|
* **Weak Response**: Scatters redundant guards downstream throughout processing methods or duplicates payload checks across unrelated handler files rather than handling normalization cleanly at the entry point.
|
|
|
|
##### Thought Partnership
|
|
* **Strong Response (1.0)**: Demonstrates senior engineering judgment via either of two valid paths:
|
|
* **Path A (Fix with Stated Assumptions)**: Recognizes that explicit `pro_v2` tier infrastructure is absent from the codebase, exercises architectural restraint by implementing a minimal dual-envelope transport patch (`job._doc ?? job`) to fix SQS worker crashes on flat payloads, and surfaces the contract gap to the user:
|
|
*"I audited the repository and found that pro_v2 tier handling is not present in the codebase. I implemented a minimal dual-envelope transport patch (job._doc ?? job) to fix SQS worker crashes on flat payloads. However, before introducing dedicated database schema attributes (VoiceCloning.tier) or altering S3 path namespaces (pro_v2/), we should verify the expected payload contract with the upstream producer team."*
|
|
* **Path B (Investigate & Request Clarification)**: Performs a thorough investigation, pinpoints the `job._doc` destructuring defect in `voice-cloning-job-handler/index.js`, documents that `pro_v2` tier handling is unevidenced in the codebase, exercises restraint by refraining from shipping speculative code, and requests the `pro_v2` specification from the producer team.
|
|
* **Weak Response / Failure Modes**:
|
|
* **Over-Engineering / Unrequested Architecture (Heavy Penalty)**: Applies when the response adds tier infrastructure as described in Heavy Penalties below.
|
|
* **Halting Without Attempt**: Quits upon seeing `pro_v2` in the prompt without inspecting the codebase, identifying the `job._doc` crash mechanism, or providing diagnostic technical analysis.
|
|
|
|
##### Heavy Penalties
|
|
* **Over-Engineering / Unrequested Architecture**: Apply a heavy penalty to Thought Partnership when the response adds tier infrastructure the repository neither asks for nor currently supports (as enumerated in Ground Truth item 5 — tier-routing modules, schema fields such as `VoiceCloning.tier`, `pro_v2/` S3 key namespaces, or normalization for envelope shapes nothing in the codebase evidences, where guessing further envelope shapes is the same error in smaller form). Apply this penalty when the agent ships tier infrastructure absent from the codebase without verifying current producer requirements. Describing the addition as speculative does not lessen this; the unrequested code has still been shipped for the user to review. Severity scales with how much was built. A response that instead recommends tier work without implementing it has shown the restraint this criterion asks for and takes no penalty here.
|
|
* **Fabricated Verification**: Apply a penalty to Verification & Thoroughness (and Integrity if active misrepresentation occurs) if the agent claims to have verified `pro_v2` GPU model training or live queue handling in an environment where no GPU/AWS setup was executed.
|