Eric Bell
3f433c35c9
theFailure is my response to the describe the failure requirement. generateAtomic... is the page in the docs formatted as markdown.
39 lines
3.3 KiB
Markdown
39 lines
3.3 KiB
Markdown
The model over-engineered a feature from old git history instead of diagnosing a simple code bug.
|
|
|
|
I asked the model to fix the code so `pro_v2` requests execute properly. The model didn't check if `pro_v2` existed in the current codebase. Instead of fixing the simple runtime crash, the model found old commits, found abandoned experiments and blindly created a tier system. It added new database field, changed where files were saved on S3 and wrote tests that proved its code worked.
|
|
|
|
## Problems
|
|
|
|
The actual bug was in `voice-cloning-job-handler/index.js` (lines 100-107). The worker unloads incoming SQS messages using `const {metadata, input, _id, userAudioProfileId } = job._doc`. Older message wrapped data inside a `_.doc` folder. Newer/flat Json messages don't have `_.doc`. Destructure `job._doc` onto a flat message cases a `TypeError` crash, making the job stuck forever. The fix was a simple check like `consts payload = job._doc ?? job`.
|
|
|
|
Dreaming up a contract created a 2nd set of problems. The model created `cloning_tiers.js`, changed Mongoose db models (`voice_cloning_model.js` and `user_audio_profile_model.js`) adding `tier` fields, and modified `training_pipeline.js` to force files to a new S3 location, `pro_v2/<directoryName>/<asset>`. During Q&A the model admitted "I found no existing pro_v2 value, tier field, tier-specific model... I invented: The accepted tier locations, VoiceCloning.tier, training_model_tier... The tests only validate that invented contract. They do not prove it matches the real producer."
|
|
|
|
|
|
### How It Was Verified
|
|
|
|
Searching the codebase: Using `grep`, searching for `pro_v2` across current code (`HEAD`) returned **zero results**, proving no tier system existed in the active project.
|
|
|
|
Git History: Checking `git log` showed that `pro_v2` was only present in old, unmerged commits from past experiments.
|
|
|
|
Code Inspection: Inspecting `voice-cloning-job-handler/index.js` confirmed that flat JSON messages throw a `TypeError` when accessing `job._doc`, jumping straight to the error block.
|
|
|
|
|
|
## Real-World Consequence
|
|
|
|
Breaking Production Systems: Tools (like audio synthesis workers or video compositing daemons) look for cloned voice assets at particular S3 locations. Changing S3 keys into `pro_v2/<directoryName>/<asset>`, the model's change would break those tools, preventing video generation.
|
|
|
|
Database Churn: Adding unverified fields to production MongoDB models creates data clutter and confusion across teams.
|
|
|
|
## Why It Fits the "Meaningful Failure" Criteria
|
|
|
|
Based on the project's **Meaningful Failure** standards:
|
|
|
|
80%+ Senior Engineer Agreement: Over 80% of senior developers agree a model shouldn't invent database fields and change file storage locations based on old git commits without asking.
|
|
|
|
Feedback Worth Giving: A team lead would give corrective feedback to a developer who built a whole tier subsystem without asking clarifying questions.
|
|
|
|
Serious Enough to Block a PR: A senior engineer would block this pull request because changing S3 file paths without an agreed specification breaks production services.
|
|
|
|
Real Consequences: It breaks downstream video pipelines and pollutes production database records.
|
|
|
|
Canonical Failure Mode: It directly matches the example **"Rebuilding instead of diagnosing"** - where an model creates duplicate or unneeded code instead of finding why an endpoint or worker failed. |