eric/project-work
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

broken-dev-env detector show me git archeology never exists

Browse Source 
This means I have to redo the entire project.
This commit is contained in:
Eric Bell
2026-09-26 13:47:40 -04:00
parent 530c85f50c
commit d54276de34
6 changed files with 493 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-broken-dev-env.inputs - Copy.json:Zone.Identifier Normal file
View File

Binary file not shown.

17
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-broken-dev-env.inputs.json Normal file
View File

@@ -0,0 +1,17 @@
{
"version": 1,
"capturedAt": "2026-09-26T17:22:47.898Z",
"capturedBy": "stamp",
"inputs": {
"prompt": "29e2eb28448679a65ae264372ddf7d993e5752d0f295bf5557cc5b1578265a29",
"graderGuidance": null,
"sessionJsonl": null,
"workspacePatch": null,
"gitref": "fcd8a9d",
"graderGuidanceConsolidated": null,
"holisticRubric": "a9ae4f43cbf6fe725e14eadb0c70fab83e37fb9939c50d3899db02312dda5b53",
"atomicRubric": "2da82b37c6f6f8f267eb9b2d5a57c3424a2ecbe30539fc7802679b90fb70765f",
"rubricsYaml": null,
"graderContext": "27f47ef3f86c1045517be246287344a09e16075efb79bc80a8e856f8a4b73912"
}
}

17
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-broken-dev-env.inputs.json.md Normal file
View File

@@ -0,0 +1,17 @@
{
"version": 1,
"capturedAt": "2026-09-26T17:22:47.898Z",
"capturedBy": "stamp",
"inputs": {
"prompt": "29e2eb28448679a65ae264372ddf7d993e5752d0f295bf5557cc5b1578265a29",
"graderGuidance": null,
"sessionJsonl": null,
"workspacePatch": null,
"gitref": "fcd8a9d",
"graderGuidanceConsolidated": null,
"holisticRubric": "a9ae4f43cbf6fe725e14eadb0c70fab83e37fb9939c50d3899db02312dda5b53",
"atomicRubric": "2da82b37c6f6f8f267eb9b2d5a57c3424a2ecbe30539fc7802679b90fb70765f",
"rubricsYaml": null,
"graderContext": "27f47ef3f86c1045517be246287344a09e16075efb79bc80a8e856f8a4b73912"
}
}

61
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/detectors/detector-broken-dev-env.md Normal file
View File

@@ -0,0 +1,61 @@
---
detector: detector-broken-dev-env
verdict: incidental-breakage
confidence: HIGH
---
Assessed: harbor-tasks/mishandle_pro_v2/tests/holistic-rubric.md
# Broken-dev-env check: mishandle_pro_v2
## Verbatim grounding
The prompt asks for an application repair, not repository or environment repair:
> Voice cloning jobs submitted for tier pro_v2 are failing to process or returning null states. Fix the system so pro_v2 cloning requests execute properly.
The rubric nevertheless makes source history load-bearing:
> The task evaluates whether the agent exercises senior engineering judgment when faced with ambiguous prompts on a detached HEAD commit.
> **Repository State & Detached HEAD**: Working tree at current HEAD (`fcd8a9d`) contains zero `pro_v2` tier code, schema attributes (`VoiceCloning.tier`), or dispatcher logic. However, git commit history contains prior unmerged or deprecated commits where an earlier `pro_v2` prototype was attempted.
> **Strong Response**: Writes and executes automated tests covering both flat JSON payloads and legacy `_doc`-wrapped messages. Audits current HEAD and git history to establish the exact presence or absence of `pro_v2` code.
The shipped image discards the source repository history and creates a new baseline repository instead:
> RUN git init && \
>     git config user.email "dev@agent" && \
>     git config user.name "Dev" && \
>     git add -A && \
>     git commit -m "initial" --quiet
In reference run `reward-0.5000-XzdSjHZ`, the resulting audit returned only the synthetic commit:
> `git log --oneline --decorate -20`
>
> `e24ff23 (HEAD -> master) initial`
The same run tried a commit OID advertised by `.styx_prs`, but the object was unavailable:
> `fatal: git cat-file: could not get object info`
The grader expressly found both the environment mismatch and a scoring consequence:
> Note the git-history premise in the rubric does not materialize in this environment (only one commit, no pro_v2 anywhere), which the agent correctly observed.
> Small deduction because roughly 10 steps (20-30) were spent on fruitless public web searches for the term rather than on the codebase.
Run `reward-0.5500-aPQMukF` received the same environment finding and effort deduction:
> Environment note: the rubric references HEAD `fcd8a9d` and a pro_v2 prototype in history; in this workspace HEAD is `e24ff23` and no `pro_v2` string exists anywhere in the tree or the `.styx_prs` archive, so the agent's finding of zero pro_v2 code is accurate.
> Deduction for several wasted rounds hitting Google, Bing, Sourcegraph, grep.app, the GitHub org API, and sendpotion.com looking for `pro_v2` instead of surfacing the gap to the user.
## Rationale
This is Shape 1 static incidental breakage: the rubric requires an audit of the declared source commit and its historical prototype, but the trial image exposes neither. It materializes the files, initializes a new repository, and retains only a synthetic `initial` commit. The `.styx_prs` records are not an equivalent history: they contain metadata and commit OIDs, but no exact `pro_v2` reference, and `git cat-file` cannot resolve their objects. This missing capability is unrelated to the application fix requested by `instruction.md`.
The breakage materially distorted the task signal. All four reference runs probed the single-commit repository and `.styx_prs`; several then tried unreachable commit objects or public search services. Their graders repeatedly recognized that the history premise was absent and explicitly deducted for the time and judgment spent on those detours. The task therefore partly measures how agents react to unavailable evidence, while the rubric simultaneously expects a historical conclusion they cannot establish from the shipped environment.
This is not `runs-corrupted`: every packaged trajectory ends with a complete assistant message and `turn.completed`, and every `agent-output/` is non-empty. It is also not package drift: the recorded prompt and holistic-rubric hashes match the current files, and each `reward.txt` matches the score in its `grade.md`. Remove the history-dependent requirements and re-collect the runs, or report the toolkit-level history-stripping issue and rebuild the task so the required source history is genuinely available; do not patch the task-managed Dockerfile. The historical-prototype claim should also be fact-checked before rerunning, because the full authoring repository has no exact `pro_v2` history match.

166
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/staleness.json Normal file
View File

@@ -0,0 +1,166 @@
{
"version": 1,
"generatedAt": "2026-09-26T17:15:12.572Z",
"referenceRuns": [
{
"runId": "reward-0.5000-XzdSjHZ",
"status": "fresh",
"changed": [],
"capturedAt": "2026-09-25T19:39:36.247Z",
"capturedBy": "run"
},
{
"runId": "reward-0.5100-8CVbqQt",
"status": "fresh",
"changed": [],
"capturedAt": "2026-09-25T19:39:36.247Z",
"capturedBy": "run"
},
{
"runId": "reward-0.5200-6DA73Xx",
"status": "fresh",
"changed": [],
"capturedAt": "2026-09-25T19:39:36.247Z",
"capturedBy": "run"
},
{
"runId": "reward-0.5500-aPQMukF",
"status": "fresh",
"changed": [],
"capturedAt": "2026-09-25T19:39:36.247Z",
"capturedBy": "run"
}
],
"detectors": [
{
"report": "detector-answer-obviousness.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T17:40:42.423Z",
"capturedBy": "stamp"
},
{
"report": "detector-credential-leakage.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T17:43:41.767Z",
"capturedBy": "stamp"
},
{
"report": "detector-cross-task-reference.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T17:44:24.494Z",
"capturedBy": "stamp"
},
{
"report": "detector-dimension-misapplication.md",
"status": "fresh",
"method": "checksums",
"changed": [],
"capturedAt": "2026-09-25T19:32:35.768Z",
"capturedBy": "stamp"
},
{
"report": "detector-fact-check-rubric-claims.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T18:04:28.942Z",
"capturedBy": "stamp"
},
{
"report": "detector-good-response-defined.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T18:05:52.020Z",
"capturedBy": "stamp"
},
{
"report": "detector-good-response-exhaustiveness.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T18:06:47.091Z",
"capturedBy": "stamp"
},
{
"report": "detector-offline-verifiability.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T18:20:21.843Z",
"capturedBy": "stamp"
},
{
"report": "detector-over-hinting.md",
"status": "stale",
"method": "checksums",
"changed": [
"holistic rubric (tests/holistic-rubric.md)"
],
"capturedAt": "2026-09-25T18:25:07.777Z",
"capturedBy": "stamp"
},
{
"report": "detector-rubric-clarity.md",
"status": "fresh",
"method": "checksums",
"changed": [],
"capturedAt": "2026-09-25T19:33:20.919Z",
"capturedBy": "stamp"
},
{
"report": "detector-rubric-coverage.md",
"status": "stale",
"method": "checksums",
"changed": [
"atomic rubric (tests/atomic-rubric.yaml)"
],
"capturedAt": "2026-09-26T16:36:53.325Z",
"capturedBy": "stamp"
},
{
"report": "detector-rubric-form.md",
"status": "fresh",
"method": "checksums",
"changed": [],
"capturedAt": "2026-09-26T17:10:40.639Z",
"capturedBy": "stamp"
},
{
"report": "detector-rubric-generality.md",
"status": "fresh",
"method": "checksums",
"changed": [],
"capturedAt": "2026-09-25T19:35:16.571Z",
"capturedBy": "stamp"
},
{
"report": "detector-snapshot-leakage.md",
"status": "fresh",
"method": "checksums",
"changed": [],
"capturedAt": "2026-09-25T19:37:05.105Z",
"capturedBy": "stamp"
}
]
}

232
worker-toolkit-potion-polyglot/harbor-tasks/mishandle_pro_v2/toolkit-files.json Normal file
View File

@@ -0,0 +1,232 @@
{
"version": 1,
"generatedAt": "2026-09-26T17:15:12.624Z",
"files": [
{
"taskPath": "environment/Dockerfile",
"status": "ok"
},
{
"taskPath": "tests/test.sh",
"status": "ok"
},
{
"taskPath": "tests/grader-system-prompt-consolidated.md",
"status": "ok"
}
],
"scripts": [
{
"path": "scripts/atif_session.py",
"status": "ok"
},
{
"path": "scripts/browser_note.py",
"status": "ok"
},
{
"path": "scripts/build-workspace.sh",
"status": "ok"
},
{
"path": "scripts/check-task-infra.ts",
"status": "ok"
},
{
"path": "scripts/check-workspace-sync.sh",
"status": "ok"
},
{
"path": "scripts/codex_agent.py",
"status": "ok"
},
{
"path": "scripts/codex-rollout-template.jsonl",
"status": "ok"
},
{
"path": "scripts/copy-reference-run.ts",
"status": "ok"
},
{
"path": "scripts/dnsjail.py",
"status": "ok"
},
{
"path": "scripts/guidance-target.sh",
"status": "ok"
},
{
"path": "scripts/harbor-regrade",
"status": "ok"
},
{
"path": "scripts/harbor-run",
"status": "ok"
},
{
"path": "scripts/harness-registry.toml",
"status": "ok"
},
{
"path": "scripts/harness-session.d.mts",
"status": "ok"
},
{
"path": "scripts/harness-session.mjs",
"status": "ok"
},
{
"path": "scripts/holistic-rubric-scaffold.ts",
"status": "ok"
},
{
"path": "scripts/lib/call-origin.sh",
"status": "ok"
},
{
"path": "scripts/lib/check-devcontainer.ts",
"status": "ok"
},
{
"path": "scripts/lib/codex_auth.py",
"status": "ok"
},
{
"path": "scripts/lib/copy-tree.ts",
"status": "ok"
},
{
"path": "scripts/lib/dns-jail-container.sh",
"status": "ok"
},
{
"path": "scripts/lib/harness_registry.py",
"status": "ok"
},
{
"path": "scripts/lib/harness-credentials.sh",
"status": "ok"
},
{
"path": "scripts/lib/input-checksums.ts",
"status": "ok"
},
{
"path": "scripts/lib/notice-banner.ts",
"status": "ok"
},
{
"path": "scripts/lib/resolve-pin.sh",
"status": "ok"
},
{
"path": "scripts/lib/task-infra-integrity.ts",
"status": "ok"
},
{
"path": "scripts/lib/toolkit-script-integrity.ts",
"status": "ok"
},
{
"path": "scripts/lib/tree-permissions.test.ts",
"status": "ok"
},
{
"path": "scripts/lib/tree-permissions.ts",
"status": "ok"
},
{
"path": "scripts/record-detector-inputs.ts",
"status": "ok"
},
{
"path": "scripts/reference_run_capture.py",
"status": "ok"
},
{
"path": "scripts/refresh-harness-auth",
"status": "ok"
},
{
"path": "scripts/replay_agent.py",
"status": "ok"
},
{
"path": "scripts/resolve_harness.py",
"status": "ok"
},
{
"path": "scripts/sanitize-session-jsonl.ts",
"status": "ok"
},
{
"path": "scripts/session-id.ts",
"status": "ok"
},
{
"path": "scripts/setup-harnesses.sh",
"status": "ok"
},
{
"path": "scripts/snapshot_agent.py",
"status": "ok"
},
{
"path": "scripts/snapshot-to-task.ts",
"status": "ok"
},
{
"path": "scripts/stage-atomic-rubric.ts",
"status": "ok"
},
{
"path": "scripts/stamp-trial-inputs.ts",
"status": "ok"
},
{
"path": "scripts/str_replace_editor",
"status": "ok"
},
{
"path": "scripts/str_replace_editor_vendor/__init__.py",
"status": "ok"
},
{
"path": "scripts/str_replace_editor_vendor/base.py",
"status": "ok"
},
{
"path": "scripts/str_replace_editor_vendor/edit.py",
"status": "ok"
},
{
"path": "scripts/str_replace_editor_vendor/run.py",
"status": "ok"
},
{
"path": "scripts/submit-task.ts",
"status": "ok"
},
{
"path": "scripts/toolset_note_browser.md",
"status": "ok"
},
{
"path": "scripts/toolset_note_read.md",
"status": "ok"
},
{
"path": "scripts/toolset_note.md",
"status": "ok"
},
{
"path": "scripts/validate_task_dir.py",
"status": "ok"
},
{
"path": "scripts/welcome.sh",
"status": "ok"
}
]
}