remove folder - untrustworthy

This commit is contained in:
2026-08-11 14:12:23 -04:00
parent f18bb0a146
commit 0012380fd3
140 changed files with 0 additions and 33136 deletions

View File

@@ -1,87 +0,0 @@
---
name: detector-offline-verifiability
description: |
Self-check whether your task makes sense in the no-network sandbox it runs
in. The test agent's environment is initialized up front — repo checked
out, packages installed — and then runs with no outbound network access, so
a good task is offline-completable and offline-verifiable: a competent SWE
could do the work AND trust their verification of it entirely from within
the repo. Flags tasks whose success criteria live materially outside the
sandbox — "speed up our CI/CD pipeline" (verifying needs the live
pipeline), "redeploy to prod" (prod doesn't exist in the sandbox),
"migrate from Zendesk to Intercom" (neither service is reachable, so
mocks are guesses that likely won't survive real integration), "check the
dashboard," published-package behavior. External services as scenario
dressing are fine; protocol-slice integrations against a faithful local
fake are fine. Explicitly advisory: every finding is something to
consider, never a failure, and it blocks nothing. Reads instruction.md +
the resolved grader guidance (+ the workspace for local fakes); runs
before or after reference runs exist.
allowed-tools: Bash, Read, Write
---
# Offline-verifiability detector
This skill checks one of your tasks for **offline-verifiability** — whether
the ask still makes sense inside the sandbox the test agent actually gets.
That sandbox is initialized before the task starts (repo checked out,
dependencies installed) and then has **no outbound network access**. So the
question is: could a competent SWE complete AND verify your task entirely
from within the initialized repo — and would their "it works" actually be
trustworthy?
The failure shape to catch: tasks whose *success criteria* live outside the
sandbox. "Speed up our CI/CD pipeline" — the pipeline the work would be
verified against isn't there. "Redeploy to prod" — there is no prod. "Migrate
from Zendesk to Intercom" — the agent can't interact with either service, so
it can only mock both ends, and mocks written without ever touching the real
services almost certainly won't work at integration time. When a task has
this shape, the grade measures how convincingly the agent pantomimes the
work, not whether the work is right — and an agent that honestly says "I
can't verify this from here" can end up scoring worse than one that
confidently fakes it.
What *doesn't* trip this check: external services as scenario dressing (a
prompt set at a company that uses Stripe is realism, as long as the graded
work and its verification are local), and integrations scoped to a documented
protocol slice with a faithful local fake — ideally wired through the fake
providers your repo already ships (see `/brainstorm-product-arcs` for the
"simulate the protocol, not the product" filter this mirrors).
**This check is advisory.** Where the line falls is a judgment call — a task
can even be deliberately built around recognizing the sandbox's limits, with
grader guidance that credits saying so. The report exists so you can *consider*
where your success criteria live: each finding quotes the passage, says what a
human SWE would need the network or a live system for, and offers a rescoping
option, so the decision stays yours. Nothing here blocks your submission.
Read these before deciding:
1. `.claude/skills/_detector-worker-shell.md` — where to write the report and how to handle re-runs.
2. `.claude/skills/detector-offline-verifiability/core.md` — the controlling test (offline-completable + offline-verifiable), the external-dependency shapes, the mock-fidelity boundary, what is NOT a finding, verdict enums, and the body schema.
Compose the report per the schema in `core.md` and write it per `_detector-worker-shell.md`.
## Acting on the verdict
- **`offline-verifiable`** — the work and its verification both live inside
the workspace; any external names are scenario context or faithfully
faked. Good. Move on.
- **`partial`** — the core of your task is offline-completable, but some
success criteria lean outside: a "works in prod"-shaped expectation, a
local proxy (config parses, unit tests pass) standing in for an external
outcome (the pipeline gets faster), or a mock whose fidelity is carrying a
lot of the grade. Read each finding and decide: tighten the prompt so it
asks for the local slice, point the criterion at your repo's fake provider,
or keep the framing deliberately and make sure your grader guidance grades
only what the sandbox can check (crediting honest disclosure of the rest).
- **`not-offline-verifiable`** — the system your task operates on (pipeline,
prod, third-party service) isn't in the sandbox and can't be faithfully
faked, so neither doing the work well nor verifying it can happen there.
Consider the rescoping option in each finding: extract the protocol slice
and build an adversarial local mock for it, reframe the ask as an
assessment or plan graded on repo evidence, or pick a different behavior to
test. If you believe the task works as-is, that's your call — but make sure
the grader guidance never asks the grader (or the agent) for a verification
the sandbox cannot perform.
- **`not-applicable`** — there's no prompt to assess yet. Draft it first.