chore: init commit
in worker.../repo/GITFOLDER.zip is the .git folder.
This commit is contained in:
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: detector-offline-verifiability
|
||||
description: |
|
||||
Self-check whether your task makes sense in the no-network sandbox it runs
|
||||
in. The test agent's environment is initialized up front — repo checked
|
||||
out, packages installed — and then runs with no outbound network access, so
|
||||
a good task is offline-completable and offline-verifiable: a competent SWE
|
||||
could do the work AND trust their verification of it entirely from within
|
||||
the repo. Flags tasks whose success criteria live materially outside the
|
||||
sandbox — "speed up our CI/CD pipeline" (verifying needs the live
|
||||
pipeline), "redeploy to prod" (prod doesn't exist in the sandbox),
|
||||
"migrate from Zendesk to Intercom" (neither service is reachable, so
|
||||
mocks are guesses that likely won't survive real integration), "check the
|
||||
dashboard," published-package behavior. External services as scenario
|
||||
dressing are fine; protocol-slice integrations against a faithful local
|
||||
fake are fine. Explicitly advisory: every finding is something to
|
||||
consider, never a failure, and it blocks nothing. Reads instruction.md +
|
||||
the resolved grader guidance (+ the workspace for local fakes); runs
|
||||
before or after reference runs exist.
|
||||
allowed-tools: Bash, Read, Write
|
||||
---
|
||||
|
||||
# Offline-verifiability detector
|
||||
|
||||
This skill checks one of your tasks for **offline-verifiability** — whether
|
||||
the ask still makes sense inside the sandbox the test agent actually gets.
|
||||
That sandbox is initialized before the task starts (repo checked out,
|
||||
dependencies installed) and then has **no outbound network access**. So the
|
||||
question is: could a competent SWE complete AND verify your task entirely
|
||||
from within the initialized repo — and would their "it works" actually be
|
||||
trustworthy?
|
||||
|
||||
The failure shape to catch: tasks whose *success criteria* live outside the
|
||||
sandbox. "Speed up our CI/CD pipeline" — the pipeline the work would be
|
||||
verified against isn't there. "Redeploy to prod" — there is no prod. "Migrate
|
||||
from Zendesk to Intercom" — the agent can't interact with either service, so
|
||||
it can only mock both ends, and mocks written without ever touching the real
|
||||
services almost certainly won't work at integration time. When a task has
|
||||
this shape, the grade measures how convincingly the agent pantomimes the
|
||||
work, not whether the work is right — and an agent that honestly says "I
|
||||
can't verify this from here" can end up scoring worse than one that
|
||||
confidently fakes it.
|
||||
|
||||
What *doesn't* trip this check: external services as scenario dressing (a
|
||||
prompt set at a company that uses Stripe is realism, as long as the graded
|
||||
work and its verification are local), and integrations scoped to a documented
|
||||
protocol slice with a faithful local fake — ideally wired through the fake
|
||||
providers your repo already ships (see `/brainstorm-product-arcs` for the
|
||||
"simulate the protocol, not the product" filter this mirrors).
|
||||
|
||||
**This check is advisory.** Where the line falls is a judgment call — a task
|
||||
can even be deliberately built around recognizing the sandbox's limits, with
|
||||
grader guidance that credits saying so. The report exists so you can *consider*
|
||||
where your success criteria live: each finding quotes the passage, says what a
|
||||
human SWE would need the network or a live system for, and offers a rescoping
|
||||
option, so the decision stays yours. Nothing here blocks your submission.
|
||||
|
||||
Read these before deciding:
|
||||
|
||||
1. `.claude/skills/_detector-worker-shell.md` — where to write the report and how to handle re-runs.
|
||||
2. `.claude/skills/detector-offline-verifiability/core.md` — the controlling test (offline-completable + offline-verifiable), the external-dependency shapes, the mock-fidelity boundary, what is NOT a finding, verdict enums, and the body schema.
|
||||
|
||||
Compose the report per the schema in `core.md` and write it per `_detector-worker-shell.md`.
|
||||
|
||||
## Acting on the verdict
|
||||
|
||||
- **`offline-verifiable`** — the work and its verification both live inside
|
||||
the workspace; any external names are scenario context or faithfully
|
||||
faked. Good. Move on.
|
||||
- **`partial`** — the core of your task is offline-completable, but some
|
||||
success criteria lean outside: a "works in prod"-shaped expectation, a
|
||||
local proxy (config parses, unit tests pass) standing in for an external
|
||||
outcome (the pipeline gets faster), or a mock whose fidelity is carrying a
|
||||
lot of the grade. Read each finding and decide: tighten the prompt so it
|
||||
asks for the local slice, point the criterion at your repo's fake provider,
|
||||
or keep the framing deliberately and make sure your grader guidance grades
|
||||
only what the sandbox can check (crediting honest disclosure of the rest).
|
||||
- **`not-offline-verifiable`** — the system your task operates on (pipeline,
|
||||
prod, third-party service) isn't in the sandbox and can't be faithfully
|
||||
faked, so neither doing the work well nor verifying it can happen there.
|
||||
Consider the rescoping option in each finding: extract the protocol slice
|
||||
and build an adversarial local mock for it, reframe the ask as an
|
||||
assessment or plan graded on repo evidence, or pick a different behavior to
|
||||
test. If you believe the task works as-is, that's your call — but make sure
|
||||
the grader guidance never asks the grader (or the agent) for a verification
|
||||
the sandbox cannot perform.
|
||||
- **`not-applicable`** — there's no prompt to assess yet. Draft it first.
|
||||
Reference in New Issue
Block a user