Loaded up for the 3rd redo

Still on potion-voice
This commit is contained in:
2026-09-26 14:57:10 -04:00
parent bceb52e8ee
commit e55ccea018
216 changed files with 43127 additions and 41 deletions

View File

@@ -0,0 +1,92 @@
---
name: detector-credential-leakage
description: |
Self-check whether your submission ships a credential inside its authored
surfaces — above all `environment/workspace.patch`. Mainly one job: find
leaked keys, tokens and secrets. Deterministic pattern checks hard-flag your
authoring environment's own env vars (`ANTHROPIC_API_KEY`,
`ANTHROPIC_BASE_URL`, `USER_ID` as an env assignment) and well-known secret
shapes (`sk-ant-…`, AWS `AKIA…`, GitHub `ghp_…`, Google `AIza…`, Stripe
secret keys, bearer tokens, private-key blocks, URL-embedded passwords) on
lines your patch adds; a placeholder test then clears dummies, `.env.example`
files, dev defaults and code identifiers. A `credential-leak` must be fixed
before submitting AND the key reported for rotation, since removing the line
doesn't un-ship it; `suspicious-content` is advisory. A second, narrow check
flags an absolute path from your own machine that continues into your checkout
on a line your patch adds (`/home/you/.../worker-toolkit-x/repo/...`) — a
patch is repo-relative, so such a path only gets in by accident: that's
`internal-leak`, fix it before submitting, nothing to rotate. The report never
reproduces secret values. Reads workspace.patch (+ Dockerfile,
instruction.md, tests/*.md); runs before or after reference runs exist.
allowed-tools: Bash, Read, Write
---
# Credential-leakage detector
This skill checks one of your tasks for **a leaked credential** — a key, token
or secret swept out of your authoring environment into the submission's
authored surfaces, above all `environment/workspace.patch`. Everything your
patch adds ships to everyone downstream, so a leaked key is compromised the
moment you submit, and scrubbing it afterwards doesn't undo that. It also
catches one closely-related shape: an absolute path from your own machine.
The failure shapes to catch:
- **Your toolkit `.env`** — your personal `ANTHROPIC_API_KEY`,
`ANTHROPIC_BASE_URL` and `USER_ID` landing in the workspace as a new `.env`
file, a `.env.bak-*` backup, or a symlink to `/home/<you>/.env`.
- **Any real third-party secret** the patch adds — an AWS or Google key, a
GitHub token, a Stripe secret key, a private-key block, a captured request
carrying a live `Authorization: Bearer …`, a database URL with the password
embedded.
- **An absolute path from your machine into your checkout**, on a line your
patch adds — `/home/you/…/worker-toolkit-<repo>/repo/app/foo.rb`. A patch is
repo-relative by construction, so this only ever gets in by accident: a
coverage report keyed by your file paths, or a helper script with your
checkout hardcoded. It ships your username and directory layout to everyone
downstream. Rare — 2 in 350 patches.
What *doesn't* trip this check: placeholder and example values (`.env.example`
with dummies, `sk-ant-...` as a literal template), dev defaults
(`POSTGRES_PASSWORD=postgres` in a local docker-compose), code identifiers
(`USER_ID = 4958` as a test constant, or any variable merely *named* `SECRET`
or `TOKEN`), and secrets on context or removed lines — those belong to the
source repo, not to you.
Nor do generic paths that name no person and no checkout — `/home/runner/work/…`
in a CI workflow, `/home/ubuntu/<app>` in a deploy config, `/home/app/…` in a
compose volume — which real repos legitimately commit.
Also out of scope, and never reported here: authoring artifacts
(`.raccoon-setup-done`, `.claude/settings.local.json`, stray logs) and patch
content that simply doesn't relate to the task.
Read these before deciding:
1. `.claude/skills/_detector-worker-shell.md` — where to write the report and how to handle re-runs.
2. `.claude/skills/detector-credential-leakage/core.md` — the deterministic pattern checks to run, the placeholder test, the redaction rule (never quote a secret value), what is NOT a finding, the out-of-scope list, verdict enums, and the body schema.
Compose the report per the schema in `core.md` and write it per `_detector-worker-shell.md`.
## Acting on the verdict
- **`clean`** — nothing your patch adds looks like a credential. Good, move on.
This is the normal answer.
- **`suspicious-content`** — no confirmed credential, but something
credential-shaped couldn't be resolved: a captured request with a real (if
low-sensitivity) token, a config file of credential-shaped values. Replace
the value with a placeholder, drop the file, or satisfy yourself it's
genuinely scenario material.
- **`credential-leak`** — a real credential (or your authoring env vars) is in
the patch. Act before submitting: (1) remove the material and regenerate the
patch with `bash scripts/check-workspace-sync.sh --update-patch
harbor-tasks/<slug>`; (2) re-run this detector to confirm it's gone;
(3) report the leaked value through your support channel so it can be
rotated — scrubbing the patch does not un-ship a key that already left your
machine in an earlier submission.
- **`internal-leak`** — your patch adds an absolute path from your own machine
into your checkout. Fix before submitting: remove or relativize the path (or
drop the file, if it's a generated artifact like a coverage report),
regenerate the patch, and re-run this detector. Nothing to rotate.
- **`not-applicable`** — there's no workspace patch to assess yet. Build the
workspace first.

View File

@@ -0,0 +1,331 @@
# Credential-leakage detector — core
Canonical, context-neutral content for the detector-credential-leakage
detector: the signal (credentials shipped inside the submission's authored
surfaces, plus absolute checkout paths in the patch), the deterministic
patterns, the verdict enums, and the output schema. Read in two contexts — the base repo's review pipeline and the
worker toolkit's self-check — so nothing here references how the report is
stored downstream.
## What this detector is for
**Primarily one job: find leaked credentials.** A key, token, or secret that
shipped inside the submission and now needs removing and rotating. Plus one
narrow, deterministic second check — an absolute path into the author's own
checkout on an added patch line, which a patch can only contain by accident.
Nothing else.
Everything a task adds to the workspace ships to everyone downstream: the test
agent reads it, graders read it, and the patch text itself travels with the
submission. The task author's *authoring environment* holds credentials that
must never make that trip. The canonical incident: a `workspace.patch` that
adds a `.env` containing
```
ANTHROPIC_API_KEY=DKRY…[redacted]
ANTHROPIC_BASE_URL=https://…/llm_proxy/…
USER_ID=6428…[redacted]
```
— the author's own API key, proxy endpoint, and user identity, swept out of
their authoring container and checked into the task. Nothing about the task
needs these; the agent under test can't use them (no network); and the key is
now distributed to every downstream consumer. The same sweep brings in a `.env`
symlink into the author's home directory, an `.env.bak-*` full of real
third-party secrets, or a captured HTTP request with a live bearer token.
A credential leak is expensive in a way other findings are not: removing the
line does not un-ship the key, so the credential has to be rotated. That
asymmetry is why this detector is deterministic and why it is blocking.
## Out of scope — do NOT flag these
Do not flag these, and do not let them change the verdict:
- **Authoring artifacts** — `.raccoon-setup-done`, `.claude/settings.local.json`,
stray build logs, session-export dumps, working-tree backups.
- **Author identity anywhere but an absolute path in the patch** — a home-dir
mention in a session transcript, a name in prose, a relative path. The one
identity shape that IS in scope is the absolute checkout path check below.
- **Internal information** — the project name, or text framing the work as an
evaluation.
- **Task-irrelevant content** — a stray `.patch` file, an empty `CLAUDE.md`,
unexplained config: content that does not serve the task but carries no
secret.
If content in one of these categories *also* contains a real credential, the
credential is the finding — report it as such, and describe the file only as
its location.
## NEVER quote secret values — redact
This report is itself distributed, so reproducing a leaked value spreads the
leak. **Never copy a candidate secret into the report.** Quote the variable
name, the file path, and at most the first 4 characters followed by
`…[redacted]`:
> `ANTHROPIC_API_KEY=DKRY…[redacted]` in `.env` (new file, line 1)
This overrides the sibling detectors' quote-verbatim convention — here,
redaction wins.
## Inputs
Read from `harbor-tasks/<slug>/`:
- `environment/workspace.patch` — the primary surface. **Added lines and newly
added files are the authored surface.** Also scan the whole patch text for
secret shapes: a secret on a context or removed line is pre-existing repo
content (see "What is NOT a finding"), but it still ships, so it earns an
informational note.
- `environment/workspace/` — some submissions ship the workspace as a
materialized directory instead of a patch (`inputs.json` records
`workspacePatch: null`). It is a checkout of the source repo at the ref
`task.toml` records, so **every file in it is pre-existing repo content**
unless the task's own material shows the author put it there. There is no
added-vs-context split to read here: absent that evidence, treat a hit as the
source repo's and take the informational path.
- `environment/Dockerfile` — task-owned build steps carry `ENV`/`ARG`
credentials the same way.
- `instruction.md` and `tests/*.md` — secondary authored surfaces; a pasted
terminal capture or setup snippet can carry the same leak.
- Session files (`environment/session.jsonl`, `session-full.jsonl`), when
present — scan for secret shapes, but report hits as informational rather
than blocking: sessions pass through a dedicated path-and-marker sanitizer,
and the full session file is not part of what the test agent receives. The
blocking surface is what packs verbatim, above all `workspace.patch`.
## The check (deterministic)
Run these over the patch. The pattern list is the contract: a hit on an
**added** line or a newly added file is a `credential-leak` unless it fails
the placeholder test below. With a materialized `environment/workspace/` there
are no added lines to key on, so run the sweeps over the tree and route every
hit by provenance — which, for that tree, means the informational path.
```bash
# Authoring-environment env vars, on added lines:
grep -nE '^\+' environment/workspace.patch \
| grep -E 'ANTHROPIC_[A-Z_]+[[:space:]]*[=:]|(^|[^A-Za-z0-9_.])USER_ID[[:space:]]*='
# Well-known secret shapes, over the WHOLE patch (added hits are findings;
# context/removed hits are informational notes):
grep -nE 'sk-ant-[A-Za-z0-9_-]{8,}|AKIA[0-9A-Z]{16}|(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}|AIza[0-9A-Za-z_-]{35}|sk_(live|test)_[A-Za-z0-9]{16,}|-----BEGIN [A-Z ]*PRIVATE KEY-----|[Aa]uthorization[^A-Za-z0-9]{0,3}Bearer [A-Za-z0-9._~+/=-]{20,}|[a-z][a-z0-9+.-]*://[^/:@[:space:]]{3,}:[^@[:space:]]{8,}@' \
environment/workspace.patch
# LLM-proxy endpoints from the authoring environment:
grep -nE '^\+' environment/workspace.patch | grep -iE 'llm[_-]?proxy|dataannotation\.tech'
```
The named env vars to hard-flag on added lines:
- **`ANTHROPIC_API_KEY`** (or any `ANTHROPIC_*` var carrying a value) — the
author's personal API credential.
- **`ANTHROPIC_BASE_URL`** — the authoring environment's proxy endpoint; not a
secret alone, but pure authoring plumbing that marks the leak.
- **`USER_ID`** *as an env-var assignment* (a `.env` line, `export USER_ID=`,
`ENV USER_ID=`, especially with a UUID value). `USER_ID` / `user_id` as a
*code identifier* — a column, a variable, a test constant like
`USER_ID = 4958` — is normal code. The flag is the env-assignment shape.
**The placeholder test.** A hit whose value is plainly not real is not a leak:
empty (`QBO_SECRET=`), a template marker (`sk-ant-...`, `<your-key>`,
`${STRIPE_KEY}`, `changeme`, `your-key-here`), a documented dummy the repo
already uses in fixtures, or a commented-out no-value line in an
`.env.example`. When in doubt — the value looks high-entropy and real — flag
it; a false "compromised" alarm is far cheaper than a shipped key.
## The second check — an absolute checkout path in the patch (deterministic)
A git patch is repo-relative by construction: its headers are `a/foo.rb
b/foo.rb`, and its content is the repo's own files. An **absolute path rooted
in someone's home directory that continues into their checkout** therefore has
no legitimate reason to be in one — it can only have come from the author's
machine, and it ships the author's username, directory layout, and often their
agency's name to everyone downstream.
This is a narrow, deterministic check with a deliberately high bar: the path
must be BOTH home-rooted AND continue into a checkout component
(`worker-toolkit-<name>`, `Toolkits`, or `repo`). Requiring both is what keeps
it quiet — a repo legitimately commits `/home/runner/work/…` in a CI workflow,
`/home/ubuntu/<app>` in a deploy config, and `/home/app/…` in a compose
volume, and none of those name a person or a checkout.
```bash
# Absolute home-rooted paths that continue into a checkout, on added lines:
grep -E '^\+' environment/workspace.patch | grep -vE '^\+\+\+' \
| grep -nE '(/home/[a-zA-Z][^/[:space:]"'"'"']*|/Users/[a-zA-Z][^/[:space:]"'"'"']*|/mnt/[a-z]/[a-zA-Z][^/[:space:]"'"'"']*)(/[^/[:space:]"'"'"']+)*/(worker-toolkit-[a-z0-9-]+|Toolkits|repo)/'
```
A hit is an `internal-leak`. Across the corpus this fires on 2 of 350 patches,
so treat a hit as genuinely anomalous rather than routine. The two real shapes
seen so far: a coverage report (`coverage/.resultset.json`) keyed by the
author's absolute file paths, and a task-authored helper script with the
author's checkout path hardcoded into it.
Scope limits that make this safe to run deterministically:
- **The patch only.** Don't run it over session files (`session.jsonl`,
`session-full.jsonl`), which have their paths rewritten at task build time and
whose hits are informational at most; nor over `instruction.md` or `tests/`.
- **Added lines only** (excluding the `+++` file header). A path on a context
or removed line is the source repo's.
- **Full absolute paths only.** A bare `/home/<user>` with nothing after it, a
relative path, or a name in prose is not this finding.
Remediation is removal and regenerating the patch — no rotation, since nothing
is compromised. Report the file and the shape; you do not need to reproduce the
full path to make the point.
## What is NOT a finding
- **Placeholder and example values.** `.env.example` / `.env.sample` /
`.env.test` with empty or dummy values, `sk_test`-style fixture strings the
repo's suite already uses as fakes, `changeme`,
`dev-insecure-session-secret-change-me`, `${VAR:-default}` expansions.
- **Dev-infrastructure defaults.** `POSTGRES_PASSWORD=postgres` in a local
docker-compose, `SESSION_SECRET: dev-…` in a dev config — local-only and
value-free by convention.
- **Code identifiers.** `SECRET`, `TOKEN`, `PASSWORD`, `USER_ID` in a variable
or column name. A real-looking *value* is the finding, never the vocabulary.
- **Env vars the task's own scenario needs.** If the product calls an external
API and the task is about that integration, documenting the env var with a
placeholder value is task material.
- **Pre-existing repo content.** Secrets the source repo committed are not the
author's leak, whichever way the workspace ships: on a *context or removed*
patch line, or anywhere in a materialized `environment/workspace/`. Don't
flag the author, and **never let one move the verdict** — a submission whose
only hits are repo-resident is `clean`. DO add an informational note routed
to the repo owner, since the secret still ships and only they can rotate it.
Removing it from the workspace is not the remedy and is not something to ask
the author for: it would edit the checkout the task depends on, and it does
not un-ship what the source history already carries.
- **A task whose subject IS a leaked credential.** A scenario can plant a fake
"leaked key" for the agent to find. Flag only if the planted value is real.
- **Generic service-account and CI paths.** `/home/runner/work/…` in a
workflow, `/home/ubuntu/<app>` in a deploy config, `/home/app/…` in a compose
volume, `/home/node/…` from a container: home-rooted but naming no person and
no checkout, so the second check stays quiet on them by design.
- **Everything in "Out of scope" above.**
## Verdict definitions
- **`clean`** — no pattern hit **on an authored surface** survives the
placeholder test. This is the expected verdict for the large majority of
submissions, including any carrying out-of-scope material, and including one
whose only hits are pre-existing source-repo credentials — however real those
are, they are the repo owner's to rotate, and they belong in an informational
finding under a `clean` verdict.
- **`suspicious-content`** — no confirmed credential, but the **author's own**
material carries something credential-shaped that could not be resolved: a
real-looking but low-sensitivity token (a public-by-design client token, a
locally-signed dev JWT), or a value whose realness is genuinely unclear.
Advisory. Never reach for this because a repo-resident secret looked real —
realness is not what this verdict turns on; provenance is.
- **`credential-leak`** — a pattern hit on added content survives the
placeholder test: a named authoring-environment variable carrying a value,
or a known secret shape. Blocking, and the strongest form of remediation:
remove the material AND treat the credential as compromised and report it
for rotation. Scrubbing the patch alone does not fix the key.
- **`internal-leak`** — the second check hit: `workspace.patch` adds an
absolute home-rooted path that continues into the author's checkout.
Blocking, but no rotation — remove the material and regenerate the patch.
When both checks hit, `credential-leak` is the verdict; list every finding
either way.
- **`not-applicable`** — nothing to assess: no `environment/workspace.patch`
and no authored Dockerfile/doc surfaces exist yet. Re-run once the workspace
lands.
`internal-leak` means ONLY the absolute-checkout-path finding above.
## Confidence
- **HIGH** — a pattern hit with a real-looking value, or plainly nothing
anywhere. The deterministic check makes most calls HIGH by construction.
- **MEDIUM** — the call rests on the placeholder test in a case a reasonable
reviewer could read either way: a token that may be public-by-design, an env
file whose values might all be dummies.
- **LOW** — limited information: the patch is enormous and only sampled.
## Relationship to other detectors
- **vs. detector-over-hinting.** Same primary surface (`workspace.patch`
additions), different defect: over-hinting reads authored comments for
content that does the agent's thinking. Verdicts are independent.
- **vs. detector-snapshot-leakage.** "Leakage" there means the *answer*
reaching the test agent through the inherited session. Here it means a
*credential* reaching the shipped workspace. The shared word is coincidence.
- **vs. detector-broken-dev-env.** A dangling `.env` symlink can also break
the workspace at runtime — that detector owns the build/run consequences.
## Anti-patterns: do not do these
- **Never reproduce a secret value in the report.** Redact to a 4-character
stub. Failing this is worse than a missed finding.
- **Don't flag vocabulary.** Run the placeholder test before flagging.
- **Don't flag anything from "Out of scope".** Not as the verdict, not as a
finding. An empty marker file is not a leak of any kind.
- **Don't widen the checkout-path check.** It needs a full absolute path that
is home-rooted AND continues into a checkout, on an added patch line. A bare
`/home/<user>`, a CI path, or a name in prose is not it.
- **Don't flag pre-existing repo secrets as author leaks.** Context and removed
lines, and every file of a materialized `environment/workspace/`, belong to
the source repo. Attribute them correctly, and leave the verdict `clean`.
- **Don't soften a real hit into advice.** A real key in the patch is not
"something to consider" — say plainly that it must be removed and rotated.
- **Don't skip the check because the patch "looks clean".** The canonical
incident sat in plain sight at the top of the patch.
- **Don't cite evidence you haven't verified in the submitted package.** Point
at the actual file and line in the actual patch.
## Frontmatter and body schema
YAML frontmatter followed by a markdown body. Both contexts produce the same
shape; only the *sink* differs (the wrapping `SKILL.md` says where to send it).
**Frontmatter** — exactly these keys, exactly these enum values:
```yaml
---
detector: detector-credential-leakage
verdict: credential-leak | internal-leak | suspicious-content | clean | not-applicable
confidence: HIGH | MEDIUM | LOW
---
```
**Body sections**, in this order:
```markdown
# Credential-leakage check: <slug>
## Findings
One block per finding, strongest first:
### <short label> — <credential | checkout-path> (<leak | suspicious | informational>)
- **Where:** the file and line (patch hunk), and whether the line is added,
context, or removed.
- **What:** the variable name(s) / secret shape, with every value REDACTED to
at most 4 characters + `…[redacted]`. Never the full value.
- **Why it's a finding:** one or two sentences — which check hit, and (for a
credential) why the value reads as real rather than a placeholder.
- **Action:** for a credential, remove the material AND treat the key as
compromised (report it for rotation). For a checkout path, remove it and
regenerate the patch — nothing to rotate. For suspicious content, the
concrete check that would resolve it.
For `clean`, name the strongest near-miss (a placeholder env file, a dev
default) and say why the placeholder test cleared it. For `not-applicable`,
name the missing artifacts.
## Overall verdict
1–2 paragraphs reducing the findings to the verdict: what shipped that
shouldn't, and what remediation looks like — including, for any real
credential, that removal from the patch does not un-ship it and rotation is
the actual fix.
```
The frontmatter is what downstream tooling parses; the body is the rationale a
human reads to confirm.