Project-2 baseline
This commit is contained in:
@@ -104,10 +104,11 @@ Read whatever you need from `harbor-tasks/<slug>/`. The load-bearing artifacts:
|
||||
the shipped `instruction.md` and the resolved guidance file — see the shape
|
||||
section below.
|
||||
- `environment/Dockerfile` plus the workspace's manifests and lockfiles — the
|
||||
static view of what the shipped image can actually do. The execution
|
||||
environment has no network access, so a tool, package, or runtime the ask or
|
||||
its verification depends on must already be present; check for it here even
|
||||
when the runs look quiet.
|
||||
static view of what the shipped image can actually do. A tool, package, or
|
||||
runtime the ask or its verification depends on must already be present — the
|
||||
sandbox does have network access, but a task whose correctness rides on a
|
||||
mid-run fetch isn't reproducible; check for it here even when the runs look
|
||||
quiet.
|
||||
- The snapshot session (`environment/session*`, when the task has one) and the
|
||||
**shipped workspace state** (the declared repo+commit plus
|
||||
`environment/workspace.patch`) — the two halves of the premise check.
|
||||
@@ -167,12 +168,13 @@ task. The agent burns turns reaching a runnable baseline (dependency versions,
|
||||
missing files, broken config, unset env). The prompt never asked for any of it.
|
||||
|
||||
Shape 1 also fires **statically**, even when no reference run visibly fights
|
||||
it: the shipped image can't support what the prompt or rubric requires. The
|
||||
execution environment has no network access, so anything the ask or its
|
||||
verification depends on must already be in the image and lockfiles — a browser
|
||||
the rubric's top tier expects the agent to verify in, a package absent from
|
||||
every manifest and lockfile, a binary that can only be installed from the
|
||||
network. The tell isn't a fight in the runs; it's verification that silently
|
||||
it: the shipped image can't support what the prompt or rubric requires.
|
||||
Anything the ask or its verification depends on should already be in the image
|
||||
and lockfiles — a browser the rubric's top tier expects the agent to verify in,
|
||||
a package absent from every manifest and lockfile, a binary that only a
|
||||
download would provide. The sandbox has network access, so the agent may well
|
||||
fetch what's missing; that it can does not make the image adequate, since the
|
||||
grade then depends on a fetch nothing pins. The tell isn't a fight in the runs; it's verification that silently
|
||||
never happens. Scope this check to capabilities the prompt or rubric actually
|
||||
require or score — not to any tool the agent might conceivably reach for.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user