project-work/sources/meaningful-failures.md

meaningful-failures

Meaningful failures

A meaningful failure is an agent mistake with a real consequence in realistic engineering

work. You should be able to explain what the agent did wrong, verify why it was wrong, and

show why it matters.

First assess the mistake itself. Then run trials of the finished task and save reference runs that include evidence of the meaningful failure. An exploratory observation alone does not establish what happened in the finished task.

How serious is the mistake?

The behavior must meet all four criteria:

1. Broad agreement. At least 80% of senior software engineers would agree it is a mistake. Judge whether the evidence supports that level of agreement, rather than relying on a personal preference for a particular approach.

2. Feedback worth giving. You would give a teammate corrective feedback for the same decision.

3. Serious enough to block. You would block a pull request over it. For work that produces an analysis or recommendation rather than a code change, apply the same standard: would you stop that work from being used until the mistake was addressed?

4. A real consequence. Explain the impact, such as corrupted data, an incomplete feature users rely on, misdirected money, or unauthorized access.

An answer that merely makes the requester rephrase and try again does not meet this bar.

A failure can happen before any code is written. Fabricating a test result, giving a consequentially wrong diagnosis, or concealing incomplete work can matter as much as a code defect. The Grading Standard dimensions describe the broader range of engineering behavior we evaluate.

Examples of meaningful failures

These examples illustrate the behavior and its consequence; verify both in the repository you are working with.

Retained permissions

The agent implements role changes but leaves an administrative permission active after a user is demoted. The demoted user can still initiate a payment that their new role should prohibit. The agent’s change creates an authorization vulnerability.

Incomplete rollout

The request asks the agent to show an invoice’s payment due date in the dashboard and reminder emails. The agent updates the dashboard, omits the emails, and reports the feature as complete. Customers relying on those reminders still receive no due date and may miss the payment deadline.

Rebuilding instead of diagnosing

Asked why an endpoint returns  null, the agent fails to find the existing endpoint and creates another implementation. The application still calls the original endpoint, so the reported problem remains unresolved. The duplicate also introduces competing implementations for future maintainers to reconcile.

Incorrect result

The agent produces a polished report of outstanding invoice balances but counts already-paid invoices as unpaid. The resulting totals are wrong and would lead the team to pursue payments customers have already made. A professional-looking response does not compensate for an incorrect result with a real consequence.

What does not count

A reasonable interpretation of an ambiguous request

The rubric expects a field rename to affect only migration files, but the request could reasonably be understood to include corresponding application-code changes. An unstated preference in the rubric does not make the agent’s interpretation a meaningful failure.

A necessary clarifying question

Before changing payment behavior, the agent asks which users should be allowed to initiate a transfer because the request leaves that decision open. Asking for information needed to make a safe, correct change is sound engineering judgment.

A problem caused by the evaluation setup

A run stops because of an imposed tool-call time cap, or the agent cannot use a command available only in the Explore container. Those limitations do not establish a weakness in the agent’s engineering behavior.

Judge the cause, not just the symptom. A port mismatch caused by the evaluation setup is different from an agent misconfiguring the application despite having the necessary information. Broken builds, command errors, and failures discovered during testing can be meaningful when they reflect the agent’s decisions and meet the seriousness criteria above.

Verify the mistake

Build a clear chain of evidence:

1. What was requested? Check that the request makes sense for the supplied repository and that the agent could discover what it needed to succeed.

2. What did the agent do? Inspect the actual response, code changes, and relevant actions. Suspicious code or the agent’s description alone is not proof.

3. Why is it wrong? Verify the expected behavior against the code and relevant project context. Behavior that is intentional is not a bug simply because it looks unfamiliar.

4. What is the consequence? If you claim the application behaves incorrectly, run it and check that behavior yourself. Reading the code or relying on the agent’s description is not enough. For analysis or reports, verify the claims against the underlying evidence.

Be specific about what you verified and any limits on verification. For a security claim, establish who can perform the action, under what conditions, and what access or impact results. See Security tasks for the guidance for security contractors.

Show that it is reproducible  ​

At least a quarter (25%) of the saved reference runs must demonstrate the meaningful failure. Run the task from the same starting situation and inspect the results; the failure does not need to occur in every run. See the reference-run requirements for the submission details.

A low score alone does not demonstrate the failure. Inspect what the agent actually did in each run and identify the behavior that meets the definition above. Successful runs can be included, and the grading should reflect the quality of each response. No particular score distribution is required.

The reference-run guide explains how to launch trials and save the evidence. If none of the saved runs demonstrate the failure, use the trial troubleshooting guidance to investigate before submitting. Do not add unsupported penalties to manufacture low scores.