eric/project-work
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

denuded instructions markdown

Browse Source
This commit is contained in:
Eric Bell
2026-09-07 14:37:04 -04:00
parent bddfed44c7
commit 6390586bad
1 changed files with 38 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
sources/260903_instructions.md
View File

@@ -24,37 +24,37 @@ regrade your reference runs. Everything else in this release is listed in 🗞
Please download the new toolkit before you start your next task.
# **Raccoon — Code Execution Task Creation**
# **theProject — Code Execution Task Creation**
⚠️ You are being given access to models via a proxy for this project. You are not permitted to use these models for any
purpose that does not support completion of a task for this specific project, and all traffic through the proxy is logged and
monitored. Abusing model access will be subject to penalties, including but not limited to removal from the platform.
*“*👋 ***New to the project?*** *Start with a* [*2-minute overview of what we're doing*](https://images-for-tasks.s3.amazonaws.com/eec61925-34b8-486c-bf6b-1ee417b2004a/raccoon/Raccoon-v2.pdf)*, put together by J.D Nichols.”*
*“*👋 ***New to the project?*** *Start with a* [*2-minute overview of what we're doing*](https://images-for-tasks.s3.amazonaws.com/eec61925-34b8-486c-bf6b-1ee417b2004a/theProject/theProject-v2.pdf)*, put together by J.D Nichols.”*
**Quick Navigation:**
[Confidentiality](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aconfidentiality) **|** [Start Here](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Astart_here_ref) **|** [Toolkit](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atoolkit_ref) **|** [Repositories](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arepo_context_ref) **|** [Agents](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Achoosing_agent_ref) **|** [Meaningful Failure](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ameaningful_failure_ref) **|** [Snapshots](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Asnapshot_ref) **|** [Workspace](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aworkspace_patch_ref) **|** [instruction.md](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Awriting_instruction_ref) **|**
[Grading](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ahow_grading_works_ref) **|** [Holistic Rubric](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Agrader_guidance_ref) **|** [Detectors](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Adetectors_ref) **|** [Reference Runs](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areference_runs_ref) **|** [Atomic Rubric](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aatomic_rubric_ref) **|** [Regrade](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aregrade_sanity_ref) **|** [Submitting & Feedback](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Afeedback_loop_ref) **|** [Versions &](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref)
[Migration](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref) **|** [Examples](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aexamples_ref) **|** [Troubleshooting](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atroubleshooting_ref)
[Confidentiality](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aconfidentiality) **|** [Start Here](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Astart_here_ref) **|** [Toolkit](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atoolkit_ref) **|** [Repositories](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arepo_context_ref) **|** [Agents](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Achoosing_agent_ref) **|** [Meaningful Failure](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ameaningful_failure_ref) **|** [Snapshots](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Asnapshot_ref) **|** [Workspace](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aworkspace_patch_ref) **|** [instruction.md](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Awriting_instruction_ref) **|**
[Grading](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ahow_grading_works_ref) **|** [Holistic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Agrader_guidance_ref) **|** [Detectors](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Adetectors_ref) **|** [Reference Runs](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areference_runs_ref) **|** [Atomic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aatomic_rubric_ref) **|** [Regrade](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aregrade_sanity_ref) **|** [Submitting & Feedback](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Afeedback_loop_ref) **|** [Versions &](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref)
[Migration](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref) **|** [Examples](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aexamples_ref) **|** [Troubleshooting](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atroubleshooting_ref)
**Quick Links****:**
[Submission Overview](https://app.dataannotation.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd)**:** View your current standing, reported-hours progress, review-queue outlook, and submission
[Submission Overview](https://app.example.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd)**:** View your current standing, reported-hours progress, review-queue outlook, and submission
history.
[**Grading Standard**](https://app.dataannotation.tech/publish/s/6d6183f7-e335-4e8a-94d2-ece2d1d5ab86.html)
[**Grading Standard**](https://app.example.tech/publish/s/6d6183f7-e335-4e8a-94d2-ece2d1d5ab86.html)
[Workflows](https://app.dataannotation.tech/publish/g/Y2IYEXllIxVDSl87an9wPgcnMyVVL3pkOwAxUg4ABAgnEmxSfQAkcwQPCSE=)
[Workflows](https://app.example.tech/publish/g/Y2IYEXllIxVDSl87an9wPgcnMyVVL3pkOwAxUg4ABAgnEmxSfQAkcwQPCSE=)
[FAQ](https://docs.google.com/document/d/1Kl0ckBAgDwdsNwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q)
[FAQ](https://docs.google.com/document/d/NwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q)
[Searching for model failures: hardness ladder](https://app.dataannotation.tech/publish/g/Y31AWnZyL2RWZhcRdBhDCS5EKRIvDylgFW5UCD1OF1QwLVNgDRMuMAs0A0w=)
[Searching for model failures: hardness ladder](https://app.example.tech/publish/g/Y31AWnZyL2RWZhcRdBhDCS5EKRIvDylgFW5UCD1OF1QwLVNgDRMuMAs0A0w=)
[~20min video of Nick talking about tasks that he likes](https://app.dataannotation.tech/publish/s/b6f62716-30d3-4dca-95e4-d4aa67e9f799.mp4)
[~20min video of Nick talking about tasks that he likes](https://app.example.tech/publish/s/b6f62716-30d3-4dca-95e4-d4aa67e9f799.mp4)
[Learning Exercise refresher](https://app.dataannotation.tech/publish/s/541aba88-70ad-4d51-867a-1fbb78a89d32.html)
[Learning Exercise refresher](https://app.example.tech/publish/s/541aba88-70ad-4d51-867a-1fbb78a89d32.html)
Task Catalog ([JSON](https://app.dataannotation.tech/publish/s/95301d04-ac07-4bd3-be80-5ee468c74fd4.json)) ([Viewer](https://app.dataannotation.tech/publish/s/4cee7539-070f-41de-98c8-e09377c2023f.html))
Task Catalog ([JSON](https://app.example.tech/publish/s/95301d04-ac07-4bd3-be80-5ee468c74fd4.json)) ([Viewer](https://app.example.tech/publish/s/4cee7539-070f-41de-98c8-e09377c2023f.html))
# 🗞️ **Recent Changes**
Dated project updates, newest first. Each entry points you to the section with the full details.
@@ -93,7 +93,7 @@ they started with, and a task in that position can be finalized without an atomi
**2026-09-03 — Codex is now the default agent for new tasks.** Start new tasks in Codex, which runs `gpt-5.6-`
`sol` at maximum reasoning effort. Claude Code remains available if you prefer it, and grading is unchanged
either way. See **Choosing Your Agent**.
**2026-09-01 — Your personal** [**Submission Overview**](https://app.dataannotation.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd) **is now live.** It brings your standing, reported-hours
**2026-09-01 — Your personal** [**Submission Overview**](https://app.example.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd) **is now live.** It brings your standing, reported-hours
progress, review-queue outlook, and submission history together in one place. Open it from Quick Links whenever
you want to check your current information.
**2026-08-31 — Task-specific tests are expected, and** `tests/test-commands.sh` **is where they get wired in.** If
@@ -179,7 +179,7 @@ anywhere public.
If you are unsure whether something is shareable, assume it is not and ask in the questions channel first.
# 🚀 **Start Here**
Welcome to project Raccoon. Your job is to build tasks that capture meaningful failures from an AI coding agent
Welcome to project theProject. Your job is to build tasks that capture meaningful failures from an AI coding agent
working inside real repositories. A failure is a moment where the agent's behavior or output falls short in a way that
matters to a real engineering team. The **What Makes a Failure Meaningful** section defines the bar a failure must
clear.
@@ -236,15 +236,15 @@ Code or Codex, and the **Choosing Your Agent** section covers both. Grading is i
choose, and the **How Grading Works** section explains the scoring in full.
## **Where to ask questions**
Questions go to the project's questions channel on Slack, `#ext-surge-raccoon-questions`. Feedback requests are
Questions go to the project's questions channel on Slack, `#ext-surge-theProject-questions`. Feedback requests are
not posted in Slack. They go in the **Review Logbook** on your task, which is where the review comes back too. Use
`#ext-surge-raccoon-feedback-requests` only to report problems with that cycle. Release announcements arrive in
`#ext-surge-theProject-feedback-requests` only to report problems with that cycle. Release announcements arrive in
`#ext-surge-raccoon-announcements`. Please do not post questions or feedback requests there. Please do not send
`#ext-surge-theProject-announcements`. Please do not post questions or feedback requests there. Please do not send
Admins direct messages unless an Admin asks you to. Please read the [FAQ document](https://docs.google.com/document/d/1Kl0ckBAgDwdsNwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q) before asking questions. If you
Admins direct messages unless an Admin asks you to. Please read the [FAQ document](https://docs.google.com/document/d/NwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q) before asking questions. If you
can't find an answer there, reach out on Slack. Before you open any repository, read the **Confidentiality** section,
which governs what you may share about this project.
@@ -265,7 +265,7 @@ preference.
Report all of the time you spend on this project, including failed or abandoned attempts and any rework after a task
comes back to you. Time spent waiting for trials or grading to finish is not reportable. Time on project work that has no
task of its own, such as the learning exercise or a survey, goes to the separate **[TIME-REPORTING PROJECT]**
**Raccoon - Off-task time reports** project. If you are going to be away for a while, let us know through the **[TIME-OFF /**
**theProject - Off-task time reports** project. If you are going to be away for a while, let us know through the **[TIME-OFF /**
**OOO / VACATION REPORTING]** project.
## **Your standing on the project**
@@ -276,7 +276,7 @@ across your most recent 100 hours of work. Once four of your tasks have been acc
to the project for as long as you remain in good standing. A task counts as accepted when a reviewer explicitly marks it
accepted, with or without edits. Only task submissions on this project count toward standing, and time you spend on
the side projects we occasionally prioritize does not count toward the 100 hours.
The [Submission Overview](https://app.dataannotation.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd), linked in Quick Links, shows your hours, your progress against these milestones, and the
The [Submission Overview](https://app.example.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd), linked in Quick Links, shows your hours, your progress against these milestones, and the
status of each submission. It is a view of your record rather than the record itself, so if something there looks wrong,
the problem is with the view and not with your standing. A submission shown as awaiting review is not a rejection, and
if it is ultimately accepted, it counts from when it was submitted.
@@ -403,7 +403,7 @@ The Setup page asks which repository you chose. Reviewers use your answer to rou
## **ZenBill (ZenBill-006)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/raccoon/Zenbill-Onboarding.pdf)
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Zenbill-Onboarding.pdf)
A **B2B payment and invoice platform** built on Rails 7 + React 18. Businesses use ZenBill to send and receive
money via ACH transfers and credit cards, manage invoices, and sync with QuickBooks Online.
@@ -422,7 +422,7 @@ Architecture
## **Palolo (Palolo-031)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/raccoon/Palolo-Onboarding.pdf)
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Palolo-Onboarding.pdf)
An **employee financial wellness platform** built as a TypeScript monorepo (pnpm, 9 packages). Employers offer
financial benefits to their employees through a dual-surface application, with one surface for employees and one for
@@ -1357,7 +1357,7 @@ When the holistic rubric is final, run the pre-trial detectors listed in the **D
generate the atomic rubric after your reference runs are graded. The **Generating the Atomic Rubric** section covers
that step, and real examples of both rubrics are collected in [Holistic + atomic rubric pairs](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arubric_pairs_ref).
that step, and real examples of both rubrics are collected in [Holistic + atomic rubric pairs](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arubric_pairs_ref).
# 🧪 **Detectors**
@@ -1771,7 +1771,7 @@ With the staged copies in place, regrade every reference run and check the resul
section covers the commands and the alignment bar, and the **Detectors** section names the two detectors that check
the atomic rubric. Real examples of both rubrics, drawn from live tasks, are collected in [Holistic + atomic rubric pairs](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arubric_pairs_ref).
the atomic rubric. Real examples of both rubrics, drawn from live tasks, are collected in [Holistic + atomic rubric pairs](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arubric_pairs_ref).
# ♻️ **Regrade & Sanity-Check**
A regrade replays a recorded reference run and grades it fresh, without re-running the agent. Once your atomic rubric
@@ -1886,11 +1886,11 @@ That includes `instruction.md`, `task.toml`, your tests with both rubrics, the e
## **Submitting on the platform**
Your answers stay on this task between rounds, so you revise here, on the same task. If you ever need an older
version of your answers, find that submission on [your past responses page](https://app.dataannotation.tech/workers/past_responses) and export it from there. Treat the past-
version of your answers, find that submission on [your past responses page](https://app.example.tech/workers/past_responses) and export it from there. Treat the past-
responses page as read-only, and never resubmit from it. Tasks do not expire, and there is no expectation to finish one
in a sitting.
On the Submit page, upload the tarball and post your message in the [🔄](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) [**Review Logbook**](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) using the four-item format
On the Submit page, upload the tarball and post your message in the [🔄](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) [**Review Logbook**](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) using the four-item format
below. The platform blocks a submission whose logbook message is empty. The Slack thread URL field on the
@@ -1917,7 +1917,7 @@ runs, as described at the end of the **Writing the Holistic Rubric** section.
## **The Review Logbook message**
The [🔄](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) [**Review Logbook**](https://app.dataannotation.tech/workers/projects/0d3bcf32-6e00-427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) is the permanent record of what you want reviewed, and the thread your reviewer answers
The [🔄](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) [**Review Logbook**](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areview_logbook_ref) is the permanent record of what you want reviewed, and the thread your reviewer answers
in. Use this format every time. Your opening message covers four items:
@@ -1965,7 +1965,7 @@ seriously, then use your judgment about what the task actually needs. If you do
a review, then the review came from a human reviewer. You may communicate through the logbook to settle this
with your reviewer, and if you feel that you need additional litigation from the admin team, make a thread in `#ext-`
`surge-raccoon-feedback-requests`.
`surge-theProject-feedback-requests`.
5. **Start your next task while you wait.** Waiting for review is never required. Once you have submitted, move on to
your next task and come back when the first one returns. Treat each submission as a checkpoint rather than a
@@ -1987,7 +1987,7 @@ consequence is simple: when a review comes back with feedback, address all of it
## **Slack is for troubleshooting the cycle**
Slack does not carry your review. Use `#ext-surge-raccoon-feedback-requests` only to report problems with the
Slack does not carry your review. Use `#ext-surge-theProject-feedback-requests` only to report problems with the
cycle itself: feedback that never arrives, a task that does not come back, or feedback that does not match the task you
submitted. If you open such a thread, paste its URL into the Slack thread URL field on the Optional comments page,
@@ -1999,7 +1999,7 @@ block formatting when you share code or logs.
# 🔄 **Toolkit Versions & Migration**
The toolkit is released in versions, and each release carries a version identifier. Release announcements are posted in
the announcements Slack channel, `#ext-surge-raccoon-announcements`, and name the identifier they introduce, so
the announcements Slack channel, `#ext-surge-theProject-announcements`, and name the identifier they introduce, so
you can always tell whether an announcement applies to the copy you are running.
@@ -2008,7 +2008,7 @@ you can always tell whether an announcement applies to the copy you are running.
Open `CHANGELOG.md` at the top level of your toolkit folder. The entry at the top of the file names the identifier of the
version you are running. If it matches the identifier in the most recent release announcement, you are on the latest
version. The current release's identifier is `7b6b67ea3d`.
version. The current release's identifier is `bogus`.
## **The stay-or-upgrade rule**
Start every new task on the latest announced toolkit version. Finish an in-progress task on the version you started it
@@ -2110,7 +2110,7 @@ crux conversion: that single penalty becomes the one crux criterion, phrased to
alongside fifteen ordinary criteria. Four reference runs, with holistic and atomic scores agreeing within
0.04 on every run.
[Download the full pair](https://app.dataannotation.tech/publish/s/55c8cff7-a27c-47e9-8e02-05b294ac90c3.zip)
[Download the full pair](https://app.example.tech/publish/s/55c8cff7-a27c-47e9-8e02-05b294ac90c3.zip)
### 🧬 **Pair 2. A conditional dealbreaker kept to one unsplittable criterion (Palolo)**
@@ -2123,7 +2123,7 @@ conjunction of a raw UTC comparison in the implementation and a confident comple
demonstrating how a conditional dealbreaker stays one unsplittable criterion. Four reference runs, with
the tightest score agreement in the set.
[Download the full pair](https://app.dataannotation.tech/publish/s/54930bf1-d059-4536-839d-cd6ae41a18b9.zip)
[Download the full pair](https://app.example.tech/publish/s/54930bf1-d059-4536-839d-cd6ae41a18b9.zip)
### 🧬 **Pair 3. Non-triggers stated as precisely as triggers, across a wide score**
### **spread (zeta)**
@@ -2136,7 +2136,7 @@ atomic rubric pairs the matching crux with a severity-free extra-credit criterio
notification gap. Eight reference runs spanning holistic scores from 0.28 to 0.98 make this the best
package for watching both rubrics discriminate between genuinely weak and genuinely strong runs.
[Download the full pair](https://app.dataannotation.tech/publish/s/d531895c-8a5d-401b-8564-f958a0ea86f4.zip)
[Download the full pair](https://app.example.tech/publish/s/d531895c-8a5d-401b-8564-f958a0ea86f4.zip)
### 🧬 **Pair 4. A strong rubric with no crux at all, and model conditional criteria**
### **(casa)**
@@ -2149,7 +2149,7 @@ findings. The atomic rubric shows what a strong no-crux rubric looks like: twent
nothing above Critical, including a conditional criterion that grants credit for a surfaced extra finding only
when it checks out against the workspace and has a real consequence. Eight reference runs.
[Download the full pair](https://app.dataannotation.tech/publish/s/6155d8e3-e9e4-466a-981c-dc40ca93d37d.zip)
[Download the full pair](https://app.example.tech/publish/s/6155d8e3-e9e4-466a-981c-dc40ca93d37d.zip)
### 🧬 **Pair 5. Criterion-targeted penalties converting without a crux (stocks-in-**
### **the-future)**
@@ -2167,10 +2167,10 @@ crux derivation rule: because no penalty targets the overall score, the twenty-c
crux, with the dealbreakers converted at Critical and an extra-credit criterion rewarding discovery of a
self-registration escalation surface. Four reference runs.
[Download the full pair](https://app.dataannotation.tech/publish/s/5ce427ba-a1f0-4d18-835d-1f656e9e956c.zip)
[Download the full pair](https://app.example.tech/publish/s/5ce427ba-a1f0-4d18-835d-1f656e9e956c.zip)
# 🛠️ **Troubleshooting & FAQ**
Start with the [FAQ document](https://docs.google.com/document/d/1Kl0ckBAgDwdsNwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q). It answers the most common questions on this project.
Start with the [FAQ document](https://docs.google.com/document/d/NwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q). It answers the most common questions on this project.
This section collects the most frequent problems and their fixes. Several fixes refer to the toolkit's two containers. The
Explore container is where you work with the repository. The Authoring container is where you run trials and package
your task. A trial is a single run of the agent against your task.
@@ -2287,6 +2287,3 @@ container, run `npm install` and retry.
| The container build stops at `could not` `read Username for 'https://github.com'` | One of GitHub's edge servers intermittently returns 401 over HTTP/2. Run `git config --global http.version HTTP/1.1` and retry the build. |
| The extracted toolkit is missing `explore/repos` or has broken links | Re-extract the zip with `unzip` or 7-Zip rather than the operating system's built-in extractor, which drops symbolic links. |
[Code of Conduct](https://docs.google.com/document/u/1/d/e/2PACX-1vQ3pmQsDlC3_aA4zcV1g3Bd9-KYrGPg9Z37FMt0C8UdBObA5qAa9f36uEDAtXtT_NCecyqKAH6xD7UZ/pub)
[Support](https://app.dataannotation.tech/workers/support)
© 2026 DataAnnotation. All rights reserved.