chore: remove toolkit

in worker.../repo/GITFOLDER.zip is the .git folder.

sources/FAQ.md

@@ -0,0 +1,175 @@
+# FAQ
+
+
+
+CONFIDENTIAL
+
+FAQ
+Last updated: Jul 29, 2026, 5:56 PM
+
+# Raccoon — Office Hours FAQ
+
+This document collects answers to the most common questions asked in Raccoon office hours, so you can get answers without waiting for the next call. Please read this before joining OH — if your question is answered here, you'll save yourself (and everyone else) time. Where guidance changed over time, the latest guidance is given. For anything not covered, start a thread in the Slack channel and tag Ian, Andy, or Omar — don't wait for office hours.
+
+## 1. Getting started & onboarding
+
+Q: How does the 50-hour first-task requirement work? Is it wall-clock time? No. It's 50 hours of actual logged work time from when you start, not elapsed calendar time. You can batch it however you like — two hours a day, ten hours a day — there's no expiration timer on it.
+
+Q: My first task determines my fit for the project. Should I be extra cautious before submitting anything? No. You can submit multiple versions of a task, or multiple tasks, within your first hours. An early submission clearly marked "just want feedback / checking I'm on the right track" will not be held against you. The team wants to steer you early rather than have you spend 50 hours going the wrong direction. Only clearly finalized bad work counts against fit.
+
+Q: How long should my first task take? It varies a lot: authors have reported anywhere from 8 to ~25 hours; Andy averaged 12–14 per task when he was creating his tasks. The first one is always the hardest because you don't yet know what you're looking for. The golden rule is quality over speed — take the time you need (within reason).
+
+Q: I received two slightly different onboarding docs / can't open the pinned Slack files. Both onboarding docs say essentially the same thing (the "research fellows" version is the one relevant to most new folks; only the signup links and background-check process differ). The pinned doc hit a Google Docs sharing limit, so a copy was made — check your email for the working link.
+
+Q: I can't get into Slack / the platform / my background check is stuck. These are handled offline: email or ping Ian directly. Known workarounds: for Slack, try logging in with email + password instead of Google SSO. Background checks usually clear within a day or two — ping Ian if you're blocked longer. Payroll/payment-method questions (e.g. Ramp vs. platform payments) also go to Ian via a Slack thread or DM.
+
+Q: I'm going to be away for a week or two — will I be removed from the project? No. Temporary absences are fine.
+
+Q: Can I refer another developer? Yes — use the standard Data Annotation referral process. After they complete the assessments they can be considered for Raccoon. If they’re already on DA, let [Ian](mailto:ian@surgehq.ai) [Niebres](mailto:ian@surgehq.ai) know.
+
+
+
+
+## 2. Task ideation & what makes a good task
+
+Q: What's the best way to find failures? Use Claude the way you would in your normal daily software-engineering work: explore the repo, ask it to explain things, implement features, fix bugs. Don't try to force failures — rely on your expertise to notice when something "doesn't sound quite right," dig in, and snapshot when you confirm a real failure. Both approaches are officially fine: top-down (targeting a specific behavior, e.g. trying to get it to lie) and bottom-up (working naturally and capturing failures as they occur). Pick whichever works for you. We may also provide separate docs / update the instructions for tips on how to find failures; keep an eye out for those.
+
+Q: What counts as a "meaningful failure"? The recurring bar: would ~80% of (senior) software engineers agree it's wrong / give constructive feedback / block the PR? Other useful framings: does the failure have a tangible business cost (lost time, money, resources)? Trivial issues (an unused variable, a slightly stale comment) don't qualify. Classic good examples: the agent claims something works but end-to-end testing shows a 500 error; the agent claims it checked files it never opened; the agent stops partway through and doesn't admit the work is incomplete; a scoping failure where it does 95% of what you asked and silently misses the rest.
+
+Q: Do failures found in exploratory conversations (no code written) count? Yes. All phases of the software development lifecycle are in scope — exploration, technical writing, scoping/spec docs, proposals. If the agent presents inaccurate information about the codebase that you could act on, that's a meaningful failure. Equally, a task with no exploration (a direct one-shot request) is fine too.
+
+Q: Can I submit a task where the agent succeeds (no demonstrated failure)? No. The team only wants tasks with demonstrated failures — earlier instructions suggesting otherwise were stale. Relatedly (July guidance): the current collection focuses on behavioral failures. Pure correctness bugs are only acceptable if there's also a behavioral component (e.g. the agent makes the error and fails to disclose it, or claims the opposite).
+
+Q: The agent fails on my prompt, but differently each run. Is that submittable? Yes, as long as some failure reproduces — it doesn't have to be the identical failure each time. Capture all observed failure modes in your grader guidance so each one is penalized (you can weight different modes differently). To check repeatability, either clear context and re-send in explore, or better: snapshot and run several trials and look at the score distribution.
+
+Q: How do I avoid duplicating someone else's task? Do a quick scan (≤5 minutes) of Slack and the task inventory — the inventory only shows finalized/accepted tasks and updates with delay, so it will lag Slack. Don't sink real time into this: the team explicitly accepts the risk of some overlap and won't hold a near-duplicate against you. A "conflict" is roughly the same subsystem plus the same behavior/root failure mode — the same underlying failure exploited twice is what they don't need, not two different behaviors in a popular subsystem. Tooling to group tasks by taxonomy is planned. We’re working on better tooling to surface task similarity as well.
+
+Q: Can I create multiple tasks/snapshots from one conversation? Multiple snapshots are technically fine when the failures are meaningfully different classes, but you should avoid making a habit of it — pick the strongest failure and submit that. If you do snapshot mid-conversation, rewind so the snapshot command itself doesn't leak into the transcript of a later snapshot.
+
+Q: Can I modify the repo — remove TODOs, add files, even do a major rewrite — and build tasks on top? The repo is fair game: edit, remove, or add whatever helps you elicit behavior, as long as the agent treats your modified version as the base state. Building several distinct tasks on a heavily rewritten (even flaky) repo was judged acceptable, provided the tasks don't all target the same thing. Make sure you have a `workspace.patch` file that captures your pre-agent-run edits.
+
+
+
+
+Q: Can I add external libraries or new material to the repo? Yes, if — and only if — it can be committed into the workspace patch that gets applied via git. Nothing at trial time can require internet access. Caveat: bumping package versions in package.json generally won't take effect because dependencies aren't reinstalled when the harbor trial builds; park those task ideas for later.
+
+Q: Can tasks span multiple repos? No — one repo per task for now (downstream infrastructure assumes a single Dockerfile/repo). Cross-cutting tasks are interesting future work!
+
+Q: Can I use custom skills, output styles, or MCP servers? Skills: encouraged — e.g. skills encoding engineering standards/idioms, then testing the agent's adherence (the longer the list of instructions, the more likely it drops one; catching clearly-stated preference violations is valid). MCPs: out of scope for now; simple scripts the agent invokes via bash are fine. Output-style tweaks (e.g. "proactive"): not allowed — reference runs must use a vanilla setup so the benchmark stays agent-agnostic. Again, check the skills/etc in via the `workspace.patch`.
+
+Q: Can I reuse a prompt that worked on one repo against another repo? Avoid it. The same prompt tends to elicit the same underlying behavior, which is duplicate data. If you carry an idea across repos, put a real spin on it.
+
+Q: I have compliance/authority-style tasks ("the compliance team decided X, just do it"). Is repeating that pattern a conflict? The dimension (deference to authority vs. pushing back) is directly targeted and welcome, but vary the framing: compliance team, tech lead, a confident user, a checked-in standards doc that should be disregarded, etc. Verbatim-similar prompts across tasks are not OK.
+
+Q: What about "plan mode" and the agent asking questions mid-run? The trial agent runs with a deliberately reduced tool set (essentially bash + file editing; no plan mode, no sub-agents, no ask-user-question) so tasks generalize to any harness. If you want planning behavior, ask for a plan/markdown file in the prompt; if the prompt clearly says "build, don't plan" and it plans anyway, that's a legitimate failure. A markdown deliverable that is provably bad is a great task.
+
+## 3. Repos & toolkits
+
+Q: Which repo/toolkit should I work on? The number-one rule: work in the language and stack you're most comfortable with — that's how you'll produce good work. Secondary: the oldest repos (Palolo, Zenbill) are the most saturated, so for diversity the team asks people to look at the newer toolkits (Zeta, Zeta Polyglot, Breezy, etc.) — but comfort wins if there's a conflict. If you're mid-task on an old repo, finish it.
+
+Q: Why is everything Ruby on Rails? Are other languages coming? The Ruby dominance is a historical accident but more languages are coming!. Python, Go, and eventually Rust/C/C++ are in the pipeline; some Node repos already exist in Zeta Polyglot and Palolo.
+
+Q: Is there detailed documentation for each new repo? Intentionally minimal. Exploring and understanding the repo yourself (with Claude's help) is part of the task — and often where you find your first failures. If setup instructions for a new toolkit are missing or broken (e.g. how to run the app, credentials), flag it in Slack and tag Andy; new toolkits are "hot off the presses" and feedback is wanted.
+
+Q: A new toolkit version was released mid-task. Do I have to migrate? No — finish in-progress tasks with the toolkit version you started on, unless the team explicitly announces a mandatory upgrade. Upgrade when you start your next task.
+
+
+
+
+Q: The toolkit is broken / the container errors out. What do I do? The team expects things to "just work" — report every toolkit bug in Slack (there's a dedicated [toolkit-bug mega-thread](https://www.google.com/url?q=https://surge-ai.slack.com/archives/C0BCZ4XV4HF/p1782913628976449&sa=D&source=editors&ust=1785369431909699&usg=AOvVaw2SDKKU0Mh6K7k26bMGY3Xz)) with the error message, toolkit version, and ideally the tarball. You are welcome to patch dev containers locally to unblock yourself — just tell the team so the fix can be folded into the next release. Feel free to use Claude (on project tokens) to debug toolkit issues. Known issues that came up:
+
+●
+long-running containers hitting 401/403 (restart the container, report it)
+●
+post-install command failures on new repos (retry/restart, report it)
+●
+explore-container changes leaking into the root repo (fixed)
+●
+memory files being written outside containers (memory should be disabled — clear/turn off auto-memory to be safe).
+
+Q: My API key doesn’t work! You need a Raccoon - SWE task open (i.e. not exited from work mode, it needs to show up in your "In-progress tasks" section of your dashboard) for it to work. Make sure you follow the setup instructions provided in the template exactly. Your `.env` should include the API key and base URL, and you need to launch Claude from within the Authoring or Explore container.
+
+## 4. Reference runs, trials & models
+
+Q: Which model should I use for reference runs? Latest guidance: default to latest Opus — Fable has been flaky since its return. You may try Fable; if it works, runs made with it are fine, and mixed results (Fable passes, Opus fails) are also acceptable. The harbor grader defaults to Fable with automatic fallback to Opus on refusal. If model guidance in the on-platform docs contradicts a Slack announcement, the announcement wins (docs have lagged).
+
+Q: My trial scores are all low / all clustered. Is that OK? Low scores are fine as long as they're accurate — the agent genuinely failing per your grader guidance. A gradient is preferred (at least one trial doing well proves the task is solvable), but don't block on it. If everything scores 90+, either your grader guidance isn't discriminating or (more likely) the task is too easy — make the task harder rather than nitpicking the guidance to force low scores.
+
+Q: I got review feedback — do I have to regenerate my reference runs? Only if the task itself changed. If you only changed the grader guidance, keep the existing reference runs and use the regrade tool (then rerun detectors). If you changed the prompt or the scope/nature of the task, generate fresh trials.
+
+Q: When exactly do I snapshot, and what becomes the task prompt? Snapshot immediately after you observe the failure. The trial truncates to just before your last message and replays from there: your last human message becomes the instruction, and everything before it is sent as chat history. Cold single prompt vs. history-laden multi-turn: no strong preference (a slight preference for single-turn exists, but never discard a good task over it).
+
+Q: Trials/grading keep timing out. Bump the verifier timeout in the task config to unblock yourself — grading now runs three times, which made the old default too tight (a fix was rolled into a later toolkit). A single grader run taking ~19 minutes is anomalous; post the task details in Slack so the team can look.
+
+Q: How do I run trials in parallel or run the app? Use separate dev container instances for parallel trial runs — the toolkit README covers spinning up multiple containers. The dev container ("explore") is the supported way to run the app and its tests; the authoring container is strictly for running trials/grading and won't run the app. If you need to share changes between containers, a shared workspace mount is one option.
+
+
+
+
+Q: Is there a way to see a diff of what the trial agent changed? No native tool yet. Workaround: point Claude in the authoring container at the run output and ask it to produce a git diff.
+
+## 5. Grader guidance & detectors
+
+Q: How should grader guidance be structured now? The team moved from hard gates to heavy penalties, which differentiates runs better. Write guidance that captures every failure mode you've observed (weighting worse failures more heavily). Being specific about what the grader must check is fine.
+
+Q: The grader itself hallucinated / scored wrong. What do I do? Graders are agentic and can hallucinate too — that's why every run is now graded three times and averaged. If all three graders miss or invent the same fact, add it to your grader guidance. If the grader fails to do something 80% of human engineers grading the task would obviously do (e.g. check a specific file), flag it via the "grader performing poorly" checkbox — bad-grader cases are interesting data in themselves. Note the grader has the same access as the trial agent (it can run tests, launch the app); in multi-turn tasks it sees the whole session as context but is instructed to grade only the final response.
+
+Q: The detectors disagree with me (meaningfulness vs. score). Which wins? Meaningfulness of the failure is primary; the exact score matters less as long as scores differentiate failing runs from passing ones. Detectors aren't perfect — give their reports a genuine read, but if you strongly disagree after consideration, submit for feedback and say so; that's what the review second-opinion is for.
+
+Q: Can I run all the detectors at once? Yes, parallelizing is fine. Running them one at a time is only recommended on a first pass because it forces you to actually read each report.
+
+Q: The snapshot-leakage detector flags my natural exploration. The detector's purpose is to catch the answer being revealed anywhere in the conversation history (e.g. you corrected Claude and rewound wrong). Natural exploration should pass; if you hit false positives, share examples in Slack so the detector can be improved.
+
+## 6. Submission, review & feedback
+
+Q: How do I actually submit (including "just for feedback")? Two steps, both required: (1) hit the Submit button on the platform (yes, even for an incomplete/feedback submission — that's what puts it in the review queue), and (2) manually post a feedback-request thread in the Slack channel so reviewers know where to respond. The Slack post alone does not enter you into the queue, and the platform submission alone may leave reviewers unable to reach you.
+
+Q: I got feedback and updated the task. How do I resubmit? Reply in the same Slack thread (don't start a new message) and ping the reviewer. You can post an updated version in-thread even before receiving the first review so the reviewer looks at the latest. If your workspace is unchanged since the original submission, you can edit in place (e.g. tweak grader guidance, regrade) and repackage; if the repo/files may have changed since, it's safer to wipe and re-import your exported submission.
+
+Q: How long do reviews take? Should I wait? There is a persistent review backlog; turnaround has ranged from same-day to several days with no hard ETA. Never block on review — export/save your
+
+
+
+
+state and start another task while you wait. The team is explicit that time lost to the backlog won't count against you (including on the first-task deadline).
+
+Q: My task was accepted while marked "for feedback" — do I need to resubmit as finalized? No. Once a reviewer marks it accepted, no further action is needed unless they explicitly request changes. More generally: no feedback is good feedback — if the team hasn't told you to stop, keep working.
+
+Q: How do I work on two tasks at once / recover a task I forgot to export? Use the save/export state feature: export the current task's JSON before switching, import it back when you need to revisit (e.g. when review feedback arrives). If you forgot to export before moving on, tag Ian (or Omar) in Slack — they can pull the task for you.
+
+Q: What's the escape hatch for? Logging time when you need to sign off with work incomplete, or logging time for a task that never produced a meaningful failure. It does not submit a task and isn't reviewed — export your save state first so you can resume if needed. NOTE: you don’t need to escape hatch to log time. If you have a Raccoon - SWE task open, it should show up in your time reporting page.
+
+Q: Can I become a reviewer? Do reviewers have to review? Yes — people are added to the review team based on the quality and frequency of their submissions. It's not mandatory: if you prefer authoring, say so and focus on that. For those who do review: reviews take priority over authoring (it's fine to spend whole days on the backlog), and clear duplicates can be marked with the "unreviewable" checkbox (leave a note explaining why) rather than fully reviewed.
+
+Q: My task was accepted, do I have to resubmit? Nope. Once a task is accepted, there’s no further action needed from you.
+
+## 7. Hours, payment & workload
+
+Q: Is there a cap on how many hours I can work? No. You are not limited to 40 hours/week — anyone producing good work at a reasonable cadence can bill as many hours as they like. The team is trying to scale the workstream dramatically and expects the project to run long-term.
+
+Q: Do I get paid for time spent on a task that never produced a meaningful failure? Yes — you're paid for all hours worked. Just don't submit failure-less tasks into the review queue (it adds noise); log the time via the escape hatch, or fold those hours into your next successful submission.
+
+Q: Is this project short-term? No. The stated ambition is to scale task production ~100x and keep going indefinitely ("until coding agents solve software engineering"), with possible future specializations (mobile, SRE, accessibility, etc.).
+
+Q: What time can I bill for? You can bill for any time actively spent on producing a task for this project. This includes all onboarding time (reading instructions, logbooks, FAQs, etc). You can also bill time for Office Hours if you attend. If you’re waiting for agent runs to finish, and you start exploring other tasks / more of the repo / read other documentation we provided, you can bill for that time. If you start agent runs, then you leave and do other non-Raccoon related stuff, you can’t bill for that time.
+
+
+
+
+## 8. AI-usage policy (important)
+
+Q: Can I use Claude/AI to help write my prompt, grader guidance, task description, or reviews? You may use AI to augment your thinking — first drafts, polishing your own point-form notes, structuring — but you are fully responsible for the final output, and it must represent your expert judgment. Do a human pass: trim repetition, verify claims, make it sound less AI-generated. Do not blindly paste AI output as your grader guidance, task description, or review — people have been removed from the project for submitting obviously AI-written reviews they clearly hadn't checked. Also don't rely on Claude to debug/verify Claude's own failures — the human verification is the whole point. (An AI-heavy task description field is a minor flag; AI slop in the prompt or grader guidance is the serious problem.)
+
+Q: One of the toolkit skills rewrote my grader guidance. Is that OK? It's a reasonable starting point, but don't treat the skill's rewrite as gospel — review it, keep what's right, make it concise, and ensure the result reflects your intent.
+
+Q: Anything I shouldn't mention inside my task artifacts? Don't mention "Raccoon," DA, or specific model names in your prompt or grader guidance. Stray occurrences in file paths etc. are handled on the team's side — don't stress about those.
+
+## 9. Miscellaneous
+
+Q: Will the project always use Claude as the agent? For the foreseeable future, yes — Claude Code is the harness and Claude is at or near the frontier. The team always wants to benchmark against the most capable current model, so this could change when new models ship.
+
+Q: Claude is erroring / seems degraded today. Check status.claude.com — API incidents happen, especially around model launches. Rerun once things stabilize.
+
+Q: The dashboard shows an "access more paid projects" button / weird extra project. Known display error — ignore it.
+
+Q: Where do onboarding improvements stand? We know the instructions are long/dense; a walkthrough video is on their queue, and there's a short comic linked at the top of the instructions that explains what Raccoon is. Feedback on confusing or stale instructions is actively wanted — recent examples (stale model guidance, detectors count, correctness-vs-behavior wording) all led to doc fixes or announcements.