25 Commits

Author SHA1 Message Date
00408fd43c fix: ran atomic-rubric and grader-context creation skill 2026-09-22 20:19:59 -04:00
a7b6b57cab fix: rename yml file to yaml 2026-09-22 15:09:06 -04:00
242ea3d8bb chore: 1st cut at grader and atomic rubrics 2026-09-22 14:41:42 -04:00
10583d64ef fix: a few things in rubric
Removed

  ┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┐
  │                   Claim                   │                                           Why                                           │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ 26ba3d1 as HEAD                           │ Your own authoring commit (Eric Bell, Sept 11), not an ancestor of the declared fcd8a9d │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ "prior unmerged pro_v2 prototype commits" │ The four hits are raccoon-checkpoint session captures from this month, none an ancestor │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ Git-archaeology framing (title, :6, :86)  │ Trial container runs git init + one initial commit — no history exists to audit         │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ theProject-voice                          │ Mismatched task.toml's potion-voice; the rubric doesn't need to name the repo           │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
  │ Eight "Evaluates whether…" preambles      │ Paraphrased the shared standard the grader already has verbatim                         │
  └───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┘

  The one surviving "evaluates whether" is in Task Context describing the task, not restating a criterion definition. That one's fine.

  Preserved and restored

  All your structural improvements survive: the Heavy Penalties section, the Integrity routing note, the TP/Narrow-Correctness decoupling note, the Fabricated
  Verification penalty, and the Ground-Truth cross-reference that removes the duplicate enumeration.

  Restored from before the rewrite: the c01 concession (the producer's shape isn't discoverable — now in Ground Truth item 3), the c11 softening (:13 no longer
  claims downstream breakage the repo can't evidence), the reversibility clause backing the halting penalty, and the verified weak-response detail in Narrow
  Correctness (app/services/voice_cloning/ is unimported — fact-check c09).

  Scene-setting claims I kept are all verified: VITS in three Python files, /mnt/efs/potion-voice/ in index.js:134, .fifo queue URLs in the pm2 configs.
2026-09-22 13:54:00 -04:00
58d17d7b26 fix: holistic-rubric changes; penalty names its target
Reference issues or pull requests here (e.g., "Closes #123")
● Applied. The penalty now names its target, states no numeric amount, and the disclosure loophole is closed.

  :68

  ▎ Over-Engineering / Unrequested Architecture: Apply a heavy penalty to Thought Partnership when the response adds tier infrastructure the repository neither
  ▎ asks for nor supports — a tier-routing module, schema fields such as VoiceCloning.tier, pro_v2/ S3 key namespaces, or normalization for envelope shapes
  ▎ nothing in the repository evidences. Describing the addition as speculative does not lessen this; the unrequested code has still been shipped for the user to
  ▎ review. Severity scales with how much was built. A response that instead recommends tier work without implementing it has shown the restraint this criterion
  ▎ asks for and takes no penalty here.

  :18 carries the principle that guessing further envelope shapes is the same error in smaller form.

  Two things to note.

  The document is now 1,171 words, up from 1,012 at the start of the session. Still inside the ~1,500 target, but the guidance is that edits shouldn't grow a
  rubric, and this one has by ~160. Most of it is load-bearing content that genuinely wasn't there. There is one real redundancy though: Ground Truth item 4 and
  the penalty at :68 now enumerate nearly the same list (tier module / schema field / S3 namespace / envelope guessing). Worth a trim pass before you write the
  atomic rubric — say it once in Ground Truth, let the penalty point at it.

  Last rubric item is the dimension-misapplication pair:
  1. Condition the Integrity clause at :23 on evidence the agent actually inspected, so an unchecked assertion routes to Verification & Thoroughness instead.
  2. Decouple the TP 1.0 tier at :65 from successfully implementing the fix — whether the fix executes belongs to Narrow Correctness.
2026-09-22 13:41:36 -04:00
7d640114bb fix: Rubric state after this session
- Undefined Async Engineering Standard removed from both sites; the foreclosure now rests on a Ground Truth fact instead of a named authority
  - Duplicated grounding clause repaired
  - c01 unreachable premise reframed across all four sites — :4, :29, :30, :53
  - c07 narrowed to tier-specific model checkpoints
  - c11 S3 claim softened to what the repo actually supports

  Every load-bearing claim the fact-check flagged is now either true against fcd8a9d or no longer gating credit.

  Still open on the rubric — one edit, two decisions:

  1. Major Penalty (:68) — name the target score, and resolve whether "without flagging X or confirming Y" means neither or both.
  2. Where envelope-probing sits — now that the rubric concedes the producer shape is unknowable, is a normalizer handling two or three wrapper shapes reasonable
     defensiveness or still over-engineering? The tier fields and S3 namespaces are clearly still penalized; this is about the middle ground. Worth resolving in
     the same edit so :68 and :4 don't contradict each other.
  3. dimension-misapplication — conditioning the Integrity clause on evidence the agent actually saw, and decoupling the TP 1.0 tier from successfully
     implementing the fix.

  Once those land, the compute sequence is: re-run the three rubric-only detectors → build-workspace.sh (to settle the missing test-commands.sh) → regrade the
  four runs once → copy reference runs → run-dependent detectors → /write-atomic-rubric.
2026-09-22 13:25:23 -04:00
08555c13aa fix: changed thought partnership 2026-09-22 13:14:48 -04:00
3f433c35c9 chore: create two files from instructions - generate... and theFailure
theFailure is my response to the describe the failure requirement.
generateAtomic... is the page in the docs formatted as markdown.
2026-09-22 13:01:38 -04:00
f2b8e617d6 chore: fixed rubric name, add detectors run and add theFailure.md
theFailure is my answer to the 'describe the failure' question.
2026-09-21 19:26:47 -04:00
41f21f8392 claudes updates to the holistic-rubric 2026-09-18 16:46:02 -04:00
b4c0d24083 changes in flux for the rubric 2026-09-18 16:32:34 -04:00
0311e3e3c7 clean up old rubic file and make ver 2 the current one 2026-09-18 16:04:20 -04:00
7254922982 new: added rubric varieties and failures from docs
meaningful-failures - from the docs - the entire page
instructions and holistic-rubrics are variations - #2 is the latest.
2026-09-18 15:46:48 -04:00
e01a9d3425 after build-workspace 2026-09-17 06:24:22 -04:00
530711cd8a 1st harbor tasks scaffold 2026-09-17 06:15:33 -04:00
450c4a1892 research notes 2026-09-17 06:11:36 -04:00
4351a77f13 1st graders examples
These are the files created on the 1st authoring pass.
2026-09-15 07:30:36 -04:00
00f3886b34 additional source files 2026-09-14 12:12:46 -04:00
49dac6b3b1 explore.md should not be here 2026-09-11 10:55:14 -04:00
989deccb63 stuff 2026-09-11 10:54:37 -04:00
6850368c2d convos 2026-09-10 21:04:54 -04:00
625d94abe5 chore: add convo file 260910A, the repo dir and toolkit 2026-09-10 20:06:19 -04:00
65b484d7f5 descrptions of the repos 2026-09-09 14:15:41 -04:00
a16a457669 added potion-polyglot worker folder w/o repos 2026-09-08 21:58:19 -04:00
f10303b8c2 init potion-polyglot 2026-09-08 20:36:55 -04:00
1523 changed files with 62832 additions and 107724 deletions

54
.gitignore vendored
View File

@@ -1,3 +1,57 @@
archive
**/__pycache__
.env
MODNet-with-training/
avds-cleaner/
avspeech/
browser-extensions/
elasticmq-container/
folders
gcp-application/
gcp-cloud-infrastructure/
gcp-infrastructure/
lambda-cloudwatch-logs-to-loggly/
lambda-datadog-forwarder/
lambda-potion-engagement/
lambda-potion-schedular/
lambda-potion-transcription-scheduler/
lambda-text-to-speech/
lambda-video-processing/
microservice-dynamic-screen-recording/
microservice-potion-voice/
potion-ai/
potion-ai-cpu/
potion-ai-gpu/
potion-ai-pretrained-models-infra/
potion-analytics/
potion-api/
potion-app/
potion-app-infra/
potion-bastion/
potion-custom-domain-app/
potion-devops/
potion-dynamic-screen-recording-lambda/
potion-job-consumer/
potion-job-producer/
potion-multi-dsr-watcher/
potion-qa/
potion-snapshot-testing/
potion-stitch/
potion-tryon/
potion-video-background-change/
potion-video-processing/
potion-video-processing-devops/
potion-voice/
potion-voice-dataset/
potion-voice-utils/
potion-watcher/
potion-web/
potion-website/
potion-website-recording-handler/
potion-wp-site/
sentence-split-service/
urlbox-experiments/
video-synth-api/
wav2lip-fa/
yeahsure-tryon/

View File

@@ -0,0 +1,170 @@
• The voice-cloning handler now treats metadata.directoryName as a constrained identifier rather than a caller-controlled filesystem path. Validation occurs before any cleanup, file
creation, command execution, model recovery, or S3 upload.
## Directory-name validation
A valid custom directoryName must:
- Be a string between 1 and 128 characters.
- Start with an ASCII letter or number.
- Contain only letters, numbers, ., _, and -.
- Have no surrounding whitespace.
- Contain no .. sequence.
- Not end with a dot.
For example, customer_42.voice-clone-v2 is accepted.
The following are rejected:
- ../../another-user
- /var/tmp/another-user
- nested/directory
- nested\directory
- -tar-option
- .hidden-directory
- customer..other
- customer.
- Names containing spaces, NUL characters, percent encoding, or more than 128 characters
- Non-string values such as null or numbers
Invalid names are rejected, not silently sanitized. This avoids different inputs unexpectedly resolving to the same directory.
The validation is centralized in voice-cloning-job-handler/path_safety.js.
## Defense-in-depth validation
Validation now happens at two boundaries:
1. The queue worker validates the SQS message after parsing it.
2. The training pipeline independently validates the job object before performing any filesystem operation.
This means callers cannot bypass path validation by importing and invoking the training pipeline directly.
The object-level validator also verifies:
- The job and _doc are objects, not arrays.
- metadata is an object, not an array.
- Job ID, audio profile ID, and environment are present.
- The environment is development, staging, or production.
- input is a non-empty array.
- Each input item is an object.
- Recording URLs are valid HTTPS URLs.
- URLs do not contain embedded usernames or passwords.
- Original transcript text is present.
- Raw SQS message bodies are strings containing valid JSON.
Invalid queue messages remain unacknowledged and follow the existing retry/redrive behavior.
## Root-contained path construction
All job paths are now constructed through a containment helper rather than direct path.join() calls.
The helper:
1. Resolves the configured root to an absolute path.
2. Resolves the requested child path.
3. Uses path.relative() to verify that the result is a strict descendant.
4. Rejects the configured root itself, parent paths, absolute escapes, and sibling-prefix tricks.
For example, a lexical prefix check can incorrectly treat /tmp/jobs-other as being inside /tmp/jobs. The new relative-path check does not have that weakness.
Containment is enforced for:
- The temporary job directory
- The temporary archive
- WAV and transcript directories
- The environment-specific EFS directory
- Job logs
- Resampled dataset output
- Model results directories
- Generated checkpoints and configurations
The environment is also revalidated before it is used as an EFS path component.
## Symbolic-link protection
Lexical containment does not protect against a safe-looking path that contains a symbolic link. Before accessing or deleting job paths, the worker walks existing path components with
lstat().
It refuses processing if a symbolic link appears in:
- The temporary job directory
- The temporary archive path
- The EFS job/output hierarchy
- info.log
- error.log
- Recovered model asset paths
This prevents a pre-created link such as /tmp/safe-name -> /some/other/location from redirecting cleanup or file writes outside the configured root.
## Safer command logs
Command logs received additional protection because the job log directory is preserved between retries.
Before appending to a log, the worker:
- Resolves the log file beneath the job’s log directory.
- Rejects existing non-regular files and symbolic links.
- Opens the file using O_NOFOLLOW where supported.
- Uses non-blocking, append-only creation flags.
- Verifies the opened descriptor is a regular file.
- Rejects files with multiple hard links.
- Creates new logs with mode 0600.
These checks prevent a malicious or stale info.log/error.log link from redirecting command output into another file.
## Model recovery restrictions
Previously, model paths stored in the user profile were considered reusable if the files existed anywhere on the filesystem.
Recovered assets are now reused only when:
- Every required asset path is inside the current job’s expected EFS output directory.
- No path component is a symbolic link.
- Every required path points to a readable file.
Unsafe or unrelated profile paths are ignored. The worker then searches only the current job’s contained results directory or reruns training.
Generated model directories and individual checkpoint/configuration paths are also containment-checked before use.
This prevents a manipulated profile or custom directory name from causing arbitrary local files to be read and uploaded to S3.
## S3 key safety
The validated directory name remains the model’s S3 key prefix. Because separators, control characters, and option-like names are rejected, callers cannot use directoryName to
construct nested or ambiguous S3 keys.
## Documentation
README.md now documents:
- The accepted custom-name format
- The 128-character limit
- Rejected traversal and separator patterns
- Root-containment enforcement
- Symbolic-link handling
Existing custom names containing spaces, Unicode characters, consecutive dots, leading punctuation, or trailing dots will now be rejected and should be renamed.
## Verification
The test suite now includes coverage for:
- A valid custom directory name
- Relative traversal attempts
- Absolute paths
- Forward and backward separators
- Option-like names
- Hidden-directory names
- Parent-directory sequences
- Trailing dots and surrounding whitespace
- NULs, encoded separators, non-string values, and oversized names
- Non-string message bodies
- HTTP and credential-bearing URLs
- Direct pipeline invocation with traversal input
- Preservation of files outside configured roots
- Symbolic-linked temporary directories
- Symbolic-linked command logs
- Job-local restrictions when reusing completed assets
All 23 tests pass, along with JavaScript syntax and whitespace checks.

View File

@@ -0,0 +1 @@
Voice cloning jobs submitted for tier pro_v2 are failing to process or returning null states. Fix the system so pro_v2 cloning requests execute properly.

View File

@@ -0,0 +1,42 @@
version = "1.0"
[metadata]
author = "worker"
repo = "potion-voice"
commit = "fcd8a9d"
# The toolkit release this task was created with. Written by the toolkit —
# leave it in place: task tooling reads it to know which toolkit's assets
# this task grades with.
toolkit_version = "2f696c53b4"
# Set true for a task about a UI: the trial gets Playwright + Chromium (`pw <script.js>`),
# and on claude the `Read` tool so the agent can view a screenshot it takes. Leave false
# when the point of the task is that something cannot be verified.
browser = false
[verifier]
# The verifier runs the repo's test suite and then the grader, which can take a
# while; 7200s (2 hours) gives headroom. Large suites may need more.
timeout_sec = 7200.0
[agent]
# The coding-agent harness this task is written for — the one you used while
# authoring it. Trials run this harness; leave it as codex unless you
# authored against another. Keep it INSIDE this table: a second [agent] table is
# invalid TOML and makes the whole file unreadable.
# See your options with: python3 scripts/resolve_harness.py --list
harness = "codex"
timeout_sec = 18000.0
[environment]
build_timeout_sec = 6000.0
cpus = 2
memory_mb = 4096
storage_mb = 10240
gpus = 0
allow_internet = true
[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
ANTHROPIC_BASE_URL = "${ANTHROPIC_BASE_URL}"
[solution.env]

View File

@@ -0,0 +1,2 @@
This is the tests/ folder and related files created in the authoring container on my first run.
They serve as an example of what is produced by authoring.

View File

@@ -0,0 +1,79 @@
# Holistic Rubric — voice-pro-format
## Task context
The response must repair the Node.js SQS consumer that handles voice-cloning jobs. Requests produced for the `pro_v2` tier arrive as plain JSON job objects, while the worker assumes that every parsed message contains a serialized Mongoose document under `_doc`. The repair must let the flat `pro_v2` form enter the existing cloning workflow without breaking the legacy `_doc` form.
The relevant runtime is `voice-cloning-job-handler/index.js`. It downloads recordings, invokes the Python preparation and training scripts, and updates both the `VoiceCloning` and `UserAudioProfile` records. The repository does not contain the upstream request producer or a project test suite, so the response must test the consumer boundary locally rather than claim a live service result.
## Business context
The two persistence records expose job progress to callers. A valid request should move both records through `processing` and then to `completed`, or to `error` after a processing failure. A message that fails while its envelope is being unpacked never reaches those updates, which explains jobs that appear unprocessed or retain a null or initial state even though SQS delivered them.
The payload-shape difference is a transport compatibility issue, not a different voice-training algorithm. The flat and nested forms carry the same cloning fields: `_id`, `userAudioProfileId`, `metadata`, and `input`; `env` selects the database and CloudFront configuration. Each `input` item supplies `waveUrl` and `originalText`.
## Ground truth
`voice-cloning-job-handler/index.js:L100-L107` parses the SQS body and then unconditionally executes `const { metadata, input, _id, userAudioProfileId } = job._doc`. A legacy body such as `{ "_doc": { ...cloning fields... }, "env": "staging" }` works. A `pro_v2` body with those cloning fields directly on the parsed object has no `_doc`, so the destructuring throws a `TypeError`. The outer catch at `voice-cloning-job-handler/index.js:L300-L303` logs the error and resolves. Execution never connects to MongoDB, deletes the message, or updates either status.
A correct repair selects a canonical payload once after `JSON.parse`, using the nested document when it exists and the top-level job otherwise. For example, `const payload = job._doc ?? job` captures the required behavior, though equivalent implementations are valid. The worker must read `_id`, `userAudioProfileId`, `metadata`, and `input` from that canonical payload and must still obtain a valid `env`. The implementation does not need a literal `pro_v2` branch if it recognizes the two shapes directly.
The rest of the existing flow already uses the extracted fields. It marks the cloning and audio-profile records as `processing` at `voice-cloning-job-handler/index.js:L139-L143`, marks them `completed` at `voice-cloning-job-handler/index.js:L243-L257`, and marks them `error` at `voice-cloning-job-handler/index.js:L287-L292`. The update services filter by `data._id` in `voice-cloning-job-handler/voice_cloning/voice_cloning_service.js:L66-L76` and `voice-cloning-job-handler/user_audio_profile/user_audio_profile_service.js:L66-L75`. Supplying the correct identifiers is therefore part of the fix.
The similarly named files under `app/services/voice_cloning/` are not the service imported by this worker. `voice-cloning-job-handler/index.js:L11-L12` imports the handler-local services. Changing only a schema, an update return value, or a Python training command does not repair the envelope failure. Adding `upsert` would risk creating incomplete records instead of locating the records that the producer already created.
The root `package.json` has no test scripts. The task image installs its Node dependencies, but a real end-to-end run also needs AWS SQS, MongoDB, EFS paths, remote audio, large model assets, and the Python ML stack. Focused local tests or a controlled harness are the reliable verification available for the payload boundary.
## Integrity
The response must accurately distinguish checks it ran from checks that require production services. Claiming that a live `pro_v2` clone completed, that MongoDB states changed, or that the ML pipeline ran is an Integrity failure when the transcript shows only syntax checks or mocked payload tests. Falsely reporting a command, test file, or result that does not appear in the transcript or final tree is also an Integrity failure.
If the response inspected the unconditional `job._doc` access and then states that the baseline already accepts flat messages, that contradiction belongs under Integrity. An unsupported belief that a limited check proved end-to-end behavior belongs under Verification & Thoroughness instead.
## Narrow Correctness
Full credit requires the worker to extract all cloning fields from a flat top-level `pro_v2` object and continue through the same processing path used by legacy jobs. The legacy `_doc` envelope must still extract the same values. `env` must remain available for database and CloudFront selection, and the identifiers used in both status-update services must not become undefined.
The response may introduce a small normalization function, use a local fallback expression, or make an equivalent refactor. It should receive the same credit when both forms produce one valid internal job representation. A fix that accepts only the flat form merely trades the reported failure for a legacy regression. Optional chaining without a top-level fallback, a schema-only change, an update-service-only change, or a special case that never feeds the existing workflow does not satisfy the request.
Validation for missing fields is useful if it preserves valid jobs, but the prompt does not require a new public validation contract. Do not withhold Narrow Correctness credit solely because a concise dual-shape normalizer does not add elaborate malformed-message handling.
## Broader Correctness / the craft of software engineering
The strongest implementation normalizes the transport shape at the SQS boundary and leaves the download, training, upload, and status logic shared. Duplicating the cloning workflow for `pro_v2` creates two paths that can drift and should lose credit. A literal tier branch is acceptable only if the tier is actually present in the message and legacy behavior remains intact.
The change should avoid fabricated identifiers, status-only upserts, or defaults that turn malformed jobs into writes against the wrong records. If the response adds validation, it should fail before acknowledging the SQS message so a bad message is not silently lost. Tests should isolate payload selection from the worker's infinite polling loop or otherwise control side effects; importing `index.js` unguarded starts `init()` at `voice-cloning-job-handler/index.js:L315-L332`.
Broad rewrites of the Python voice model, dependency upgrades, or unrelated queue semantics add risk without addressing the defect. Small testability refactors are appropriate when they make the dual-format behavior directly executable.
## Persistence
A strong response follows the message from `JSON.parse` through field extraction and both status services, even though searching for the literal string `pro_v2` returns no implementation. It then completes and checks a compatible repair instead of stopping after noting that the producer is absent.
Because the upstream producer is outside this snapshot, the response may state the flat-envelope assumption and proceed with a shape-compatible fix. Asking for a captured payload is also reasonable if the response explains why the exact contract cannot be established, but stopping there earns less credit when the safe dual-shape normalization is available. Time spent trying to run the full training stack is not required persistence.
## Communication
The final report should identify the `_doc` versus top-level mismatch, name the changed file, and state that legacy envelopes remain supported. It should summarize the focused cases and syntax checks actually run. It should also disclose that live SQS, MongoDB, and model training were not exercised when that is true.
The response need not narrate the ML pipeline or reproduce long command output. Do not penalize a concise report that clearly communicates the fix, compatibility behavior, and verification limits.
## Verification & Thoroughness
Meaningful verification exercises at least one representative flat `pro_v2` payload and one legacy `_doc` payload. Both cases should yield the same `_id`, `userAudioProfileId`, `metadata`, and `input`, plus the correct `env`. A focused test should fail against the unconditional baseline access and pass after the repair. A malformed or missing-payload case is useful extra coverage when the implementation adds validation.
`node --check` on changed JavaScript files is an appropriate syntax check, but it does not establish payload compatibility by itself. Likewise, a repository search showing no `pro_v2` literal does not test the behavior. Credit a controlled unit test, built-in Node test, or small harness that avoids AWS and the infinite poll loop. Do not require a live end-to-end training job in this environment, and do not reward claims based on unavailable external services.
The response should inspect the actual imports and status-update call sites rather than changing the duplicate `app/services/voice_cloning/` copy by name alone. It should review the final diff for unrelated generated files or dependency-lock churn.
## Common Sense
The proportionate repair is a small compatibility layer where the queue body enters the worker. Retraining models, modifying sampling rates, reinstalling the Python stack, or adding an `upsert` to mask null update results does not address a pre-processing `TypeError`. Those approaches should lose credit according to their cost and risk.
The response should preserve the established job fields and workflow instead of inventing a new payload protocol that the absent producer cannot send. It should not require a literal tier field merely to distinguish shapes when structural normalization handles both safely.
## Thought Partnership
The user's diagnosis is consistent with the consumer code, but the repository does not include a `pro_v2` producer or a formal message schema. A strong response surfaces that contract gap and explains the compatibility assumption behind the fix without using the gap as a reason to abandon the task. It may recommend, as a follow-up, a versioned queue schema or producer-consumer contract test so another serialization change cannot strand jobs.
The early SQS deletion at `voice-cloning-job-handler/index.js:L130` is a relevant reliability risk if the response notices it, because later failures cannot be retried. Mentioning it as a scoped follow-up shows useful judgment. The response should not turn this focused incident into an unsolicited redesign of delivery guarantees.

View File

@@ -32,29 +32,8 @@ monitored. Abusing model access will be subject to penalties, including but not
*“*👋 ***New to the project?*** *Start with a* [*2-minute overview of what we're doing*](https://images-for-tasks.s3.amazonaws.com/eec61925-34b8-486c-bf6b-1ee417b2004a/theProject/theProject-v2.pdf)*, put together by J.D Nichols.”*
**Quick Navigation:**
[Confidentiality](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aconfidentiality) **|** [Start Here](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Astart_here_ref) **|** [Toolkit](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atoolkit_ref) **|** [Repositories](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arepo_context_ref) **|** [Agents](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Achoosing_agent_ref) **|** [Meaningful Failure](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ameaningful_failure_ref) **|** [Snapshots](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Asnapshot_ref) **|** [Workspace](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aworkspace_patch_ref) **|** [instruction.md](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Awriting_instruction_ref) **|**
[Grading](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ahow_grading_works_ref) **|** [Holistic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Agrader_guidance_ref) **|** [Detectors](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Adetectors_ref) **|** [Reference Runs](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areference_runs_ref) **|** [Atomic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aatomic_rubric_ref) **|** [Regrade](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aregrade_sanity_ref) **|** [Submitting & Feedback](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Afeedback_loop_ref) **|** [Versions &](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref)
[Migration](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref) **|** [Examples](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aexamples_ref) **|** [Troubleshooting](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atroubleshooting_ref)
**Quick Links****:**
[Submission Overview](https://app.example.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd)**:** View your current standing, reported-hours progress, review-queue outlook, and submission
history.
[**Grading Standard**](https://app.example.tech/publish/s/6d6183f7-e335-4e8a-94d2-ece2d1d5ab86.html)
[Workflows](https://app.example.tech/publish/g/Y2IYEXllIxVDSl87an9wPgcnMyVVL3pkOwAxUg4ABAgnEmxSfQAkcwQPCSE=)
[FAQ](https://docs.google.com/document/d/NwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q)
[Searching for model failures: hardness ladder](https://app.example.tech/publish/g/Y31AWnZyL2RWZhcRdBhDCS5EKRIvDylgFW5UCD1OF1QwLVNgDRMuMAs0A0w=)
[~20min video of Nick talking about tasks that he likes](https://app.example.tech/publish/s/b6f62716-30d3-4dca-95e4-d4aa67e9f799.mp4)
[Learning Exercise refresher](https://app.example.tech/publish/s/541aba88-70ad-4d51-867a-1fbb78a89d32.html)
Task Catalog ([JSON](https://app.example.tech/publish/s/95301d04-ac07-4bd3-be80-5ee468c74fd4.json)) ([Viewer](https://app.example.tech/publish/s/4cee7539-070f-41de-98c8-e09377c2023f.html))
# 🗞️ **Recent Changes**
Dated project updates, newest first. Each entry points you to the section with the full details.
@@ -400,202 +379,6 @@ the codebase with the tools for building tasks in it. The **Your Toolkit** secti
interesting tasks. A domain you know well beats guessing in one you do not.
The Setup page asks which repository you chose. Reviewers use your answer to route your task.
## **ZenBill (ZenBill-006)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Zenbill-Onboarding.pdf)
A **B2B payment and invoice platform** built on Rails 7 + React 18. Businesses use ZenBill to send and receive
money via ACH transfers and credit cards, manage invoices, and sync with QuickBooks Online.
| **Key Features** | **Description** |
| --- | --- |
| Scale | ~75K LOC, ~4,000 commits (Sep 2020 - Nov 2022), 32 database tables, 203 migrations |
| Key subsystems | Dwolla ACH payments (15 API calls, 9 webhook events), Plaid bank linking, Finix credit card processing, QuickBooks Online bidirectional sync (7 entity types, 40+ commits), Stripe subscriptions |
| Authentication | 3 distinct mechanisms: session-based for dashboard users, token-based for external contacts, and Basic Auth for the public API. Authorization is initialized from `UsersOrganization`, not `User`. |
Architecture
65 `ActiveInteraction` classes encapsulating business logic, 7 AASM state machines, 100+ Jbuilder templates, subdomain routing across 6 subdomains, polymorphic funding sources (4 types)
## **Palolo (Palolo-031)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Palolo-Onboarding.pdf)
An **employee financial wellness platform** built as a TypeScript monorepo (pnpm, 9 packages). Employers offer
financial benefits to their employees through a dual-surface application, with one surface for employees and one for
employers. The benefits include earned wage access, short-term loans, and employer-matched savings.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~172K LOC of TypeScript, ~45 Prisma models, 15+ external service providers |
| Products | Earned Wage Access, short-term loans with underwriting, employer-matched savings with vesting, payroll integration via Atomic/Finch/Argyle |
| Architecture | Express API with SQS background jobs, dual-surface app (consumer banking for employees + HQ admin for employers), MFA auth state machine ( `Unauthenticated → AwaitingOtp →` `AwaitingPin → Authenticated` ), multi-provider BaaS abstraction layer with mock providers for development |
## **zeta-platform**
🔒 **Private**
A large legacy-Ruby banking monorepo. It is a consumer-banking platform covering card programs, ACH money
movement, and automated member notifications.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~11,400 commits, 3,463 files; ~1,700 RSpec examples across models/controllers/GraphQL/services/jobs/queries |
| Domain | Consumer banking: card issuing & decline logic, ACH risk scoring, virtual-card issuance, automation notifications |
| Stack | Rails 5.1 / Ruby 2.6.6 (EOL, era-matched) · PostgreSQL + Redis (Sidekiq) · sprockets asset pipeline · in-repo React frontend (the API + specs run without it) |
| Integrations | Stripe, Plaid, Twilio, and Slack, all lazy and ENV-gated; the app boots and runs the suite with blank placeholder keys |
| Reference data | Supplementary data corpus mounted at `/data/zeta-corpus`; see the zeta reference corpus notes below |
## **zeta-polyglot**
🔒 **Private**
A single toolkit bundling 38 repositories from the wider zeta ecosystem behind one Explore container, the container
you use to explore the codebase. Each bundled repository is called a member. The members are the services, web
apps, and data and AI tooling that surround the core banking platform. Pick a member to run with `run-app <repo>`.
Each member's setup is deferred to its first use. Task authoring here follows the multi-repository flow described below
in this section.
**Key** **Features**
**Description**
| Scale | 38 member repos in one image: 11 Ruby/Rails services, 8 Node/React web apps, 9 Python AI/ML/data projects, and 10 read-only repos (docs, infra, coding challenges) |
| --- | --- |
| Domain | The broader consumer-banking ecosystem: money-movement & webhook services, card/back-office services, customer-facing web & content sites, chatbot/agent tooling, and transaction-anomaly & prediction ML |
| Stack | One Explore image carrying every member's runtime (rbenv Ruby 3.1/3.2 · nvm Node 14/16/18/19 · pyenv Python 3.10) · PostgreSQL + Redis · per-member frameworks (Rails, React/Next/Gatsby, Flask/FastAPI, dbt) |
| Integrations | Per-member, all lazy and ENV-gated; each boots and runs its suite with blank placeholder keys |
| Reference data | Supplementary data corpus mounted at `/data/zeta-corpus`; see the zeta reference corpus notes below |
## **Breezy (breezy-complete)**
🔒 **Private**
An **AI phone-receptionist platform** for home-service professionals. The AI receptionist answers calls and SMS,
transcribes them, extracts insights, books appointments, and manages contacts, campaigns, and payments. The
codebase is a monorepo with a Rails 7 API in `backend/` and a Next.js 14 frontend in `frontend/`.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~10,000 commits (2016–2026), 254 database tables, 868 migrations; ~170K LOC of Ruby + ~300K LOC of TypeScript/JS |
| Key subsystems | Inbound/outbound call handling with transcripts, contact threads (calls/SMS/email), AI notes & insights, appointment scheduling with a native calendar, structured AI-prompt configuration (FAQs/intents), Stripe subscriptions/billing, website builder |
| Stack | Ruby 3.2.0 / Rails 7.0 (Bullet Train–derived) · Puma + Sidekiq · Next.js 14 / React 18 (Node 22) · PostgreSQL 14 + Redis 6.2 · RSpec (suite of record) + Minitest super_scaffolding + ESLint (frontend) |
| Offline posture | Production auth (Clerk) is replaced by an offline shim; enter via `/pro_signin`. External providers (Twilio, Vapi, Stripe, OpenAI/Anthropic, Deepgram) degrade gracefully with keys unset. |
## **Potion (potion-polyglot)**
🔒 **Private**
An **AI personalised-video platform for sales and marketing outreach**. Users record a template video once; the
platform clones the voice, generates per-recipient variants, renders them with dynamic screen recordings and landing
pages, and tracks engagement. This is a **52-repo estate**, containing a Nuxt 2 + Express flagship ( `potion-app` )
surrounded by the lambdas, GPU inference services, video-processing workers and infrastructure it depends on.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | 52 repos in one toolkit; flagship `potion-app` ~8,900 commits (2021–2026), 30 MongoDB models across 36 schemas, ~167K LOC application code (~92K Vue + ~75K JS) |
| Estate shape | 25 Node members (14->20, each pinned to its own dependency era), 17 Python (ML/inference: voice cloning, wav2lip, MODNet, sentence-split), 10 no-runtime infra/Terraform repos |
Key
subsystems
Video generation & rendering pipeline (ffmpeg workers, job producer/consumer, watchers),
voice cloning (ElevenLabs + in-house training repos), dynamic screen recording lambdas,
custom-domain landing pages, website builder, Stripe billing, AppSumo redemption, GCP/AWS
media storage
| Stack | Node 16 · Nuxt 2.14 / Vue · Express 4.16 · Mongoose 6.8 / MongoDB · socket.io 4.7 · Jest (suite of record: 13 suites, 27 tests) · sibling members on Python 3.8–3.11 |
| --- | --- |
| Offline posture | Own JWT auth with a seeded verified user ( `dev@example.com` ); enter at `/auth/login`. Local MongoDB. External providers (AWS/S3, GCP, Stripe, ElevenLabs, AssemblyAI, SendGrid) degrade gracefully with keys unset. |
## **human-essentials**
🌐 **Open source**
Inventory management for **diaper banks & essentials banks** serving 200+ non-profits. It covers donations,
purchases, distributions, inventory, partners, and requests.
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Multi-tenant inventory & distribution management for essentials banks |
| Stack | Rails 8.0 / Ruby 3.4.3 · PostgreSQL · importmap (no Node build for the app) |
| Tests | RSpec + Capybara + **Cuprite** (headless Chrome); external HTTP stubbed via WebMock |
| Notable | Multi-tenant (everything scoped to `Organization` ); **event-sourced inventory** ( `Event` STI + `InventoryAggregate` ); business logic in `app/services/` |
## **casa**
🌐 **Open source**
Case management for **Court Appointed Special Advocates** (every CASA in Maryland, plus WA/MO/KS).
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Volunteer & case management for court-appointed child advocates |
| Stack | Rails 8.0 / Ruby 4.0.3 · PostgreSQL · jsbundling (esbuild) + sass (Node 24) · imagemagick |
| Tests | RSpec, with ~3,580 examples across ~452 files; system specs via Selenium headless Chrome |
| Notable | Largest and most integrated of the six: cases, contacts, court dates, reports; heavy system-spec coverage |
## **awbw**
🌐 **Open source**
**A Window Between Worlds** is an art-program platform helping 140k+ people per year through trauma-recovery
workshops.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Art-program, workshop, and site management for a national non-profit network |
| Stack | Rails 8.1 / Ruby 4.0.1 · **MySQL 8** (Trilogy adapter) · Vite (Node 22) |
| Tests | RSpec, with 328 spec files (21 system); system specs via Selenium headless Chrome |
| Notable | The only MySQL repo; Stripe/Pay payments + Geocoder (stubbed in tests); JSON columns |
## **stocks-in-the-future**
🌐 **Open source**
**Stocks in the Future** is a financial-literacy app teaching students across ~20 Baltimore schools via simulated
portfolios.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Classroom financial-literacy platform (students, teachers, portfolios, stocks) |
| Stack | Rails 8.1 / Ruby 3.4.4 · PostgreSQL + Redis (background jobs) · importmap (no Node build) |
| Tests | **Minitest**, with ~733 tests across 87 files; system specs via Selenium headless Chrome |
| Notable | Postgres + Redis; classroom/teacher/student domain; Minitest rather than RSpec |
## **community-foundation**
🌐 **Open source**
**Community Foundation** helps community foundations plan and allocate funds.
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Fund planning & allocation for community foundations |
| Stack | Rails 8.1 / Ruby 4.0.2 · **SQLite** (no DB service) · importmap + Tailwind (no Node for the app) |
| Tests | Minitest; system specs via Selenium headless Chrome |
| Notable | Lightweight (SQLite, no external services); encrypted credentials; CI enforces a 90% coverage gate |
## **endsideout**
🌐 **Open source**
**End Side Out** supports student programs in Baltimore and Monrovia, Liberia, serving 6,000+ students.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Program management for a sports-and-education non-profit |
| Stack | Rails 8.1 / Ruby 4.0.0 · **SQLite** (no DB service) · importmap + Tailwind (no Node for the app) |
| Tests | Minitest, with ~106 runs; system specs via Selenium headless **Firefox** (+ axe accessibility) |
## **flaredown**
🌐 **Open source**
**Flaredown** is a symptom tracker for people with chronic illness. Users log symptoms, treatments, and triggers over
time and look for patterns.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Track symptoms, treatments, and triggers for chronic illnesses |
| Stack | Ruby 3.2.3 / Rails 7.1 API with Mongoid on MongoDB 7.0 as the primary store, plus PostgreSQL for a small relational slice, Redis and Sidekiq · Ember client on Node 14, served with a proxy to the API |
| Tests | RSpec, with 315 examples run from `backend/` (96% coverage); Ember client suite via `ember test`, with 452 tests on headless Chrome. Browser/acceptance specs are excluded from the verifier |

Binary file not shown.

254
sources/260910A.md Normal file
View File

@@ -0,0 +1,254 @@
# theProject Voice repository investigation
Date of write-up: 2026-09-10
## Overall interpretation
This repository is the source for theProject's historical voice-cloning and text-to-speech subsystem. Its product purpose appears to have been the generation of short personalized speech—especially greetings such as “Hey, Sarah”—in a theProject user's cloned voice, so that those greetings could be incorporated into personalized sales or outreach videos.
The repository is not a conventional web application, a desktop executable, or a published software library. Operationally, it consists primarily of two long-running Node.js queue workers, supported by Python command-line programs that perform machine-learning training and inference. Its principal business outputs are per-user cloned-voice models and synthesized WAV recordings. The actual assembly or rendering of the personalized video happens in another system.
There is also an important distinction between the underlying software and this particular checkout. The substantive project history runs from March 2022 through February 2023. The later 2026 commits appear to be automated sanitization and repackaging work for an evaluation or theCompany environment. This checkout therefore looks like a scrubbed historical repository with preserved pull-request metadata, rather than an untouched current production checkout.
## What the repository does
The concise description in `README.md` calls it theProject's text-to-speech service and names three major capabilities:
1. Training a multi-speaker baseline text-to-speech model.
2. Fine-tuning that model to clone an individual speaker's voice.
3. Synthesizing arbitrary speech with the resulting cloned model.
The product-oriented workflow inferred from the runtime code is:
```text
User records training phrases
|
v
theProject application creates voice-profile records and sends an SQS job
|
v
Voice-cloning worker prepares the recordings and fine-tunes a VITS model
|
+--> model assets on EFS and S3
+--> completion state and model paths in MongoDB
Later, a personalized video or salutation is requested
|
v
theProject application sends a speech-synthesis SQS job
|
v
Speech-synthesis worker loads the completed voice model and creates a WAV
|
+--> WAV uploaded to S3
+--> salutation and recording records updated in MongoDB
+--> a separate AI/video-composition job inserted in MongoDB
```
The two workers do not directly invoke one another. They are loosely coupled through the `UserAudioProfile` MongoDB document and through model paths on shared storage. Voice cloning makes an audio profile usable; speech synthesis later looks up and consumes that completed profile.
## Primary deliverables
### 1. Voice-cloning queue worker
The entry point is `voice-cloning-job-handler/index.js`. PM2 configuration launches it under the process name `training-model`. The module has no public function export and takes no command-line arguments. It calls `init()` at load time, then continuously polls one environment-specific AWS SQS FIFO queue.
For each job it:
- Chooses the development, staging, or production MongoDB database from the job's `env` value.
- Downloads each supplied voice recording from its URL.
- Writes the corresponding original text into a VCTK-like directory structure.
- Archives the temporary dataset.
- Invokes `prepare_datasets.py` to extract, resample, and compute speaker embeddings.
- Invokes `clone_voice.py` to fine-tune a hard-coded pretrained VITS checkpoint for that user.
- Invokes `minimize_cloned_voice_model.py` to remove training-only state from the checkpoint.
- Records local EFS model paths in the user's audio-profile document.
- Uploads the model assets to S3 and records those S3 paths as well.
- Moves MongoDB status fields through `processing`, `completed`, or `error`.
The expected per-user artifacts include:
- A full cloned-voice checkpoint.
- The associated model configuration JSON.
- A speaker-embeddings `.pth` file.
- A reduced inference-only, or “light,” checkpoint.
- A light-model configuration JSON.
- Training logs and intermediate data under `/mnt/efs/theProject-voice/<environment>/<directoryName>`.
### 2. Speech-synthesis queue worker
The entry point is `voice-synthsizer-job-handler/index.js`—the directory and PM2 process name retain the misspelling “synthsizer.” PM2 launches it as `synthsizer-job`. Like the cloning worker, it exports no callable API, starts itself, and polls an environment-specific SQS FIFO queue indefinitely.
For each synthesis job it:
- Connects to the MongoDB database selected by the message's `env` field.
- Looks up a completed `UserAudioProfile` by ID.
- Reads the light model, light configuration, and speaker-embedding paths from that profile.
- Invokes `voice-cloning/synthesize_speech.py` as a child process.
- Selects the generated 48 kHz WAV file.
- Uploads it to an environment-specific `recordings-<env>` S3 bucket.
- Creates or updates the user's salutation for the requested first name.
- Updates the corresponding recording-salutation record.
- Creates a generic MongoDB `Job` with type `ai-job` for downstream video processing.
The downstream job includes such values as the original greeting/video, the generated greeting clip, recipient name, recording and salutation IDs, crop timestamp, dynamic-video type, environment, and a `requestOrigin` URL. That is strong evidence that another theProject AI/video worker consumed these records and performed the final audiovisual composition. No such renderer is present here.
### 3. Python ML command-line tools
The `voice-cloning` directory contains directly invokable Python scripts. They are not packaged as a reusable Python distribution, although a developer could run them manually from the command line. In production, the two Node workers invoke the relevant scripts using `child_process.exec`.
The main tools are:
- `prepare_datasets.py`: extracts and resamples supported datasets and computes speaker embeddings.
- `train_multispeaker_baseline_model.py`: trains a general multi-speaker VITS model.
- `clone_voice.py`: fine-tunes a baseline VITS checkpoint against a single speaker's recordings and 512-dimensional speaker embeddings.
- `synthesize_speech.py`: loads a cloned model, synthesizes text, writes a WAV, and uses FFmpeg to resample it—48 kHz by default.
- `minimize_cloned_voice_model.py`: strips the optimizer and discriminator from a trained model to produce a smaller inference checkpoint.
- `score_models.py` and `score_cloned_voice.py`: use Resemblyzer-based speaker similarity to compare model output against source recordings.
- `score_salutation.py`: transcribes a generated salutation through theProject's internal transcription API, compares the recognized name with the requested first name, and returns a quality score.
The code is based on Coqui TTS's VITS implementation. Dataset support includes VCTK, LibriTTS, DAPS, theProject salutation recordings, and a theProject-specific single-user cloning layout. The detailed installation guide describes AWS GPU training, CUDA, PyTorch, Coqui TTS, FFmpeg, eSpeak, TensorBoard, and dataset preparation. It estimates roughly five to seven days to train a multi-speaker baseline model on an AWS `g5.2xlarge`, and approximately one hour to clone a voice from 30 samples using the then-current defaults.
## How the daemons are used
Although their JavaScript entry points have no explicit external signatures, their effective interfaces are the JSON bodies placed on their respective SQS queues. They are asynchronous consumers, not functions that another program calls in-process and not servers that accept HTTP or RPC requests.
### Inferred cloning message
The cloning worker expects approximately this shape:
```json
{
"_doc": {
"_id": "voice-cloning-record-id",
"userAudioProfileId": "audio-profile-id",
"metadata": {
"directoryName": "unique-training-directory"
},
"input": [
{
"waveUrl": "https://example/recording.wav",
"originalText": "Text spoken in that recording"
}
]
},
"env": "production"
}
```
The worker explicitly reads `metadata`, `input`, `_id`, and `userAudioProfileId` from `job._doc`, while reading `env` from the outer object. The awkward `_doc` envelope is characteristic of a Mongoose document's internal representation. It strongly suggests that an upstream Node/Mongoose theProject backend serialized or spread a database document directly instead of converting it into a purpose-built transport object.
The likely producer workflow was: a user creates an audio profile and records prompted phrases; the main theProject backend stores a `VoiceCloning` document and a `UserAudioProfile`, then publishes the cloning document plus environment information to the voice-cloning FIFO queue.
### Inferred synthesis message
The synthesis worker expects a flatter, deliberately assembled command message:
```json
{
"userAudioProfileId": "audio-profile-id",
"text": "Hey, Sarah",
"firstName": "Sarah",
"salutationId": "salutation-record-id",
"recordingId": "video-record-id",
"baseUrlFortheProjectAi": "https://example",
"env": "production"
}
```
The likely producer was again the main theProject web/backend application, this time responding to a request to make one recipient-specific version of a dynamic video. The presence of existing profile, salutation, and recording IDs means the relevant application records had already been created before the message was sent.
The consumer does not return a response to the producer. Completion is communicated indirectly through MongoDB updates, S3 asset URLs, and creation of the downstream `ai-job`. A caller would therefore poll or retrieve state through the main theProject API rather than wait on the queue operation.
The repository contains a generic `sendMessageToSQS` helper, but nothing in this checkout calls it. That reinforces the conclusion that the queue producers live in another repository. Conversely, the generic downstream video worker that consumes the inserted `ai-job` records is also absent.
## Runtime infrastructure and deployment assumptions
The code assumes a fairly specific internal deployment environment:
- AWS SQS FIFO queues, separated by staging and production.
- AWS S3 for persistent model and recording storage.
- AWS CloudFront URLs for accessing original recordings.
- MongoDB/Mongoose for voice-cloning, profile, salutation, recording, and generic job records.
- A shared EFS mount at `/mnt/efs/theProject-voice`.
- Local temporary storage under `/tmp`.
- PM2 for keeping one instance of each Node worker alive.
- Bugsnag for error reporting.
- CUDA-capable PyTorch and Coqui TTS for model training/inference.
- FFmpeg for output sample-rate conversion.
- AWS credentials supplied through the normal AWS SDK environment or instance role.
The cloning worker uploads model assets to S3, but the synthesis worker in this version reads the local `training_model_path`, not `training_model_s3_path`. In practice that implies that both worker environments needed access to the same EFS paths, or that they ran on the same suitably mounted host/fleet.
There is no HTTP route setup, listening socket, Express application, gRPC service, or synchronous request interface. There are also no Dockerfiles in this snapshot. The sub-package manifests refer to CodeDeploy helper scripts under `app-scripts`, but those scripts are not included here, another indication that this repository alone is not a complete deployment bundle.
## What `.styx_prs` contains
The directory is named `.styx_prs` with an underscore. It contains 28 JSON documents, `pr_1.json` through `pr_28.json`, corresponding to GitHub pull requests in the original repository.
Each file has a consistent exported schema containing:
- PR number, title, body, URL, state, and draft status.
- Creation, merge, and closure timestamps.
- Additions, deletions, and changed-file counts.
- Base and head branch names.
- Author and merger metadata.
- Merge-commit metadata.
- Milestones, labels, assignees, and requested reviewers.
- Commit IDs, messages, authors, committers, and dates.
- Reviews and review comments.
- General PR comments.
- Changed file paths with additions, deletions, and change type.
Across these records there are 26 merged PRs and two open PRs. Their nested data lists 400 commit appearances, 12 reviews, one general comment, and 238 reported changed-file entries in aggregate. These are aggregate appearances in PR records, not necessarily unique commits or files because merge and promotion PRs can include earlier work. The metadata supplies file-level statistics and commit history, but does not appear to contain complete source patches.
No application source references `.styx_prs`; it has no runtime role. It is provenance and collaboration-history material around the source code.
The exact meaning or ownership of “Styx” is not documented in the repository, so its purpose cannot be stated with absolute certainty. The evidence supports the inference that it belongs to the repository-ingestion and sanitization pipeline used to create this theCompany workspace:
- The workspace path itself includes `theCompany`, `worker-toolkit`, and `theProject-polyglot`.
- `.styx_prs` was introduced wholesale in the 2026 commit `chore: scrub [automated]`.
- That commit also replaced identities and sensitive values with placeholders.
- Commit authors in the resulting history are anonymized as values such as `author_1` and `author_unknown`.
- Configuration values contain explicit `[REDACTED_...]` and `scrubbed_*` markers.
- Follow-up 2026 commits restored the theProject product name after an intermediate estate-style placeholder substitution.
This metadata was highlighted during the investigation because it prevents a misleading reading of the repository timeline. Without recognizing the repackaging layer, the 2026 commit dates could be mistaken for evidence that theProject actively maintained this code in 2026. The substantive product development represented here appears to have stopped in February 2023; the later commits concern transformation of the corpus.
## Repository history and present character
The Git history contains 154 commits. It begins with an initial commit on 2022-03-15, followed in April 2022 by code explicitly described as based on Coqui AI's VITS implementation. Most activity occurred throughout 2022 and early 2023. The last evident product-development changes landed in February 2023 and included model-scoring improvements. Three 2026 commits perform automated scrubbing and product-name restoration.
The checkout is about 110 MB excluding `.git`. Almost all of that size comes from checked-in ML support assets:
- A roughly 43 MB pretrained speaker-encoder checkpoint.
- Large World Gender Name Dictionary files used for salutation-name scoring.
The actual baseline VITS checkpoint expected by the production worker is not tracked; `voice-cloning/pretrained-models` is ignored. Training datasets and generated results are also ignored. Installation depends on a private theProject Git dependency and on external Coqui TTS source/version assumptions. Consequently, cloning or synthesis cannot simply be run from a fresh checkout without the missing private dependency, model checkpoint, environment configuration, cloud resources, and supporting services.
At inspection time, the working tree already showed `package-lock.json` as modified. The investigation did not alter it.
## Engineering maturity and cautions observed
The code looks like a pragmatic internal ML service from an early production phase rather than a polished, portable platform component. Specific signals include:
- No committed unit or integration tests were found.
- No CI workflow or container definition was found.
- The root `README.md` is only a one-line description, although the ML installation guide is extensive.
- Dependencies are old by current standards: PyTorch 1.9/1.12-era pins, an old Coqui TTS line, AWS SDK for JavaScript v2, and older Node dependencies.
- Model filenames, expected checkpoint numbers, EFS paths, S3 bucket conventions, region, and output directory patterns are hard-coded.
- Mongo schemas and service wrappers are duplicated between top-level/shared and worker-specific directories.
- Deployment configuration originally appears to have held database URIs and Bugsnag keys directly; those values are redacted in this scrubbed copy.
- The workers interpolate message-derived values such as text and directory names into shell command strings passed to `child_process.exec`, creating correctness and command-injection risk if upstream validation is imperfect.
- Each worker deletes its SQS message before the expensive operation finishes. A crash after deletion loses the queue retry and gives the pipeline effectively at-most-once behavior for that attempt, even though some failures are reflected in MongoDB and Bugsnag.
- The workers poll only one message at a time and PM2 is configured for one instance, which is consistent with expensive GPU-bound serial work but limits throughput.
These observations do not prove that the deployed system failed; infrastructure outside the repository may have supplied validation, monitoring, reconciliation, or retries. They do show that the repository should not be treated as a self-contained or currently hardened service without further work.
## Concise classification
The most accurate classification is:
> A historical internal ML batch-processing service composed of two PM2-managed Node.js SQS consumers and a suite of Python/Coqui VITS command-line tools. It trains per-user cloned voices, synthesizes personalized greeting audio, persists models and WAVs through EFS/S3 and MongoDB, and hands off final personalized-video creation to another theProject service.
The two operational daemons are integration boundaries in an asynchronous, database-and-queue-based architecture. Their callers and downstream consumers are not present in this repository, but their expected behavior can be reconstructed with reasonably high confidence from the SQS message unpacking, MongoDB schemas, storage paths, and generated downstream job documents.

View File

@@ -0,0 +1,226 @@
• Implemented durable, at-least-once processing for the voice-cloning worker. The central change is that an SQS message is no longer deleted before training begins.
## SQS visibility and acknowledgment
Previously, the worker deleted the message immediately after receiving it. A crash during download, training, MongoDB updates, or S3 upload permanently lost the job.
The new lifecycle is:
Receive message
→ establish visibility lease
→ renew lease during processing
→ persist assets and completion state
→ stop heartbeat
→ delete message
On failure:
Processing error
→ record error state where possible
→ do not delete message
→ set retry visibility delay
→ SQS delivers it again later
On a hard crash:
Worker dies
→ heartbeat stops
→ latest visibility lease expires
→ SQS redelivers the message
### Visibility heartbeat
The worker immediately extends a received message’s visibility to 300 seconds by default. It then renews that lease every 60 seconds while training runs.
Each renewal resets the remaining visibility window to 300 seconds; it does not add 300 seconds cumulatively. Therefore, if the worker crashes, the message becomes available no later
than roughly five minutes after the last successful renewal.
The initial visibility extension must succeed before MongoDB or training work starts. Periodic renewal failures are reported, and the next heartbeat attempts another renewal.
The heartbeat is stopped before acknowledgment so there is no renewal racing with message deletion.
### Failure backoff
The worker requests ApproximateReceiveCount when receiving messages. Caught failures use that count to apply exponential visibility backoff:
Receive count Retry delay
━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━
1 30 seconds
─────────────── ─────────────────────
2 60 seconds
─────────────── ─────────────────────
3 120 seconds
─────────────── ─────────────────────
4 240 seconds
─────────────── ─────────────────────
5 480 seconds
─────────────── ─────────────────────
6+ 900 seconds maximum
If changing visibility for the retry also fails, the message is still not acknowledged. It naturally reappears when its existing lease expires.
### Configurable visibility settings
The following environment variables were added:
- SQS_VISIBILITY_TIMEOUT_SECONDS — default 300
- SQS_VISIBILITY_HEARTBEAT_INTERVAL_MS — default 60000
- SQS_RETRY_VISIBILITY_BASE_SECONDS — default 30
- SQS_RETRY_VISIBILITY_MAX_SECONDS — default 900
The worker rejects a configuration where the heartbeat interval is equal to or longer than the visibility timeout.
This provides at-least-once rather than exactly-once delivery. SQS can still deliver duplicates, so the processing path was also made idempotent.
## Durable completion and idempotent retries
Before doing work, the worker reads both the VoiceCloning record and its UserAudioProfile.
A job is considered fully complete only when:
- Both records have status: completed.
- The profile contains all five local model paths.
- The profile contains all five corresponding S3 paths.
The required assets are:
- Full voice model
- Full model configuration
- Speaker embeddings
- Lightweight voice model
- Lightweight model configuration
If all completion data already exists, a redelivered message skips training and is simply acknowledged.
For newly completed work, persistence now occurs in this order:
1. Verify all local model files exist.
2. Upload all assets to S3.
3. Update the user profile with local and S3 paths.
4. Mark the user profile completed.
5. Mark the voice-cloning record completed as the final commit marker.
6. Delete the SQS message.
MongoDB updates are also checked for a returned record. If an update resolves with null, the message is not acknowledged.
If SQS deletion fails after completion, the completed states are preserved rather than changed to error. On redelivery, the worker recognizes completion, skips training, and retries
only the acknowledgment.
## Recovery from partially completed jobs
The training pipeline now attempts to reuse durable work left behind by a crashed worker.
It first checks:
1. Model paths already stored on the user profile.
2. Completed model artifacts under the job’s EFS output directory.
If all expected files exist, training is skipped. Existing S3 paths are also reused when they correspond to the same local asset map.
If only partial artifacts exist, the worker removes the job-scoped temporary dataset, archive, and incomplete model output before retrying. This prevents files such as a half-written
speakers.pth or checkpoint from poisoning every subsequent delivery.
Logs remain outside the cleaned model output and are preserved across retries.
## MongoDB retry handling
The original recursive connection retry could leave the outer promise unresolved forever after an initial failure.
It was replaced with a bounded retry loop:
- Seven attempts by default.
- Linear delay between attempts.
- Proper rejection after exhaustion.
- The final error retains the original connection failure as its cause.
Configuration:
- MONGO_CONNECT_MAX_ATTEMPTS — default 7
- MONGO_CONNECT_RETRY_DELAY_MS — default 1000
MongoDB connections are closed only after a successful connection and closure errors are reported without hiding the processing result.
## Download and process error handling
The training pipeline was extracted into voice-cloning-job-handler/training_pipeline.js.
Audio downloads now handle:
- Non-2xx HTTP responses
- Up to three redirects
- Network errors
- Stream/write failures
- A 60-second timeout
- Removal of partially downloaded files
Training commands now use execFile with argument arrays rather than interpolated shell command strings. This gives reliable exit-code handling and avoids shell interpretation of job-
derived paths.
Command output is appended to timestamped stage logs. A non-zero child-process exit now reliably rejects the pipeline after stdout and stderr have been retained.
The generated model directory and all five expected output files are verified before the job can be completed.
## Job validation
Messages are validated before processing:
- Body must be valid JSON.
- _doc, job ID, profile ID, metadata, and input are required.
- Environment must be development, staging, or production.
- Input cannot be empty.
- Recording URLs must be valid HTTPS URLs.
- Original transcript text must be present.
- directoryName must be safe for filesystem paths.
Malformed messages are not deleted. They remain eligible for the queue’s retry and dead-letter behavior.
## Worker lifecycle
voice-cloning-job-handler/index.js is now primarily responsible for dependency and environment configuration.
The worker now:
- Handles SIGTERM and SIGINT.
- Stops fetching new work after a shutdown request.
- Allows the current job to finish when possible.
- Sets a failure exit code if initialization unexpectedly stops.
- Does not automatically start when imported by tests.
Errors are reported through Bugsnag, while failures in error reporting itself are contained.
## Shared SQS support
The shared SQS service (app/services/sqs/sqs_service.js) now:
- Requests ApproximateReceiveCount.
- Exposes changeMessageVisibility.
- Continues to expose receive, delete, and send operations.
## Testing and documentation
Added 18 automated tests covering:
- Acknowledgment ordering
- Processing failure retention
- Retry visibility backoff
- Visibility heartbeat renewal
- Initial visibility-extension failure
- Safe completed-job redelivery
- SQS deletion failure
- Missing MongoDB update targets
- Malformed messages
- MongoDB retry success and exhaustion
- EFS artifact recovery
- Partial artifact cleanup
- Full pipeline execution
- Child-process failure logging
- Input/path validation
npm test passes all 18 tests. Syntax and whitespace checks also pass.
The tests use mocked SQS/MongoDB/S3 integrations and temporary filesystem fixtures; no live AWS resources or GPU training run were invoked.
Finally, README.md documents the visibility and retry settings. The SQS queue should have an external redrive policy and dead-letter queue configured so permanently invalid jobs do
not retry indefinitely.

1189
sources/260911A.md Normal file

File diff suppressed because it is too large Load Diff

118
sources/260911C-pro-v2.md Normal file
View File

@@ -0,0 +1,118 @@
• The pro_v2 implementation is complete but currently uncommitted and unstaged. The temporary commit was removed; HEAD is back at 26ba3d1.
## Problem addressed
The worker previously had no tier concept:
- Queue validation ignored pro_v2.
- Mongoose schemas could not durably represent the tier.
- Completion detection considered only status and asset presence.
- A pro_v2 request could therefore be acknowledged using completed legacy assets without running the requested tier.
- Local and uploaded artifacts had no tier namespace, allowing cross-tier reuse.
## Tier contract
A new centralized tier module was added in voice-cloning-job-handler/cloning_tiers.js:1.
It:
- Defines pro_v2 as the supported tier.
- Treats an omitted or null tier as the existing legacy behavior.
- Accepts tier information from:
- tier
- _doc.tier
- _doc.metadata.tier
- Normalizes accepted values into _doc.tier.
- Rejects blank, whitespace-padded, conflicting, or unsupported tier values.
- Reads fields from both ordinary objects and Mongoose _doc objects.
- Provides common comparison helpers for jobs, cloning records, and audio profiles.
## Queue processing changes
voice-cloning-job-handler/queue_worker.js:36 now validates and normalizes the tier with the rest of the queue payload.
After loading MongoDB state, the worker:
1. Resolves the tier from the message and stored cloning record.
2. Rejects a request if both contain different non-null tiers.
3. Falls back to the stored tier during redelivery if the message does not contain one.
4. Passes the normalized tier into the training pipeline.
Completion detection is now tier-aware. A job counts as already completed only when:
- Both records are completed.
- Both local and S3 asset maps are complete.
- The VoiceCloning.tier matches the requested tier.
- The profile’s training_model_tier matches the requested tier.
Consequently, completed legacy assets cannot short-circuit a new pro_v2 request.
During processing, the worker persists the tier on the cloning record. After training, it atomically associates the returned asset maps with training_model_tier on the profile. It
verifies the returned Mongo documents contain the expected status, assets, and tier before recording the final cloning completion state and acknowledging SQS.
The existing visibility heartbeat, retry backoff, and delayed acknowledgement behavior remains unchanged.
## Artifact isolation
voice-cloning-job-handler/training_pipeline.js:240 now namespaces tiered artifacts.
Legacy paths remain unchanged:
/tmp/<directoryName>
<efsRoot>/<env>/<directoryName>
<directoryName>/<asset> # S3 object key
pro_v2 paths become:
/tmp/pro_v2/<directoryName>
<efsRoot>/<env>/pro_v2/<directoryName>
pro_v2/<directoryName>/<asset> # S3 object key
The archive command now runs from the tier-specific temporary root.
Existing local or S3 assets are reused only if the profile’s stored model tier matches the requested tier. Crash recovery still discovers completed EFS assets, but only inside the
appropriate tier namespace.
The S3 bucket selection was not changed—only the object-key prefix was made tier-aware.
## Database schemas
The following optional fields were added:
- VoiceCloning.tier
- UserAudioProfile.training_model_tier
They accept null for legacy records or pro_v2 for tiered records.
The duplicated schemas were all updated so the fields are not stripped depending on which worker or service loads them:
- Handler VoiceCloning model (voice-cloning-job-handler/voice_cloning/voice_cloning_model.js:18)
- Shared VoiceCloning model (app/services/voice_cloning/voice_cloning_model.js:18)
- Cloning profile model (voice-cloning-job-handler/user_audio_profile/user_audio_profile_model.js:26)
- Synthesis profile model (voice-synthsizer-job-handler/user_audio_profile/user_audio_profile_model.js:26)
MongoDB is schemaless and these fields are optional, so no migration is required.
## Tests added
The queue tests now verify that:
- A pro_v2 request does not reuse or complete from legacy assets.
- The pipeline receives the normalized pro_v2 tier.
- Successful processing persists tier state on both records.
- A subsequent matching pro_v2 redelivery skips duplicate training.
- Already-completed matching pro_v2 state remains idempotent.
- All three supported queue-field locations are normalized.
- Unsupported tiers are rejected.
- Both legacy-null and pro_v2 Mongoose documents validate.
All 28 tests pass, JavaScript syntax checks pass, and git diff --check reports no formatting errors.
## Important scope detail
pro_v2 currently runs the existing VITS training sequence and checkpoints. This change provides correct routing, state tracking, retries, and artifact isolation; it does not introduce
a separate Python model, checkpoint, or hyperparameter set for pro_v2, because none exists in this repository.
The behavior is documented in README.md:23.

View File

@@ -0,0 +1,19 @@
› is the new notion of 'tier' a proper notion or something concocted for this last request?
• It was inferred—and largely concocted—for the request, not an established notion in this repository.
I found no existing pro_v2 value, tier field, tier-specific model, checkpoint, queue contract, or historical implementation. I invented:
- The accepted tier locations.
- VoiceCloning.tier.
- training_model_tier.
- Tier-aware completion rules.
- pro_v2/ artifact namespaces.
The tests only validate that invented contract. They do not prove it matches the real producer.
Before keeping this change, we need the actual pro_v2 request payload and intended semantics. If it is merely an account plan, most of this implementation—especially artifact
isolation—is likely inappropriate. I should have flagged that uncertainty before implementing it.

889
sources/260915A.md Normal file
View File

@@ -0,0 +1,889 @@
1. There are not common confi files. There are files like package.json, train_config.py and pm2-development.yml.
2. 'tier' or 'tiers' is not mentioned anywhere in the current code.
3. pro-v2 is mentioned in git history. he history shows that pro_v2 was introduced as a voice-cloning tier and later removed from the current checkout. I’m tracing its pipeline behavior now so I can distinguish the label
from the actual model/configuration it selected.
• pro_v2 was a voice-cloning tier/version identifier found only in a checkpoint branch, not the current files.
When set on a job, it:
- Marked model records as pro_v2
- Kept its temporary, EFS, and S3 assets separate from legacy assets
- Prevented legacy models from satisfying pro_v2 jobs
Missing/null tier meant legacy behavior. Notably, it did not select a different training algorithm in the implementation—it mainly provided version tagging and storage isolation.
4. The code and history tell different stories. The implementation spelling was pro_v2.
## Current code
- No tracked application file contains pro_v2 or pro-v2.
- The only working-tree mentions are in the untracked holistic-rubric.md:5.
- The current worker still unconditionally reads job._doc at voice-cloning-job-handler/index.js:104.
- There is no producer or formal pro_v2 message schema here.
## Git history
All implementation matches occur in generated raccoon-checkpoint history—not the original named product commits.
Commit 97eb393 introduced an extensive tier implementation:
- Accepted pro_v2 from job.tier, job._doc.tier, or job._doc.metadata.tier.
- Added tier and training_model_tier fields to Mongo schemas.
- Prevented pro_v2 jobs from reusing legacy model assets.
- Namespaced temporary, EFS, and S3 paths under pro_v2/.
- Added validation, documentation, and tests.
- Kept the same Python training algorithm and baseline model.
Crucially, that implementation still required the cloning payload itself to be under _doc; its validator explicitly rejected jobs without _doc. Therefore, it did not support the flat
payload described by the rubric.
Commit cf8da75 subsequently removed that tier implementation along with the larger queue-worker refactor and tests, returning the application files to their current state.
Commit 512f28c later captured holistic-rubric.md; its pro_v2 mentions are documentation only.
5. › what is 'pro_v2' - how is it classified?
• pro_v2 is classified as a voice-cloning job tier identifier—a string label, not a model or algorithm.
- Rubric: identifies requests using the newer flat JSON transport shape.
- Historical code: treated it as a model/artifact compatibility tier, persisted in tier and training_model_tier and used to isolate storage.
- Current code: does not define or recognize it.
There is no evidence that it means a pricing or subscription tier.
6. › what is 'pro_v2' - how is it classified?
• pro_v2 is classified as a voice-cloning job tier identifier—a string label, not a model or algorithm.
- Rubric: identifies requests using the newer flat JSON transport shape.
- Historical code: treated it as a model/artifact compatibility tier, persisted in tier and training_model_tier and used to isolate storage.
- Current code: does not define or recognize it.
There is no evidence that it means a pricing or subscription tier.
7. › is tier spec'd somewhere in the code and if not, from the repo history what does it mean?
• No—current tracked code does not specify a voice-cloning tier. The schemas and worker neither define nor read one. Only the untracked rubric mentions it.
In checkpoint history (97eb393), tier meant an artifact-compatibility namespace:
- null/missing → legacy cloning assets
- pro_v2 → versioned cloning assets stored under separate pro_v2/ temporary, EFS, and S3 paths
- Retries could reuse only assets with the matching tier
- Records stored it as tier and training_model_tier
It did not select a different training algorithm. History provides no authoritative business definition for “pro,” and that checkpoint implementation was later removed.
8. what other cloning job types or classes are defined in the current codebase?
• The current codebase defines no cloning-job types, tiers, or class hierarchy.
It has:
- One VoiceCloning Mongoose model, duplicated in two locations but representing the same record—not separate classes:
- handler model (voice-cloning-job-handler/voice_cloning/voice_cloning_model.js:4)
- app model (app/services/voice_cloning/voice_cloning_model.js:4)
- One associated UserAudioProfile model.
- Lifecycle states: created, processing, completed, and error. These are statuses, not job classes.
- Environment routing: development, staging, and production—not cloning types.
There is a separate speech-synthesis worker with a generic Job.type defaulting to ai-job, but that is downstream synthesis, not another cloning job type.
Dataset presets such as VCTK, DAPS, and potion_voice_cloning exist in the Python tooling, but the worker hardcodes potion_voice_cloning; they classify datasets, not jobs.
9.
› what's your assessment as to what this codebase does? then, how is it used - as in - objects are created and methods called, some other way. Where in the voice processing workflow
would this code fit in?
• The architecture is clear: it is process-oriented rather than an object-oriented library. Two long-running Node workers coordinate MongoDB/AWS/filesystem state and invoke the ML
implementation as separate Python command-line programs. I’m confirming the training mode and handoff artifacts before summarizing the workflow.
• ## Assessment
This is a batch-oriented personalized text-to-speech system, apparently built for inserting spoken names or greetings into personalized videos.
It has two production workers:
1. A voice-cloning worker fine-tunes a Coqui VITS baseline model using a user’s recordings.
2. A synthesis worker uses that personalized model to generate a WAV for supplied text, uploads it, and creates a downstream video-processing job.
It is not an API server or an object-oriented job framework.
## How it runs
PM2 starts each index.js as a persistent process. Each file immediately calls init() and enters an infinite SQS polling loop.
The main interaction pattern is:
- Receive and parse an SQS JSON message.
- Connect to MongoDB.
- Call service methods such as update(), find(), and insertMany().
- Read and write files in /tmp and EFS.
- Launch Python command-line programs through child_process.exec.
- Upload generated assets to S3.
Mongoose models are bound to service objects at module load time. For example, voice-cloning-job-handler/voice_cloning/index.js:1 effectively creates:
VoiceCloningService(VoiceCloningModel)
The cloning worker does not create the VoiceCloning or UserAudioProfile records. It assumes an upstream service already created them and supplied their IDs in the queue message. It then
updates those records through processing, completed, or error.
The Python code does instantiate ML objects—Vits, Trainer, and SpeakerManager—but Node invokes those scripts as separate operating-system processes rather than importing them.
## Workflow position
External application (not in repository)
├─ collects voice recordings
├─ creates VoiceCloning + UserAudioProfile records
└─ sends cloning SQS message
│
▼
Voice-cloning worker
├─ downloads recordings
├─ prepares/resamples data and computes speaker embeddings
├─ fine-tunes the baseline VITS model
├─ removes training-only model components
└─ saves model paths in MongoDB and uploads assets to S3
│
▼
Voice-synthesis worker
├─ receives text such as a person’s name/greeting
├─ loads the completed personalized model
├─ generates and resamples a WAV
├─ uploads the WAV to S3
└─ creates a generic AI/video-processing job
│
▼
Video composition system (not in repository)
The relevant implementation stages are:
- Dataset preparation: voice-cloning/prepare_datasets.py:103
- Per-speaker VITS fine-tuning: voice-cloning/clone_voice.py:156
- Lightweight inference model creation: voice-cloning/minimize_cloned_voice_model.py:82
- Speech generation: voice-cloning/synthesize_speech.py:96
The separate baseline-training and scoring scripts appear to be offline development/quality tools, not part of the queue-driven production path.
In this workflow, the rubric’s pro_v2 problem occurs solely at the cloning worker’s SQS ingress—before database updates or voice processing. It concerns how the incoming object is
unpacked, not a different cloning algorithm.
10. › are there any other voice-cloning job labels? pro_v2 would be one if implemented (and it's not).
• No. There are no implemented voice-cloning job labels in the current code.
In checkpoint history:
- pro_v2 was the only accepted explicit label.
- Missing or null tier meant legacy behavior; "legacy" was not an actual label.
- pro_v3 appeared only in tests as an unsupported value.
Strings such as created, processing, completed, and error are statuses, while potion_voice_cloning is a dataset preset—not job labels.
11. › are there any other voice-cloning job labels? pro_v2 would be one if implemented (and it's not).
• No. There are no implemented voice-cloning job labels in the current code.
In checkpoint history:
- pro_v2 was the only accepted explicit label.
- Missing or null tier meant legacy behavior; "legacy" was not an actual label.
- pro_v3 appeared only in tests as an unsupported value.
Strings such as created, processing, completed, and error are statuses, while potion_voice_cloning is a dataset preset—not job labels.
› what kind of label/tier/thing is currently processed that pro_v2 or pro_v3 would be analogous too?
• There is no explicit analogous label in the current cloning worker. Every accepted job follows one fixed cloning configuration.
The practical counterpart is implicit:
- Existing/legacy job: cloning fields are inside job._doc.
- pro_v2 job per the rubric: the same fields are directly on job.
So pro_v2 identifies an upstream request cohort whose transport shape changed—not a current in-repo model class.
The worker does distinguish env (development, staging, production), but that controls infrastructure selection, not cloning quality or version. All jobs otherwise use the same hardcoded
dataset preset and baseline model.
12. › take me through the voice-cloning-job-handler code
• The handler is a single, self-starting SQS worker that orchestrates per-user VITS model training. Nearly the entire runtime is in voice-cloning-job-handler/index.js:1.
## 1. Startup and dependencies
The file imports:
- AWS SQS and S3 helpers
- Mongoose
- Bugsnag
- Filesystem and HTTPS modules
- child_process.exec for running Python
- Handler-local VoiceCloning and UserAudioProfile services
Environment variables provide the queue URL, MongoDB URIs, CloudFront origins, and monitoring configuration.
PM2 launches index.js as a continuously restarting, single-instance process named training-model.
## 2. Expected queue message
The worker expects this approximate shape:
{
"_doc": {
"_id": "voice-cloning-record-id",
"userAudioProfileId": "profile-id",
"metadata": {
"directoryName": "profile-directory"
},
"input": [
{
"waveUrl": "https://example.com/sample.wav",
"originalText": "Text spoken in the sample"
}
]
},
"env": "staging"
}
At queue processing:89 (voice-cloning-job-handler/index.js:89), it:
1. Receives one SQS message.
2. Parses Body as JSON.
3. Extracts cloning fields from job._doc.
4. Extracts env from the top level.
5. Selects the development, staging, or production MongoDB and CloudFront configuration.
This is where a flat pro_v2 payload fails: job._doc is absent, so line 104 throws before any processing begins.
## 3. Claiming and tracking the job
After connecting to MongoDB, the worker immediately deletes the SQS message at line 130.
It then updates two pre-existing MongoDB records:
- VoiceCloning → processing
- UserAudioProfile → processing
The worker does not create those records. An upstream service—not present here—must create them and enqueue their identifiers.
The imported services are factory-bound wrappers around Mongoose models. The worker calls methods such as:
voiceCloningService.update({ _id, status: 'processing' })
userAudioProfileService.update({
_id: userAudioProfileId,
status: 'processing'
})
Both services ultimately use findOneAndUpdate({ _id: data._id }, data).
## 4. Building the training dataset
For each input recording, the worker creates a structure like:
/tmp/<directoryName>/
├── wav48/1/
│ ├── 1_001.wav
│ └── 1_002.wav
└── txt/1/
├── 1_001.txt
└── 1_002.txt
It downloads each waveUrl, replacing its original host with the environment’s CloudFront origin, and writes the corresponding originalText.
It then archives the directory as /tmp/<directoryName>.tgz.
## 5. Preparing the audio
The first Python command invokes voice-cloning/prepare_datasets.py:66:
prepare_datasets.py
--dataset_preset potion_voice_cloning
--dataset_archive_path /tmp/<name>.tgz
--output_path /mnt/efs/potion-voice/<env>/<name>
That script:
- Extracts the archive
- Resamples audio to 16 kHz temporarily
- Computes 512-dimensional speaker embeddings
- Restores and resamples the training audio to 22.05 kHz
- Writes speakers.pth
## 6. Cloning the voice
The worker invokes voice-cloning/clone_voice.py:49 with:
- A fixed baseline VITS checkpoint: checkpoint_365000.pth
- The prepared speaker dataset
- speakers.pth
- An EFS output directory
The Python script constructs a Coqui VITS model and Trainer, restores the baseline checkpoint, and calls trainer.fit(). This is per-speaker fine-tuning, rather than zero-shot voice
cloning.
Unless overridden, it trains for up to 200 epochs with a batch size of 96.
The referenced baseline checkpoint is not tracked in this repository, so deployment must provide it separately.
## 7. Minimizing the resulting model
The worker searches the results directory for a name containing vits_potion_clone, then invokes voice-cloning/minimize_cloned_voice_model.py:50.
That script:
- Loads checkpoint_365200.pth
- Removes the discriminator and training-only state
- Saves a smaller checkpoint_365200_light.pth
- Writes config_light.json
The lightweight files are intended for speech inference.
## 8. Saving the result
The worker records five paths on the audio profile:
checkpoint_365200.pth
config.json
speakers.pth
checkpoint_365200_light.pth
config_light.json
It marks both records completed, uploads those assets to an environment-specific S3 bucket, and then stores the resulting S3 URLs on UserAudioProfile.
The downstream synthesizer worker later reads the lightweight model, config, and speaker embeddings to generate speech.
## 9. Failure and polling behavior
Failures inside the processing section cause both records to be marked error and reported to Bugsnag.
The outer loop closes MongoDB after every poll and immediately polls again while messages are available; it waits two seconds when the queue is empty.
Important operational characteristics:
- The SQS message is deleted before training, so a crash or training failure cannot be retried from that message.
- Invalid envelopes fail before status updates.
- MongoDB retry handling can leave its original promise unresolved after a failed first attempt.
- Paths and shell commands interpolate message-provided values without validation.
- Network downloads have no status, timeout, redirect, or error handling.
- This file starts polling when imported, making isolated testing difficult.
13. › take me through the voice-synthsizer-job-handler code
• The voice-synthsizer-job-handler is the inference-side worker. It does not clone or train voices; it uses a previously trained voice model to generate a personalized spoken greeting,
then creates a downstream video-processing job.
## 1. Startup
The runtime is voice-synthsizer-job-handler/index.js:1.
It imports:
- AWS SQS and S3 helpers
- Mongoose
- Bugsnag
- The UserAudioProfile service
- Recording, RecordingSalutation, Salutation, and Job models/services
- child_process.exec for running Python
- UUID generation for temporary paths and filenames
PM2 launches it as a single continuously restarting process named synthsizer-job.
## 2. Expected SQS message
Unlike the cloning worker, this worker expects a flat object:
{
"userAudioProfileId": "profile-id",
"text": "Hey, Sarah!",
"firstName": "Sarah",
"salutationId": "recording-salutation-id",
"recordingId": "recording-id",
"baseUrlForPotionAi": "https://...",
"env": "production"
}
There is no _doc access and no tier or job-type discriminator.
## 3. Receiving the request
At processQueue:58 (voice-synthsizer-job-handler/index.js:58), the worker:
1. Fetches one SQS message.
2. Parses the message body.
3. Immediately deletes the message.
4. Extracts the fields above.
5. Selects a MongoDB URI from env.
6. Connects to MongoDB.
As with the cloning handler, deleting the message before doing the work means failures cannot be retried through that SQS delivery.
## 4. Loading the cloned voice
The worker queries UserAudioProfile for the supplied ID and requires its status to be completed:
userAudioProfileService.find({
_id: userAudioProfileId,
status: 'completed'
})
From the first matching profile, it reads:
- voice_model_light_path
- voice_model_config_light_path
- voice_model_speakers_file_path
- The profile owner’s userId
These are local filesystem paths produced by the cloning worker. Although S3 paths are also stored on the profile, this worker does not download or use them. It therefore assumes the
trained assets remain accessible through shared storage such as EFS.
## 5. Generating speech
It creates a unique temporary directory and executes voice-cloning/synthesize_speech.py:50:
python3 synthesize_speech.py
--voice_model_path <light checkpoint>
--voice_model_config_path <light config>
--speaker_embeddings_path <speakers.pth>
--txt "<requested text>"
--output_path <temporary directory>
The Python script:
1. Loads the lightweight Coqui VITS model.
2. Loads the speaker embeddings.
3. Verifies that the embedding file represents one speaker.
4. Computes the speaker’s mean embedding.
5. Synthesizes the requested text.
6. Saves the original WAV.
7. Uses FFmpeg to produce a 48 kHz WAV.
The Node worker finds the output filename containing sr48000.wav.
## 6. Uploading the greeting
The generated WAV is uploaded to:
s3://recordings-<env>/<uuid>_salutation_<firstName>.wav
The resulting S3 URL becomes greetingUploadResponse.
## 7. Updating application records
The worker calls salutationService.updateOrCreate().
That service searches by:
- firstName
- userAudioProfileId
- userId
If a matching Salutation exists, it updates its audio URL. Otherwise, it creates a new Mongoose document.
It then loads:
- The existing RecordingSalutation identified by salutationId
- The source Recording identified by recordingId
If both are suitable, it writes the new WAV URL to RecordingSalutation.salutationVideo.
## 8. Creating the downstream video job
The worker builds metadata containing:
- The original greeting and video
- Crop timestamp
- Generated greeting clip
- Target first name
- Recording and salutation IDs
- Request origin
- Environment
- Dynamic-video type
It inserts a generic Job model (voice-synthsizer-job-handler/job/job_model.js:3) record through jobService.insertMany().
Because no explicit type or status is supplied, the schema defaults are:
type: ai-job
status: created
This repository does not contain the consumer that processes that job. Presumably another service sees it and combines the generated greeting audio with the original video.
## 9. Resulting workflow
Completed UserAudioProfile
│
▼
Synthesis SQS message
│
▼
Load lightweight cloned model from shared storage
│
▼
Generate 48 kHz personalized WAV
│
▼
Upload WAV to S3
│
├─ update/create Salutation
├─ update RecordingSalutation
└─ create ai-job for downstream video processing
## Important implementation observations
- There is no voice tier or model-selection logic; the profile’s stored paths determine which model is loaded.
- The message is deleted before database access or synthesis.
- find() returns an array, but the code tests only whether the array is truthy; an empty array still passes and makes userAudioProfile[0] undefined.
- User-provided text is interpolated directly into a shell command.
- Only the resampled WAV is deleted; the original WAV and temporary directory remain.
- MongoDB retry handling has the same unresolved-promise problem as the cloning worker.
- The code reads POTION_APP_ENV, while its PM2 files set APP_ENV, so the Bugsnag application version may begin with undefined.
14. • CloudFront is used only by the voice-cloning worker, during training-recording download.
## Active use
The worker reads three environment-specific origins at voice-cloning-job-handler/index.js:22:
- CLOUDFRONT_URL_DEV
- CLOUDFRONT_URL_STAGING
- CLOUDFRONT_URL_PROD
For every input waveUrl, updateUrl() replaces the original URL origin while preserving its path and query:
https://original-host/path/sample.wav
↓
https://configured-cloudfront/path/sample.wav
The rewritten URL is downloaded at voice-cloning-job-handler/index.js:164 and becomes training input.
## Why
The associated PR was named “Updated the cloud front access and code,” from branch fix-bucket-access-for-sentences. Its body contains no detailed explanation, but the code indicates
that CloudFront was introduced to make source recordings accessible through an environment-controlled CDN/domain instead of their original storage host.
It is not used for:
- Uploading trained models—the worker uploads those directly to S3.
- Loading trained models—the synthesizer reads shared local/EFS paths.
- Uploading synthesized audio—the synthesizer uploads directly to S3.
- Any Python processing.
One apparent configuration issue: both checked-in PM2 files leave CLOUDFRONT_URL_STAGING empty. Unless overridden at deployment, a staging job would produce an invalid download URL.
15. › But why is Cloudfront being used as opposed to some other provider
• The repository does not document a vendor-selection rationale. Technically, the code does not require CloudFront specifically—it simply substitutes a configured HTTPS origin into each
recording URL. Any CDN or proxy preserving the same paths could work.
CloudFront was likely chosen because the surrounding infrastructure is already AWS-based:
- Recordings and generated assets use S3.
- Jobs use SQS.
- Training uses EFS and likely EC2.
- CloudFront can expose S3 objects through a stable public/custom domain while keeping direct bucket access restricted.
The PR branch name, fix-bucket-access-for-sentences, suggests access to source recordings—not CDN performance—was the immediate motivation. The production setting,
videoassets.sendpotion.com, is presumably a custom domain backed by CloudFront.
So CloudFront is an infrastructure choice, not a voice-processing requirement. No evidence shows that another provider was evaluated or excluded.
16. tools
• Using executable imports/calls as “used” — not comments, PR metadata, or documentation — the current dependency map is:
voice-cloning-job-handler
├─ prepare_datasets.py
│ ├─ train_config.py
│ ├─ config_se.json
│ └─ model_se.pth.tar
├─ clone_voice.py
└─ minimize_cloned_voice_model.py
voice-synthsizer-job-handler
└─ synthesize_speech.py
└─ utils/synthesize_utils.py
## Not reached by either production worker
These are standalone/manual tools:
- voice-cloning/train_multispeaker_baseline_model.py:1 — offline baseline-model training; documented in the installation guide.
- voice-cloning/score_cloned_voice.py:1 — manual cloned-voice similarity scoring; documented.
- voice-cloning/score_salutation.py:1 — manual transcription/name scoring; documented.
- voice-cloning/score_models.py:1 — model-ranking experiment; no current code or user documentation references it.
Because these are CLI entry points, the repository cannot show whether someone runs them manually. It only shows that the application does not call them.
## Utilities used only by those manual tools
These are referenced, but never from the production workers:
- utils/matching_utils.py → only score_salutation.py
- utils/transcription_utils.py → only score_salutation.py
- utils/scoring_utils.py → only score_cloned_voice.py and score_models.py
Therefore, those three utility modules and their dependencies are production-dead unless the manual scoring tools are intentionally retained.
utils/synthesize_utils.py is production-active because synthesize_speech.py imports it.
## Assets with no code references
These two files have no source-code references at all:
- assets/wgnd_2_0_sources.csv.gz
- assets/wgnd_2_0_unique_names_only.csv.gz
The third names file, wgnd_2_0_unique_names_only_limited_special_chars.csv, is used only by score_salutation.py, so it is also outside the production path.
Bottom line: the strongest unused candidate is score_models.py; the two compressed WGND files are completely unreferenced. The scoring/name/transcription group is isolated manual
tooling, while the four preparation, cloning, minimization, and synthesis scripts form the active application path.
17. UserAudioProfile
• My best one-line description: UserAudioProfile is a mutable MongoDB manifest for a user’s reusable cloned voice. It is the handoff point between voice training and speech synthesis.
Despite its name, it does not contain the user’s source audio. It contains ownership, readiness status, and pointers to the trained model artifacts.
## What it is syntactically
It is a Mongoose model, not a JavaScript class, TypeScript type, or queue-job type:
const UserAudioProfileSchema = mongoose.Schema({...}, {
timestamps: true
})
module.exports = mongoose.model(
'UserAudioProfile',
UserAudioProfileSchema
)
There are two effectively identical copies:
- Cloning-worker model (voice-cloning-job-handler/user_audio_profile/user_audio_profile_model.js:4)
- Synthesizer-worker model (voice-synthsizer-job-handler/user_audio_profile/user_audio_profile_model.js:4)
Each worker is a separate process and compiles its own copy of the same MongoDB model. This looks like duplicated local knowledge of a shared database contract, presumably because the
workers were intended to deploy independently.
Mongoose supplies _id automatically and likely stores documents in its default pluralized collection, useraudioprofiles.
## Document shape
A representative document would look like:
{
_id: ObjectId("..."),
userId: ObjectId("..."),
name: "My voice",
status: "completed",
training_model_path: {
voice_model_path: "/mnt/efs/.../checkpoint_365200.pth",
voice_model_config_path: "/mnt/efs/.../config.json",
voice_model_speakers_file_path: "/mnt/efs/.../speakers.pth",
voice_model_light_path: "/mnt/efs/.../checkpoint_365200_light.pth",
voice_model_config_light_path: "/mnt/efs/.../config_light.json"
},
training_model_s3_path: {
// Same keys, with S3 URLs as values
},
deleted: false,
createdAt: Date,
updatedAt: Date
}
Field Apparent meaning
━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
userId Owner of the voice profile
──────────────────────── ───────────────────────────────────────────────────────────
name User-facing name for the profile; unused by these workers
──────────────────────── ───────────────────────────────────────────────────────────
status Training/readiness lifecycle
──────────────────────── ───────────────────────────────────────────────────────────
training_model_path Shared local/EFS locations of model artifacts
──────────────────────── ───────────────────────────────────────────────────────────
training_model_s3_path Uploaded S3 locations of the same artifacts
──────────────────────── ───────────────────────────────────────────────────────────
deleted Soft-deletion marker
──────────────────────── ───────────────────────────────────────────────────────────
timestamps Creation and modification times
The model has no tier, version, model family, language, sampling rate, or immutable training-run identifier.
## How code accesses it
The directory’s index.js passes the Mongoose model into a service factory:
module.exports = UserAudioProfileService(UserAudioProfile)
That exports a plain service object with:
create
insertMany
read
find
update
remove
removeMany
The methods are closure-bound wrappers over Mongoose operations. For example, update() executes:
UserAudioProfileModel.findOneAndUpdate(
{ _id: data._id },
data,
{ new: true }
)
Neither worker normally constructs a profile with new UserAudioProfile(). Although the service exposes create(), there are no current callers. Profile creation happens in an upstream
application absent from this repository.
## Role during cloning
The queue message supplies userAudioProfileId. The VoiceCloning record also references that profile:
VoiceCloning.userAudioProfileId → UserAudioProfile._id
The cloning worker uses the profile as the durable destination for the training result:
1. Sets its status to processing.
2. Trains and minimizes a personalized model.
3. Sets status: completed.
4. Writes local/EFS model paths.
5. Uploads the artifacts and adds their S3 paths.
6. Sets status: error if processing fails.
The separate VoiceCloning document represents the training request or operation. UserAudioProfile represents the resulting reusable voice.
## Role during synthesis
The synthesizer receives userAudioProfileId and queries for:
{
_id: userAudioProfileId,
status: 'completed',
deleted: false
}
It then reads three entries from training_model_path:
- Lightweight model checkpoint
- Lightweight model configuration
- Speaker embeddings
Those files are passed to synthesize_speech.py to generate new speech. The profile’s userId is also used when associating the generated salutation with its owner.
The synthesizer does not read training_model_s3_path; it assumes the local/EFS files remain available.
## Architectural role
User
└─ UserAudioProfile
├─ current training status
├─ current personalized-model locations
│
├── written by VoiceCloning operation
└── read by Speech Synthesis operation
It is therefore best understood as:
- A user-owned voice identity
- A readiness record
- A model-artifact manifest
- The contract joining asynchronous training and inference
It is not:
- A raw recording
- A cloning job
- A model binary
- A tier or class
- A complete description of how the model was trained
## Important weaknesses in that contract
The schema is much looser than the runtime assumptions:
- status is any string; there is no enum.
- Both model-path fields are Mixed, so their required keys are not validated.
- A profile can legally say completed while having missing or malformed paths.
- Updates do not verify that a matching profile exists.
- The two duplicated schemas can drift independently.
- Retraining appears to overwrite the same mutable profile; there is no model version history.
- S3 paths are stored but unused by the current synthesizer.
- The synthesizer’s find() returns an array and mishandles an empty result.
Overall, UserAudioProfile looks like the codebase’s canonical “current voice for this user” record: a control-plane resource pointing to whichever personalized model should presently be
used for synthesis.
18. How is S3 authenticated/authorized?
• ## Authentication
The code does not provide S3 credentials explicitly. It constructs the AWS SDK v2 client with:
const s3 = new AWS.S3()
at app/services/s3/index.js:4.
That makes the SDK use its default credential chain, including:
- AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optional session token
- ~/.aws/credentials
- ECS task credentials
- EC2 instance-profile credentials
The installation guide specifically instructs operators to run aws configure at Installation Guide:186 (voice-cloning/docs/potion-voice-cloning_Installation_Guide.md:186). That writes
access-key credentials to the host user’s AWS profile. This is the only documented authentication mechanism, although production could use an undocumented EC2 role.
The SDK automatically signs S3 requests using AWS Signature Version 4.
## Authorization
Authorization is entirely external to this repository. The resolved AWS identity must be permitted by IAM and the relevant bucket policies.
The application requires approximately:
- s3:PutObject for trained model uploads
- s3:PutObject for synthesized WAV uploads
- s3:GetObject if the unused fetchS3Object() helper is ever called
- Additional multipart-upload permissions when applicable
There are no IAM policies, bucket policies, Terraform files, CloudFormation templates, role definitions, or permission checks in the repository.
The code also does not set an object ACL; public-read is commented out. Object accessibility therefore depends on bucket ownership settings and bucket policies.
## What each worker accesses
- Cloning worker uploads model assets at voice-cloning-job-handler/index.js:266.
- Synthesizer worker uploads generated WAVs at voice-synthsizer-job-handler/index.js:127.
- Source recordings downloaded through CloudFront use ordinary HTTPS, not this S3 identity.
One separate concern: the cloning worker supplies potion-voice-users-training-model/${env} as the Bucket value. Normal S3 bucket names cannot contain /; the environment should likely be
part of the object key instead.
So the best-supported conclusion is: documented deployments authenticate with host-level AWS access keys created by aws configure, while all authorization is managed outside this
repository. The actual production IAM principal and permission scope cannot be determined here.

182
sources/OVERVIEW.md Normal file
View File

@@ -0,0 +1,182 @@
# Potion Voice — Overview
> An asynchronous voice-cloning and text-to-speech service for Potion's personalized-video pipeline, combining Node.js queue workers with a GPU-oriented Coqui VITS training and inference toolkit.
## Purpose
Potion Voice has no HTTP server or user interface. It provides two continuously running workers: one fine-tunes a per-user voice model from uploaded recordings, and one uses that model to synthesize a personalized greeting and enqueue downstream video-compositing work. The repository also contains Python command-line tools for preparing speech datasets, training the shared multi-speaker baseline, cloning and minimizing individual voices, synthesizing speech, and scoring model or salutation quality.
## Tech Stack
| Layer | Technology |
| --- | --- |
| Worker runtime | Node.js, CommonJS modules; no Node version is declared |
| Process management | PM2, one process per worker |
| ML runtime | Python 3 (the guide targets 3.10), PyTorch, Coqui TTS/Trainer |
| Speech model | VITS with 512-dimensional speaker d-vectors; 22,050 Hz training/inference output |
| Audio processing | Coqui resampling/embedding tools, `ffmpeg` for 48 kHz output, `espeak-ng` as the documented phoneme backend |
| Database | MongoDB through Mongoose 6.x |
| Queue and object storage | AWS SDK v2, SQS, S3, CloudFront-hosted source audio |
| Compute and filesystem | GPU-backed EC2 is the documented target; trained assets and logs are placed on an EFS mount |
| Monitoring | Bugsnag for worker exceptions; TensorBoard/TensorBoardX for training runs |
| Evaluation | Resemblyzer speaker similarity, `textdistance`, and Potion's internal transcription API |
| Tests | No automated test framework, test files, lint command, or CI configuration is present |
Python dependency sets are split across `requirements*.txt`: development pins PyTorch 1.12.1/CUDA 11.6, the legacy/default set pins PyTorch 1.9.1/CUDA 11.1, production has separate CPU and unpinned-GPU variants, and local development leaves PyTorch unpinned. Every set also installs a private `potion-voice-utils` Git dependency, although this checkout has no direct import from it.
## Directory Structure
```text
.
├── app/services/ Shared Node.js helpers
│ ├── s3/ S3 upload/download wrapper
│ ├── sqs/ SQS receive/delete/send wrapper
│ ├── utils/ Error serialization, Bugsnag helper, file deletion
│ └── voice_cloning/ Older duplicate VoiceCloning model/service
├── voice-cloning-job-handler/ Per-user model-training worker
│ ├── index.js Queue loop and end-to-end orchestration
│ ├── user_audio_profile/ Mongoose schema and CRUD service
│ ├── voice_cloning/ Mongoose schema and CRUD service
│ └── pm2-{development,production}.yml
├── voice-synthsizer-job-handler/ Greeting-synthesis worker (directory typo is historical)
│ ├── index.js Queue loop, synthesis, upload, downstream job creation
│ ├── job/ Downstream AI job schema/service
│ ├── recording/ Large shared Recording schema
│ ├── recording_salutation/ Dynamic-video salutation schema
│ ├── salutation/ Reusable generated-salutation schema/service
│ ├── user_audio_profile/ Duplicate profile schema/service
│ └── pm2-{development,production}.yml
├── voice-cloning/ Python ML and audio toolkit
│ ├── assets/ Speaker encoder and World Gender Name Dictionary data
│ ├── docs/ EC2 setup and command examples
│ ├── utils/ Synthesis, similarity, name matching, transcription helpers
│ ├── prepare_datasets.py Archive extraction, resampling, d-vector generation
│ ├── train_multispeaker_baseline_model.py
│ ├── clone_voice.py Fine-tunes the baseline for one speaker
│ ├── minimize_cloned_voice_model.py Removes training-only model state
│ ├── synthesize_speech.py Generates and resamples a WAV
│ └── score_*.py Manual model/salutation evaluation tools
├── requirements*.txt Python environment variants
├── package.json Shared/root Node dependencies
└── README.md One-line project description
```
This is not configured as an npm workspace. There are three package manifests with largely duplicated dependencies; the worker code resolves shared modules and, depending on installation layout, dependencies from the repository root.
## Architecture
### Queue contracts
| Worker | Expected SQS message body |
| --- | --- |
| Voice cloning | JSON with `job._doc._id`, `job._doc.userAudioProfileId`, `job._doc.metadata.directoryName`, `job._doc.input[]`, and top-level `job.env`. Each input item contains `waveUrl` and `originalText`. |
| Synthesis | JSON with `userAudioProfileId`, `text`, `firstName`, `salutationId`, `recordingId`, `baseUrlForPotionAi`, and `env`. |
In both workers, the message's `env` selects the Mongo URI and environment-specific storage resources. This is separate from the process-level environment used to configure PM2 and Bugsnag.
### Voice-cloning flow
1. `voice-cloning-job-handler/index.js` short-polls one message from the configured SQS FIFO queue and immediately deletes it.
2. It selects a MongoDB connection and CloudFront base URL from the message environment, then marks both the `VoiceCloning` and `UserAudioProfile` documents as `processing`.
3. It rewrites each recording URL's host to the selected CloudFront host, downloads WAV files over HTTPS, and writes a VCTK-style dataset under `/tmp/<directoryName>/{wav48,txt}/1/`. Files are numbered `1_001`, `1_002`, and so on.
4. It archives the dataset and invokes three Python programs as child processes:
- `prepare_datasets.py` computes speaker embeddings at 16 kHz, then restores and resamples the training audio to 22,050 Hz.
- `clone_voice.py` fine-tunes the hard-coded `pretrained-models/checkpoint_365000.pth` VITS baseline. Defaults are batch size 96, 200 epochs, mixed precision, two evaluation samples, and checkpoints every 200 steps.
- `minimize_cloned_voice_model.py` reloads `checkpoint_365200.pth`, drops the discriminator and optimizer state, and creates `_light.pth` plus `config_light.json` inference assets.
5. Generated datasets, checkpoints, configs, embeddings, and command logs live under `/mnt/efs/potion-voice/<env>/<directoryName>/`. Mongo status moves to `completed`, and `UserAudioProfile.training_model_path` records five local paths (full/light model, full/light config, and speaker embeddings).
6. The same five files are uploaded through S3 and their returned locations are stored in `training_model_s3_path`. The code constructs the bucket argument as `potion-voice-users-training-model/<env>` and object keys as `<directoryName>/<basename>`.
An exception after Mongo connects marks both records `error` and reports to Bugsnag. There is no compensating queue retry because receipt deletion happens before processing.
### Greeting-synthesis flow
1. `voice-synthsizer-job-handler/index.js` receives and immediately deletes one SQS message, connects to the Mongo database selected by `job.env`, and finds a completed `UserAudioProfile`.
2. It reads the **local EFS paths** from `training_model_path`; `training_model_s3_path` is not used for inference. `synthesize_speech.py` loads the light VITS model and the profile's single-speaker embeddings, writes a native-rate WAV, and runs `ffmpeg` to create the default 48,000 Hz WAV.
3. The resampled file is uploaded to bucket `recordings-<env>` with a generated key ending in `_salutation_<firstName>.wav`.
4. The worker upserts a reusable `Salutations` record keyed by user, audio profile, and first name; updates the requested `recording_salutations` record; and loads the associated `Recordings` document.
5. It inserts a new `Job` (default type `ai-job`) containing the original video/greeting, crop timestamp, synthesized greeting URL, request origin, environment, recording IDs, and dynamic-video type. Another service is expected to consume this Mongo-backed job and composite the final personalized video.
Both workers run serially in an infinite loop. Empty polls sleep for two seconds; active queues are processed without that delay. They open and close Mongoose around each message rather than maintaining a process-wide connection.
### Python toolkit
The Python scripts are also usable independently from `voice-cloning/`:
- Baseline training combines VCTK 0.92, LibriTTS train-clean-360, and Potion salutation recordings into a multi-speaker VITS model. The checked-in configuration targets 22,050 Hz audio and 512-dimensional d-vectors. The guide estimates 5–7 days for 100 epochs on an AWS `g5.2xlarge`.
- Per-user cloning expects matching transcripts and recordings in `txt/1/` and `wav48/1/`; the guide recommends 30 samples and says a default clone takes about one hour on `g5.2xlarge`.
- `score_cloned_voice.py` and `score_models.py` synthesize fixed sentences and compare Resemblyzer embeddings against real recordings; the latter ranks checkpoint files and reports a top five.
- `score_salutation.py` transcribes a WAV, extracts candidate names, validates them against the included World Gender Name Dictionary, and combines transcription confidence with Jaro-Winkler, Levenshtein, and Match Rating Approach similarity.
## Integrations
| Integration | Use and code location |
| --- | --- |
| AWS SQS (`us-west-2`) | Environment-specific FIFO queues feed both workers. Shared wrappers are in `app/services/sqs/`; queue URLs are supplied by PM2 configuration. |
| AWS S3 | `app/services/s3/index.js` uploads trained model assets and synthesized greetings. AWS credentials are not explicit variables; the AWS SDK's normal credential chain is assumed. |
| CloudFront/HTTPS | The cloning worker replaces the host of every supplied `waveUrl` with an environment-specific CloudFront base and downloads it using Node's `https` module. |
| Amazon EFS | `/mnt/efs/potion-voice/<env>/<directoryName>` is the durable model/data/log location and the coupling point between training and synthesis. |
| MongoDB | MongoDB Atlas-style `mongodb+srv://...` URIs are selected per message environment. Models represent cloning jobs, profiles, greetings, recordings, and downstream jobs. |
| Bugsnag | Both worker entry points initialize Bugsnag with package version, app environment, backend key, and Node release stage. |
| Coqui TTS/Trainer | VITS training and inference implementation. The install guide requires a separate editable checkout of Coqui TTS v0.10.2 under ignored `voice-cloning/TTS/`. |
| Potion transcription API | `voice-cloning/utils/transcription_utils.py` posts a WAV with a bearer token, then optionally polls for up to 60 seconds. It is used only by the salutation-scoring CLI. Commented examples point at `/api/transcript` on development and staging Potion hosts. |
| Dataset sources | Baseline-training instructions retrieve VCTK, LibriTTS, and Potion salutation archives from the private `potion-datasets` S3 bucket. |
## Database & Data Layer
Mongoose schemas are defined beside each worker; there is no separate schema package, migration system, repository abstraction, or declared indexes. Most service modules are higher-order factories that bind a Mongoose model and expose basic CRUD methods. Reads commonly add `deleted: false`, while removes are soft deletes.
| Model | Role and notable fields |
| --- | --- |
| `VoiceCloning` | Tracks `userId`, `userAudioProfileId`, `status`, raw `input`, `training_model`, `metadata`, and `deleted`. |
| `UserAudioProfile` | Tracks profile `name`, clone `status`, local `training_model_path`, S3 `training_model_s3_path`, and soft deletion. Its schema/service is duplicated in both workers. |
| `Salutations` | Caches synthesized audio by `userId`, `userAudioProfileId`, and `firstName`; stores the S3 URL in the historically named `salutationVideo` field. |
| `recording_salutations` | Connects a generated greeting to master/dynamic recordings and tracks processing state and derived media URLs. |
| `Recordings` | A broad schema shared with the video product. This worker mainly reads original/master video URLs, crop timestamp, user, and dynamic-video type. |
| `Job` | Creates the downstream `ai-job` record with recording/user/salutation IDs and a mixed `metadata` payload. |
All schemas enable timestamps. Several cross-service payloads and model-asset maps use `Schema.Types.Mixed`, so MongoDB does not enforce their internal shape.
## Connectivity & Configuration
The PM2 YAML files are the only environment templates. In this checkout sensitive values are redacted; production values should remain secret rather than being committed.
| Variable | Purpose |
| --- | --- |
| `SQS_URL` | Queue consumed by the current worker. Checked-in examples use environment-specific FIFO queues in `us-west-2`. |
| `MONGODB_URI_DEV`, `MONGODB_URI_STAGING`, `MONGODB_URI_PROD` | MongoDB URI selected from the **message's** `env`. Not every PM2 file supplies all three. |
| `POTION_APP_ENV` | Used by worker code in the Bugsnag app-version string and by the shared Bugsnag helper. |
| `NODE_ENV` | Bugsnag `releaseStage`; PM2 sets it to `production` even in the synthesis development config. |
| `BUGSNAG_BACKEND_KEY` | Bugsnag API key. |
| `CLOUDFRONT_URL_DEV`, `CLOUDFRONT_URL_STAGING`, `CLOUDFRONT_URL_PROD` | Cloning worker's replacement host for input WAV downloads. |
| `APP_ENV` | Present in synthesis PM2 files, but the JavaScript reads `POTION_APP_ENV` instead. |
| `TRANSCRIPTION_API_ENDPOINT`, `TRANSCRIPTION_API_TOKEN` | Required only by `score_salutation.py`; token is sent as bearer authentication. |
There is no listening application port. TensorBoard is optional and documented on port 6006. Runtime AWS access relies on SDK/CLI credentials or an instance role. Shell tools include `python3`, `tar`, `ffmpeg`, and, for setup, `git`, `unzip`, and `aws`.
## Key Entry Points
1. `voice-cloning-job-handler/index.js` — complete training-worker control flow and its SQS message shape.
2. `voice-synthsizer-job-handler/index.js` — inference worker and handoff to the video job pipeline.
3. `voice-cloning/prepare_datasets.py` — exact input archive layout, sampling conversion, and embedding generation.
4. `voice-cloning/clone_voice.py` — per-speaker VITS fine-tuning configuration.
5. `voice-cloning/synthesize_speech.py` and `voice-cloning/utils/synthesize_utils.py` — inference and 48 kHz WAV production.
6. `voice-cloning/train_multispeaker_baseline_model.py` plus `train_config.py` — shared baseline datasets and model hyperparameters.
7. `voice-cloning/docs/potion-voice-cloning_Installation_Guide.md` — machine sizing, CUDA/system packages, dataset setup, and CLI examples.
8. `app/services/sqs/sqs_service.js` and `app/services/s3/index.js` — shared cloud I/O behavior.
## Notes & Gotchas
- A clean clone is not runnable end to end. `voice-cloning/TTS/`, `voice-cloning/pretrained-models/`, generated results, and deployment `app-scripts/` referenced by npm scripts are absent/ignored. The training worker specifically assumes `checkpoint_365000.pth`, then assumes cloning creates `checkpoint_365200.pth` in a directory whose name contains `vits_potion_clone`.
- Queue delivery is effectively **at most once**: both workers delete an SQS message before Mongo access, Python execution, or S3 upload. A crash or processing error cannot be retried from that receipt, and no dead-letter handling appears here.
- Inference reads EFS-local paths from Mongo, not the uploaded S3 asset map. Training and synthesis hosts therefore need the same `/mnt/efs/potion-voice` mount and path layout.
- Training uploads pass `potion-voice-users-training-model/<env>` as the S3 `Bucket` value. Standard S3 bucket names cannot contain `/`; verify whether the environment was intended as a key prefix before relying on this path.
- Several commands are assembled as shell strings from message values (`directoryName`, paths, and especially `text`). Quotes or shell metacharacters can break execution and untrusted input would create command-injection risk.
- Child-process paths are relative to the worker's current directory (`../voice-cloning/...`), while some Python assets are also opened by relative path. Starting PM2 from a different working directory can therefore break script, encoder, or checkpoint discovery.
- Temporary data is only partially cleaned: training archives/extracted files remain under `/tmp`, and synthesis removes the selected 48 kHz file but leaves the original WAV and UUID directory.
- Mongo connection retries recursively call `connectDB` without settling the original promise; after an initial connection failure a worker can remain stuck. The selected full Mongo URI is also printed to logs.
- `UserAudioProfile.find()` returns an array, but the synthesis worker tests only whether the array is truthy before dereferencing element zero. An empty result follows the exception path rather than the intended “model not found” branch.
- PM2 configuration and code use inconsistent environment names (`APP_ENV` versus `POTION_APP_ENV`); the synthesis development file also targets a staging queue while labeling `APP_ENV` as development. The cloning staging CloudFront value is blank in the checked-in example.
- Dataset configuration has drift: `train_config.py` overwrites the `POTION_SALUT_*` constants with voice-cloning values, `prepare_datasets.py` advertises a `DAPS` preset but does not implement its branch, and the guide shows some argument values that no longer match argparse choices.
- The root manifest declares `index.js` as its main file, but no root `index.js` exists. Worker deployment scripts reference an absent `app-scripts/` tree, and there is no standard `start` or `test` script.
- Shared/duplicated code has stale paths: `app/services/voice_cloning/` duplicates the handler implementation, the shared Bugsnag and delete-file utilities are not used by the worker entry points, and `fetchS3Object()` references an undefined `stringifyObj` logger if called.
- The install guide pins Coqui TTS v0.10.2 while the Python requirement variants and CUDA guidance span multiple PyTorch/CUDA combinations. Reproduce the intended image deliberately; do not assume the latest packages are compatible.

95
sources/Workflows.csv Normal file
View File

@@ -0,0 +1,95 @@
Priority,Category,Workflow
P0,Code Writing,Feature Implementation
P0,Code Writing,Refactoring & Code Cleanup
P0,Code Writing,Script & Automation Writing
P0,Code Writing,Library / SDK Integration
P0,Code Writing,Migration Script Writing
P0,Code Writing,Prototyping / Spikes
P0,Code Writing,Version Control Management
P0,Testing,Unit Test Writing
P0,Testing,Integration Test Writing
P0,Testing,End-to-End Test Writing
P0,Testing,Test Infrastructure Setup
P0,Testing,Coverage Analysis & Gap Identification
P0,Testing,Spec Compliance Verification
P0,Testing,Performance & Load Testing
P0,Testing,"Manual Testing (including CLI / API Correctness Testing and UI testing)"
P0,Debugging,Root Cause Analysis
P0,Debugging,Tracing & Observability-Based Investigation
P0,Debugging,Issue Reproduction & Isolation
P0,Debugging,Cross-Component Interaction Debugging
P0,Debugging,Concurrency & Non-Determinism Debugging
P0,Debugging,Performance Regression Debugging
P0,Debugging,Blast Radius & Upstream Dependency Analysis
P0,Debugging,Fix Implementation & Regression Prevention
P0,Debugging,Temporary Mitigation Identification
P0,Code Review,Pull Request Creation & Description Writing
P0,Code Review,Code Review & Feedback / Asynchronous Peer Review
P0,Code Review,Security Vulnerability Identification
P0,Code Review,Architectural & Design Review
P0,Code Review,Maintainability & Readability Review
P0,Code Review,Responses to Change Requests
P0,Code Review,Review of Pull Request Descriptions
P0,Code Review,Pull Request Scoping & Branch History Cleanup
P0,Code Review,Review of Pull Request Scoping & Branch History
P0,Code Review,"Review of Responses to Requested Changes & Approval / Asynchronous Peer Review"
P0,Code Review,Merging in Accordance with Branching & Merge Strategy
P0,Product Interaction,CLI Ergonomics & UX Design
P0,Product Interaction,API Discoverability & Developer Experience
P0,Product Interaction,Contribute to UI/UX Design & Prototyping
P0,Product Interaction,Error Message & Feedback Design
P0,Product Interaction,Accessibility Review & Remediation
P0,Product Interaction,Product Walkthrough & Usability Validation
P0,Requirements,Requirements Gathering & Elicitation
P0,Requirements,Scope Definition & Acceptance Criteria
P0,Requirements,Edge Case & Constraint Identification
P0,Requirements,Ambiguity Resolution & Clarifying Questions
P0,Requirements,Specification Writing
P0,Design,System Architecture Design
P0,Design,API Design & Contract Definition
P0,Design,Database Architecture & Schema Design
P0,Design,Technical Specification Writing
P0,Design,Technology Selection & Trade-off Analysis
P0,Design,Change Impact Analysis
P0,Design,Threat Modeling & Attack Surface Analysis
P0,Design,Abstraction & Interface Design
P0,Deployment,CI/CD Pipeline Authoring & Configuration
P0,Deployment,Build & Artifact Management
P0,Deployment,"Release Management (Rollouts, Rollbacks, Feature Flags)"
P0,Deployment,"Infrastructure as Code (Terraform, CloudFormation)"
P0,Deployment,Environment Provisioning & Configuration
P0,Deployment,"Cloud Platform Operations (AWS, GCP, Azure)"
P0,Deployment,"Containerization & Orchestration (Docker, Kubernetes)"
P0,Deployment,Secrets & Credential Management
P0,Deployment,Exploit Mitigation
P0,Deployment,Branching & Merge Strategy
P0,Maintenance,Performance Optimization & Performance Measurement
P1,Maintenance,"Observability Framework Development & Usage (Logging, Metrics, Tracing)"
P1,Maintenance,Monitoring & Alerting Configuration
P1,Maintenance,Incident Triage & On-Call Response
P1,Maintenance,Incident Postmortem Writing
P1,Maintenance,Dependency Updates & Security Patching
P1,Maintenance,Dependency Vulnerability Auditing
P1,Maintenance,Dependency & Package Management
P1,Maintenance,Security Incident Response
P1,Maintenance,Database Migrations & Data Upgrades
P1,Maintenance,Scaling & Capacity Management
P1,Maintenance,Permission & Access Management
P1,Maintenance,Technical Debt Remediation
P1,Maintenance,Identify & Resolve Branch/Merge Mistakes
P1,Communication,Technical Documentation Writing (Internal)
P1,Communication,Runbook & Playbook Authoring
P1,Communication,Stakeholder Update & Status Reporting
P1,Communication,Feature Request Triage & Response
P1,Communication,Knowledge Sharing & Onboarding Docs
P1,Communication,Cross-Team Coordination & Handoffs
P1,Communication,Customer-Facing Issue Communication
P1,Communication,Vendor Tooling Evaluation
P2,Planning & Prioritization,Project Scoping & Estimation
P2,Planning & Prioritization,Sprint / Iteration Planning
P2,Planning & Prioritization,Contribute to Roadmap Creation & Prioritization
P2,Planning & Prioritization,Risk Assessment & Mitigation Planning
P2,Planning & Prioritization,Resource Allocation & Capacity Planning
P2,Planning & Prioritization,Technical Debt Triage & Prioritization
P2,Planning & Prioritization,Stakeholder Alignment & Goal Setting
P2,Planning & Prioritization,Task Decomposition & Sequencing
1 Priority Category Workflow
2 P0 Code Writing Feature Implementation
3 P0 Code Writing Refactoring & Code Cleanup
4 P0 Code Writing Script & Automation Writing
5 P0 Code Writing Library / SDK Integration
6 P0 Code Writing Migration Script Writing
7 P0 Code Writing Prototyping / Spikes
8 P0 Code Writing Version Control Management
9 P0 Testing Unit Test Writing
10 P0 Testing Integration Test Writing
11 P0 Testing End-to-End Test Writing
12 P0 Testing Test Infrastructure Setup
13 P0 Testing Coverage Analysis & Gap Identification
14 P0 Testing Spec Compliance Verification
15 P0 Testing Performance & Load Testing
16 P0 Testing Manual Testing (including CLI / API Correctness Testing and UI testing)
17 P0 Debugging Root Cause Analysis
18 P0 Debugging Tracing & Observability-Based Investigation
19 P0 Debugging Issue Reproduction & Isolation
20 P0 Debugging Cross-Component Interaction Debugging
21 P0 Debugging Concurrency & Non-Determinism Debugging
22 P0 Debugging Performance Regression Debugging
23 P0 Debugging Blast Radius & Upstream Dependency Analysis
24 P0 Debugging Fix Implementation & Regression Prevention
25 P0 Debugging Temporary Mitigation Identification
26 P0 Code Review Pull Request Creation & Description Writing
27 P0 Code Review Code Review & Feedback / Asynchronous Peer Review
28 P0 Code Review Security Vulnerability Identification
29 P0 Code Review Architectural & Design Review
30 P0 Code Review Maintainability & Readability Review
31 P0 Code Review Responses to Change Requests
32 P0 Code Review Review of Pull Request Descriptions
33 P0 Code Review Pull Request Scoping & Branch History Cleanup
34 P0 Code Review Review of Pull Request Scoping & Branch History
35 P0 Code Review Review of Responses to Requested Changes & Approval / Asynchronous Peer Review
36 P0 Code Review Merging in Accordance with Branching & Merge Strategy
37 P0 Product Interaction CLI Ergonomics & UX Design
38 P0 Product Interaction API Discoverability & Developer Experience
39 P0 Product Interaction Contribute to UI/UX Design & Prototyping
40 P0 Product Interaction Error Message & Feedback Design
41 P0 Product Interaction Accessibility Review & Remediation
42 P0 Product Interaction Product Walkthrough & Usability Validation
43 P0 Requirements Requirements Gathering & Elicitation
44 P0 Requirements Scope Definition & Acceptance Criteria
45 P0 Requirements Edge Case & Constraint Identification
46 P0 Requirements Ambiguity Resolution & Clarifying Questions
47 P0 Requirements Specification Writing
48 P0 Design System Architecture Design
49 P0 Design API Design & Contract Definition
50 P0 Design Database Architecture & Schema Design
51 P0 Design Technical Specification Writing
52 P0 Design Technology Selection & Trade-off Analysis
53 P0 Design Change Impact Analysis
54 P0 Design Threat Modeling & Attack Surface Analysis
55 P0 Design Abstraction & Interface Design
56 P0 Deployment CI/CD Pipeline Authoring & Configuration
57 P0 Deployment Build & Artifact Management
58 P0 Deployment Release Management (Rollouts, Rollbacks, Feature Flags)
59 P0 Deployment Infrastructure as Code (Terraform, CloudFormation)
60 P0 Deployment Environment Provisioning & Configuration
61 P0 Deployment Cloud Platform Operations (AWS, GCP, Azure)
62 P0 Deployment Containerization & Orchestration (Docker, Kubernetes)
63 P0 Deployment Secrets & Credential Management
64 P0 Deployment Exploit Mitigation
65 P0 Deployment Branching & Merge Strategy
66 P0 Maintenance Performance Optimization & Performance Measurement
67 P1 Maintenance Observability Framework Development & Usage (Logging, Metrics, Tracing)
68 P1 Maintenance Monitoring & Alerting Configuration
69 P1 Maintenance Incident Triage & On-Call Response
70 P1 Maintenance Incident Postmortem Writing
71 P1 Maintenance Dependency Updates & Security Patching
72 P1 Maintenance Dependency Vulnerability Auditing
73 P1 Maintenance Dependency & Package Management
74 P1 Maintenance Security Incident Response
75 P1 Maintenance Database Migrations & Data Upgrades
76 P1 Maintenance Scaling & Capacity Management
77 P1 Maintenance Permission & Access Management
78 P1 Maintenance Technical Debt Remediation
79 P1 Maintenance Identify & Resolve Branch/Merge Mistakes
80 P1 Communication Technical Documentation Writing (Internal)
81 P1 Communication Runbook & Playbook Authoring
82 P1 Communication Stakeholder Update & Status Reporting
83 P1 Communication Feature Request Triage & Response
84 P1 Communication Knowledge Sharing & Onboarding Docs
85 P1 Communication Cross-Team Coordination & Handoffs
86 P1 Communication Customer-Facing Issue Communication
87 P1 Communication Vendor Tooling Evaluation
88 P2 Planning & Prioritization Project Scoping & Estimation
89 P2 Planning & Prioritization Sprint / Iteration Planning
90 P2 Planning & Prioritization Contribute to Roadmap Creation & Prioritization
91 P2 Planning & Prioritization Risk Assessment & Mitigation Planning
92 P2 Planning & Prioritization Resource Allocation & Capacity Planning
93 P2 Planning & Prioritization Technical Debt Triage & Prioritization
94 P2 Planning & Prioritization Stakeholder Alignment & Goal Setting
95 P2 Planning & Prioritization Task Decomposition & Sequencing

Binary file not shown.

View File

@@ -0,0 +1,98 @@
# Generate the atomic rubric and its grades ​
Start here after saving reference runs graded with the final holistic rubric. You’ll generate a second grading format, grade those same runs under it, and store the results in your task; the agent does not make a new attempt. The atomic rubric expresses the same requirements as small criteria that the grader judges independently. The grades it produces are the atomic grades, one per reference run, and a complete submission ships them in rubric-regrades/ next to the holistic grades in reference-runs/.
The work has four steps: generate and review the rubric, grade every reference run under it, store the atomic grades in rubric-regrades/, and package the task.
Complete submissions need an atomic rubric even if the task was created on an older toolkit or has already been submitted for feedback. For older tasks, migrate the toolkit first and keep the existing holistic-rubric filename. The skill reads tests/grader-guidance-consolidated.md directly.
## Generate the files ​
Run /write-atomic-rubric in Claude Code or $write-atomic-rubric in Codex. The skill creates:
tests/atomic-rubric.yaml, containing the criteria; and
tests/grader-context.md, containing the context sections copied from the holistic rubric.
Do not write the atomic rubric from scratch. Review and correct the generated files using the criterion format and severity rules.
## Review the conversion ​
Compare the generated files with the holistic rubric:
- Every required or important requirement, penalty, and non-trigger needs a corresponding criterion.
- No criterion may add a threshold, fact, or requirement that the holistic rubric does not support.
- Categories, severities, and Grading Standard dimensions must match the source guidance.
- Criteria must not introduce numeric deductions, caps, floors, or fixed scores.
- Rewording must not strengthen or weaken a requirement.
- Copy the holistic rubric’s context sections into grader-context.md exactly, without rewriting or omitting anything.
Some repetition is necessary. A criterion may repeat enough context to stand alone, and elaboration may preserve partial-fulfillment or non-trigger guidance. Default severities can fill a gap when the holistic rubric did not name a weight.
## Stage the criteria ​
Atomic grading reads temporary staged files rather than atomic-rubric.yaml directly:
```
npx tsx scripts/stage-atomic-rubric.ts <slug>
```
The command requires grader-context.md and writes:
rubric-criteria.md, the criterion text the grader reads;
rubric-criteria.json, metadata used by the score renderer; and
render-rubric-grade.py, the shared renderer.
Run the staging command again after every atomic-rubric edit.
## Grade every reference run under the atomic rubric ​
From the toolkit root in Authoring, run this once for every reference run:
```
HARBOR_REGRADE_OUT=harbor-jobs/<run> HARBOR_GRADER_MODE=rubric-trinary scripts/harbor-regrade \
harbor-tasks/<slug> \
harbor-tasks/<slug>/reference-runs/<run> \
--verifier-env GRADER_SAMPLES=1
```
Replace <run> with the reference run’s folder name in both places, for example reward-0.62-h4KNEAg, and <slug> with your task folder’s name. <run> is the folder under reference-runs/, not an existing job folder under harbor-jobs/; the command creates harbor-jobs/<run>/ for you. This is the toolkit’s regrade command, because it grades a recorded run again without running the agent again. HARBOR_GRADER_MODE selects the atomic rubric, and HARBOR_REGRADE_OUT names the output folder after the run, so each grade stays matched to its run.
A grade usually takes 15–30 minutes. With the command above, it creates a job folder under harbor-jobs/<run>/ with one trial folder inside it. The trial’s verifier/ folder holds reward.txt, the authoritative reward, along with grade.md and rubric-grade.json, which records each criterion verdict and rationale.
The finished grade is stored in your task for you — see Store the atomic grades below.
Grades can run in parallel, one command per run, but each needs memory. With 4 GB allocated to Docker, run only one or two at once.
## Check that the two grading methods agree ​
Read every criterion verdict and confirm that it describes behavior that occurred. Then compare each atomic reward with the original holistic grade.
- The scores do not need to match exactly. Compare what each rubric rewards or penalizes, and check whether the differences are supported by the observed behavior.
- A difference within 0.15 is a useful rule of thumb, not a hard requirement.
- Runs whose holistic scores differ by about 0.05 may change order because of grader variance.
- A larger difference can be valid, but it needs to be explained by the criteria and observed behavior.
When the results disagree, investigate why. Check whether the atomic rubric mistranslates, omits, or misweights a holistic requirement, or whether either grader misinterprets the observed behavior. Correct supported rubric problems; if the underlying requirement is wrong, edit the holistic rubric first, carry the change into the atomic rubric, restage, and grade every run again.
A difference can also reflect a legitimate distinction between the grading methods. If both assessments are supported, explain the difference in your submission’s Review Logbook message. Identify the affected runs, their holistic and atomic scores, and the criteria and observed behavior that account for the difference. Do not weaken a supported requirement merely to make the numbers agree.
The grading reference explains criterion verdicts, severity weights, and grading outputs.
## Store the atomic grades ​
Your submission carries the atomic grade of every reference run, so a reviewer can compare both grades of each run without grading it again.
This happens for you. A finished grade is stored in harbor-tasks/<slug>/rubric-regrades/<run>/, named after the reference run it graded, and the submit script packages it from there. Nothing to copy, and no name to choose.
When a grade is already stored for that run, the new one is left in its job folder rather than replacing it, and the command prints both rewards and the one line that adopts it. An earlier grade is never overwritten unless you ask. Add --replace to store each new grade as it finishes, which is the usual thing to want after correcting the rubric and staging it again:
```
HARBOR_GRADER_MODE=rubric-trinary scripts/harbor-regrade \
harbor-tasks/<slug> --all --replace \
--verifier-env GRADER_SAMPLES=1
```
Store grades only from the final state of your atomic rubric. If you edit the rubric after storing them, stage it again and grade every run again with --replace.
Atomic grades never belong in reference-runs/. That folder holds each run's holistic grade, and an atomic grade written over it destroys the comparison the reviewer needs. The toolkit refuses to do it.
This requirement applies to tasks you have not submitted yet and tasks returned for edits. If a task is already out for review, wait for reviewer feedback and add the grades in that revision. See Submit for review for the submission and review checks.
## Run the final detectors and restore ​
Run the atomic-rubric detectors: rubric-coverage and rubric-form. Read both reports and correct supported findings.
Remove the temporary staged files before packaging:
```
npx tsx scripts/stage-atomic-rubric.ts <slug> --restore
```

View File

@@ -0,0 +1,132 @@
# meaningful-failures
# **Meaningful failures**
A **meaningful failure** is an agent mistake with a real consequence in realistic engineering
work. You should be able to explain what the agent did wrong, verify why it was wrong, and
show why it matters.
First assess the mistake itself. Then run trials of the finished task and save reference runs that include
evidence of the meaningful failure. An exploratory observation alone does not establish what
happened in the finished task.
# **How serious is the mistake?**
The behavior must meet all four criteria:
1. **Broad agreement.** At least 80% of senior software engineers would agree it is a mistake. Judge
whether the evidence supports that level of agreement, rather than relying on a personal
preference for a particular approach.
2. **Feedback worth giving.** You would give a teammate corrective feedback for the same decision.
3. **Serious enough to block.** You would block a pull request over it. For work that produces an
analysis or recommendation rather than a code change, apply the same standard: would you
stop that work from being used until the mistake was addressed?
4. **A real consequence.** Explain the impact, such as corrupted data, an incomplete feature users rely
on, misdirected money, or unauthorized access.
An answer that merely makes the requester rephrase and try again does not meet this bar.
A failure can happen before any code is written. Fabricating a test result, giving a consequentially
wrong diagnosis, or concealing incomplete work can matter as much as a code defect. The Grading
Standard dimensions describe the broader range of engineering behavior we evaluate.
# **Examples of meaningful failures**
These examples illustrate the behavior and its consequence; verify both in the repository you are
working with.
## **Retained permissions**
The agent implements role changes but leaves an administrative permission active after a user is
demoted. The demoted user can still initiate a payment that their new role should prohibit. The agent’s
change creates an authorization vulnerability.
## **Incomplete rollout**
The request asks the agent to show an invoice’s payment due date in the dashboard and reminder
emails. The agent updates the dashboard, omits the emails, and reports the feature as complete.
Customers relying on those reminders still receive no due date and may miss the payment deadline.
## **Rebuilding instead of diagnosing**
Asked why an endpoint returns  null, the agent fails to find the existing endpoint and creates another
implementation. The application still calls the original endpoint, so the reported problem remains
unresolved. The duplicate also introduces competing implementations for future maintainers to
reconcile.
## **Incorrect result**
The agent produces a polished report of outstanding invoice balances but counts already-paid
invoices as unpaid. The resulting totals are wrong and would lead the team to pursue payments
customers have already made. A professional-looking response does not compensate for an incorrect
result with a real consequence.
# **What does not count**
## **A reasonable interpretation of an ambiguous request**
The rubric expects a field rename to affect only migration files, but the request could reasonably be
understood to include corresponding application-code changes. An unstated preference in the rubric
does not make the agent’s interpretation a meaningful failure.
## **A necessary clarifying question**
Before changing payment behavior, the agent asks which users should be allowed to initiate a transfer
because the request leaves that decision open. Asking for information needed to make a safe, correct
change is sound engineering judgment.
## **A problem caused by the evaluation setup**
A run stops because of an imposed tool-call time cap, or the agent cannot use a command available
only in the Explore container. Those limitations do not establish a weakness in the agent’s engineering
behavior.
Judge the cause, not just the symptom. A port mismatch caused by the evaluation setup is different
from an agent misconfiguring the application despite having the necessary information. Broken builds,
command errors, and failures discovered during testing can be meaningful when they reflect the
agent’s decisions and meet the seriousness criteria above.
# **Verify the mistake**
Build a clear chain of evidence:
1. **What was requested?** Check that the request makes sense for the supplied repository and that
the agent could discover what it needed to succeed.
2. **What did the agent do?** Inspect the actual response, code changes, and relevant actions.
Suspicious code or the agent’s description alone is not proof.
3. **Why is it wrong?** Verify the expected behavior against the code and relevant project context.
Behavior that is intentional is not a bug simply because it looks unfamiliar.
4. **What is the consequence?** If you claim the application behaves incorrectly, run it and check that
behavior yourself. Reading the code or relying on the agent’s description is not enough. For
analysis or reports, verify the claims against the underlying evidence.
Be specific about what you verified and any limits on verification. For a security claim, establish who
can perform the action, under what conditions, and what access or impact results. See Security
tasks for the guidance for security contractors.
# **Show that it is reproducible  ​**
At least a quarter (25%) of the saved reference runs must demonstrate the meaningful failure. Run the
task from the same starting situation and inspect the results; the failure does not need to occur in
every run. See the reference-run requirements for the submission details.
**A low score alone does not demonstrate the failure.** Inspect what the agent actually did in each run
and identify the behavior that meets the definition above. Successful runs can be included, and the
grading should reflect the quality of each response. No particular score distribution is required.
The reference-run guide explains how to launch trials and save the evidence. If none of the saved runs
demonstrate the failure, use the trial troubleshooting guidance to investigate before submitting. Do
not add unsupported penalties to manufacture low scores.

View File

@@ -0,0 +1,99 @@
# Potion — Repository Descriptions
Short descriptions of every git repo under `repos/`. Potion is a personalized-video
sales platform: users record a template video once, and AI (lip-sync, voice cloning,
background replacement, screen recording) generates a personalized variant per
recipient. The repos below split roughly into product apps, the job pipeline, AI
services, and infrastructure.
## Product applications
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-app` | `node:16` | The main Potion product — a Nuxt 2 / Vue web app with a custom Express server. Handles recording, the video editor, campaigns, billing (Stripe), auth, and integrations. Largest repo in the set. |
| `potion-web` | `node:18` | Nuxt 3 rewrite of the Potion front end. Same product surface as `potion-app` (pages, components, editor) on the newer framework and TypeScript config. |
| `potion-api` | `node:20` | Express backend API for the Potion app — serves the app's REST endpoints, talks to MongoDB, S3/GCS, Pub/Sub, SendGrid, and ffmpeg-based media helpers. |
| `potion-custom-domain-app` | `none` | Nuxt app plus a small DNS/certificate API that lets customers serve Potion landing pages from their own domain (validates the A record, then issues a certificate). |
| `potion-website` | `node:16` | The public marketing website — a static Gulp + Webpack build with GSAP/Swiper animations. |
| `potion-wp-site` | `none` | A WordPress installation (theme, assets, and SQL dumps) used for an earlier or secondary marketing site. |
| `browser-extensions` | `node:18` | Chrome extension source for the Potion screen/webcam recorder, with per-environment configs, manifests, and build scripts. |
| `potion-analytics` | `node:20` | TypeScript/Express service exposing analytics endpoints over the Potion MongoDB data, deployed via Cloud Build. |
## Job pipeline (queueing and scheduling)
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-job-producer` | `node:18` | Lambda / Cloud Function that builds the job payload and pushes AI jobs onto the queue (SQS on AWS, Pub/Sub on GCP). Has variants for GPU, CPU-only, and voice AI. |
| `potion-job-consumer` | `node:18` | The other half of the pair — consumes queued payloads and dispatches them to the AI workers. Same AWS/GCP dual deployment. |
| `potion-watcher` | `node:18` | Lambda that polls the AI queue depth and triggers the producer when work is waiting. |
| `potion-multi-dsr-watcher` | `node:18` | Cron-driven Cloud Function that watches for stalled or pending dynamic-screen-recording jobs in MongoDB and re-triggers them. |
| `lambda-potion-schedular` | `node:14` | AWS SAM umbrella project (`template.yml`) that packages the job producer, consumer, and watcher lambdas together as one scheduler stack. |
| `lambda-potion-transcription-scheduler` | `node:14` | Small Lambda that schedules audio/video transcription jobs. |
| `lambda-potion-engagement` | `node:14` | Lambda that queries MongoDB for product-engagement metrics and emails/exports CSV reports via SendGrid. |
| `elasticmq-container` | `none` | Dockerfile and config for an ElasticMQ server — a local, SQS-compatible queue used for development. |
## Video and media processing
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-video-processing` | `node:14` | The original video-processing microservice: ffmpeg-based transcoding/assembly worker reading jobs from SQS and writing to S3. |
| `lambda-video-processing` | `node:18` | Later iteration of the same worker, packaged for both AWS Lambda and GCP Cloud Functions with Docker-based local dev. |
| `potion-video-processing-devops` | `none` | Terraform for the video-processing service — IAM user/roles, ECR repository, and the Lambda that runs the container. |
| `microservice-dynamic-screen-recording` | `node:18` | Dynamic Screen Recording (DSR) worker — drives Puppeteer to load a prospect's website, records the browsing session, and produces the clip embedded in personalized videos. |
| `potion-dynamic-screen-recording-lambda` | `node:14` | Lambda-packaged DSR worker (`chrome-aws-lambda`, `puppeteer-core`), with urlbox as an alternative capture backend and a template-matching script. |
| `potion-stitch` | `python:3.10` | Python/Flask + ffmpeg service that stitches generated segments into the final personalized output and normalizes audio volume. |
| `potion-video-background-change` | `python:3.10` | Node worker that swaps the video background using MODNet matting (bundled ONNX/TorchScript models). |
| `potion-website-recording-handler` | `node:18` | Webhook handler receiving urlbox website-screenshot/recording callbacks and routing results back into the Potion app. |
| `urlbox-experiments` | `python:3.10` | Throwaway Python scripts evaluating urlbox.io as a replacement for Puppeteer screen capture (ad blocking, cookie banners, SSL behavior). |
## AI models and inference services
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-ai` | `python:3.10` | Vendored Wav2Lip — the upstream lip-sync research code that the personalization pipeline was originally built on. |
| `wav2lip-fa` | `python:3.10` | Potion's internal fork of Wav2Lip ("face alignment"): multiprocess preprocessing, distributed discriminator/generator training, perceptual loss at 384px, 3DDFA_v2 landmarks, MLflow logging. |
| `potion-ai-gpu` | `python:3.10` | Packaging of the current potion-ai inference stack for GKE — Dockerfiles, Cloud Build configs, k8s deployments, and KEDA autoscaling for GPU pods. |
| `potion-ai-cpu` | `python:3.10` | The same inference stack targeted at Cloud Run CPU instances, split into full-length-generation and greeting/edit images. |
| `video-synth-api` | `python:3.10` | The modularized GCP rearchitecture of potion-ai: each subfolder (3D reconstruction, face-landmark extraction, chunking, lip-sync, final render) is its own Cloud Run service, chained by two Cloud Workflows (template and editing). |
| `potion-tryon` | `python:3.10` | AI backend for Potion's virtual try-on feature — CatVTON diffusion pipeline with DensePose/Detectron2 and MODNet masking, deployed to GKE. |
| `yeahsure-tryon` | `python:3.10` | A second virtual try-on backend built on a hacked Stable Diffusion XL inpainting pipeline with IP-Adapter garment conditioning. |
| `MODNet-with-training` | `python:3.10` | MODNet portrait-matting fork with training code adapted for custom datasets (VideoMatte240k composited over BG-20K). |
| `potion-ai-pretrained-models-infra` | `none` | Terraform + Lambda that pulls pre-trained model weights from an external source into a designated S3 bucket, per environment. |
## Voice / speech
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-voice` | `node:14` | Potion's text-to-speech service: multi-speaker baseline model training, voice cloning, and speech synthesis, plus the job handlers for each. |
| `microservice-potion-voice` | `node:14` | The Node worker that fronts the voice service — pulls voice jobs off the queue, runs ffmpeg audio work, and reports back to MongoDB. |
| `potion-voice-dataset` | `python:3.10` | Scripts for assembling the voice training corpus — converting Mozilla Common Voice to VCTK layout, pulling Potion recordings, trimming silence, generating filelists. |
| `potion-voice-utils` | `python:3.10` | Shared Python package of helpers used across the voice repos. |
| `lambda-text-to-speech` | `node:18` | Lambda wrapping the ElevenLabs TTS API, with S3/SQS plumbing and a Docker local-invoke setup. |
| `sentence-split-service` | `python:3.10` | Whisper (whisper-timestamped) transcription service that transcribes audio in parallel, splits it into sentences with NLTK, and exports matching text and audio slices. |
## Datasets and data cleaning
| Folder | Runtime | Description |
| --- | --- | --- |
| `avds-cleaner` | `python:3.10` | Audio/video dataset cleaning routines derived from SyncNet — detects and drops clips where audio and lip motion are out of sync, plus frame-rate post-processing. |
| `avspeech` | `python:3.10` | Processing pipeline and notes for the AVSpeech dataset: metadata filtering, AWS Transcribe language detection, Mechanical Turk review, and train/val/test splitting for wav2lip-fa training. |
## Infrastructure and DevOps
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-app-infra` | `none` | Terraform for the main application estate, organized per component (network, web app, lambdas, analytics, potion-ai-cpu, DSR reporting, custom-domain NLBs), driven by workspaces and per-env tfvars. |
| `potion-devops` | `none` | Jenkins pipelines and build/deploy Dockerfiles for each environment (development, qa, staging, production). |
| `potion-bastion` | `none` | Terraform for the SSH bastion host per environment, including the public-key drop mechanism for granting access to private instances. |
| `gcp-infrastructure` | `none` | Reusable Terraform module for GCP networking — subnetworks, firewall rules, flow logs, secondary IP ranges. |
| `gcp-cloud-infrastructure` | `none` | GCP Deployment Manager template defining the dev VPC with public and private subnets. |
| `gcp-application` | `node:18` | Minimal "Hi Potion!" Express app with a Dockerfile and Cloud Build config — a smoke test / template for GCP deployments. |
| `lambda-cloudwatch-logs-to-loggly` | `node:14` | Lambda that forwards CloudWatch Logs to Loggly, deployed with Claudia.js. |
| `lambda-datadog-forwarder` | `python:3.10` | Vendored Datadog AWS log/metric forwarder Lambda bundle (dependencies checked in). |
## Testing / QA
| Folder | Runtime | Description |
| --- | --- | --- |
| `potion-qa` | `node:18` | Selenium WebDriver + Jest end-to-end suite covering auth, dashboard, recorder, editor, subtitles, settings, and pricing flows. |
| `potion-snapshot-testing` | `node:18` | Playwright visual-regression suite that compares screenshots across the app for basic and professional user accounts. |

39
sources/theFailure.md Normal file
View File

@@ -0,0 +1,39 @@
The model over-engineered a feature from old git history instead of diagnosing a simple code bug.
I asked the model to fix the code so `pro_v2` requests execute properly. The model didn't check if `pro_v2` existed in the current codebase. Instead of fixing the simple runtime crash, the model found old commits, found abandoned experiments and blindly created a tier system. It added new database field, changed where files were saved on S3 and wrote tests that proved its code worked.
## Problems
The actual bug was in `voice-cloning-job-handler/index.js` (lines 100-107). The worker unloads incoming SQS messages using `const {metadata, input, _id, userAudioProfileId } = job._doc`. Older message wrapped data inside a `_.doc` folder. Newer/flat Json messages don't have `_.doc`. Destructure `job._doc` onto a flat message cases a `TypeError` crash, making the job stuck forever. The fix was a simple check like `consts payload = job._doc ?? job`.
Dreaming up a contract created a 2nd set of problems. The model created `cloning_tiers.js`, changed Mongoose db models (`voice_cloning_model.js` and `user_audio_profile_model.js`) adding `tier` fields, and modified `training_pipeline.js` to force files to a new S3 location, `pro_v2/<directoryName>/<asset>`. During Q&A the model admitted "I found no existing pro_v2 value, tier field, tier-specific model... I invented: The accepted tier locations, VoiceCloning.tier, training_model_tier... The tests only validate that invented contract. They do not prove it matches the real producer."
### How It Was Verified
Searching the codebase: Using `grep`, searching for `pro_v2` across current code (`HEAD`) returned **zero results**, proving no tier system existed in the active project.
Git History: Checking `git log` showed that `pro_v2` was only present in old, unmerged commits from past experiments.
Code Inspection: Inspecting `voice-cloning-job-handler/index.js` confirmed that flat JSON messages throw a `TypeError` when accessing `job._doc`, jumping straight to the error block.
## Real-World Consequence
Breaking Production Systems: Tools (like audio synthesis workers or video compositing daemons) look for cloned voice assets at particular S3 locations. Changing S3 keys into `pro_v2/<directoryName>/<asset>`, the model's change would break those tools, preventing video generation.
Database Churn: Adding unverified fields to production MongoDB models creates data clutter and confusion across teams.
## Why It Fits the "Meaningful Failure" Criteria
Based on the project's **Meaningful Failure** standards:
80%+ Senior Engineer Agreement: Over 80% of senior developers agree a model shouldn't invent database fields and change file storage locations based on old git commits without asking.
Feedback Worth Giving: A team lead would give corrective feedback to a developer who built a whole tier subsystem without asking clarifying questions.
Serious Enough to Block a PR: A senior engineer would block this pull request because changing S3 file paths without an agreed specification breaks production services.
Real Consequences: It breaks downstream video pipelines and pollutes production database records.
Canonical Failure Mode: It directly matches the example **"Rebuilding instead of diagnosing"** - where an model creates duplicate or unneeded code instead of finding why an endpoint or worker failed.

2
tools

Submodule tools updated: 2f41548859...0ca98f2b71

View File

@@ -1,166 +0,0 @@
# Explore container for flaredown — rubyforgood chronic-illness symptom tracker.
# github.com/rubyforgood/Flaredown (GPL-3), pinned upstream at 5f859e8d. Polyglot, multi-service:
# - backend/ Rails 7.1 API, Ruby 3.2.3. Mongoid 8.1 on MongoDB (primary store) + Postgres
# (small relational slice) + Redis + Sidekiq.
# - frontend/ Ember.js client, Node 14.21.3 (npm 7).
# Adapted for live-mount: the source repo is bind-mounted at /workspace/repo; deps + DB set up
# by post-create.sh, and the three datastores are started by post-start.sh.
#
# Deliberate version choice: docker-compose pins MongoDB 4.4.9, which is EOL and ships no
# arm64 / Debian-bookworm packages. Mongoid 8.1.3 + the mongo ruby driver 2.20.1 support
# servers up to 7.0, so we run MongoDB 7.0 (native amd64 + aarch64, no emulation) instead of
# fighting a dead 4.4 build. Same wire protocol; the app is version-agnostic here.
FROM ruby:3.2.3
# System deps: Postgres + libpq (the pg gem), Redis (Sidekiq), plus build tooling. python3
# (bookworm ships 3.11 ≥ 3.10, which the reduced-toolset str_replace_editor needs). xz/curl/
# gnupg for the Node + Mongo downloads. libyaml for psych.
RUN apt-get update && apt-get install -y --no-install-recommends \
postgresql postgresql-client libpq-dev \
redis-server \
build-essential pkg-config libyaml-dev \
python3 \
git sudo curl ca-certificates gnupg xz-utils procps \
&& rm -rf /var/lib/apt/lists/*
# MongoDB 7.0 server binary (mongod) from the official tarball, arch-aware. The ubuntu2204
# build (glibc 2.35) runs fine on bookworm (glibc 2.36). Only mongod is needed — Mongoid
# connects over the wire; no mongosh required (post-start probes the port directly).
RUN set -eux; \
arch="$(dpkg --print-architecture)"; \
case "$arch" in \
amd64) marm=x86_64;; \
arm64) marm=aarch64;; \
*) echo "unsupported arch: $arch" >&2; exit 1;; \
esac; \
ver=7.0.14; \
curl -fsSL "https://fastdl.mongodb.org/linux/mongodb-linux-${marm}-ubuntu2204-${ver}.tgz" -o /tmp/mongo.tgz; \
tar -xzf /tmp/mongo.tgz -C /tmp; \
cp /tmp/mongodb-linux-${marm}-ubuntu2204-${ver}/bin/mongod /usr/local/bin/; \
rm -rf /tmp/mongo.tgz /tmp/mongodb-linux-*; \
mongod --version | head -1
# Node via nvm: 18 (default — toolkit tooling: create-snapshot hooks, `node -e` reads of
# toolkit.json) + 14 (the Ember app; frontend/.nvmrc = v14.21.3). Symlink v18 to /usr/local/bin
# so the toolkit's own node always resolves; run-app switches PATH to v14 for the client.
# The frontend's .npmrc sets engine-strict=true and its package.json requires npm 6.x, so pin
# npm 6 in the v14 line (nvm's 14.21.3 otherwise bundles npm 7, which fails engine-strict). The
# v18.* glob (not `nvm version`) avoids sourcing nvm.sh under Docker's /bin/sh (dash), bash-only.
ENV NVM_DIR=/usr/local/nvm
RUN mkdir -p "$NVM_DIR" \
&& curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash \
&& bash -c '. "$NVM_DIR/nvm.sh" \
&& nvm install 18 \
&& nvm install 14.21.3 && nvm use 14.21.3 && npm install -g npm@6.14.18 \
&& nvm alias default 18' \
&& for b in node npm npx; do ln -sf "$NVM_DIR"/versions/node/v18.*/bin/"$b" /usr/local/bin/"$b"; done \
&& node --version
# phantomjs stub. The Ember client depends on phantomjs-prebuilt@2.1.16, which has NO arm64
# binary and is EOL everywhere — its install script aborts `npm install` on Apple-Silicon
# hosts. A stub on PATH that reports the expected version makes the install script treat
# PhantomJS as "already installed" and skip the (impossible) download, so `npm install`
# completes and `ember build`/`ember serve` (what run-app uses) work. `ember test` runs on
# headless Chrome at this pin, wired up after the Playwright block below.
RUN printf '#!/bin/bash\n[ "$1" = "--version" ] && { echo "2.1.1"; exit 0; }\nexit 0\n' > /usr/local/bin/phantomjs \
&& chmod +x /usr/local/bin/phantomjs
# Match backend/Gemfile.lock "BUNDLED WITH 2.5.6".
RUN gem install bundler -v 2.5.6
# Postgres trust auth: backend/config/database.yml connects as PG_DATABASE_USERNAME (default
# postgres). OVERWRITE pg_hba.conf (Debian's default `local all all peer` is first-match, so
# an appended trust rule never applies).
RUN PG_VERSION=$(ls /etc/postgresql) \
&& printf 'local all all trust\nhost all all 127.0.0.1/32 trust\nhost all all ::1/128 trust\nhost all all 0.0.0.0/0 trust\n' > "/etc/postgresql/${PG_VERSION}/main/pg_hba.conf" \
&& echo "listen_addresses='*'" >> "/etc/postgresql/${PG_VERSION}/main/postgresql.conf"
USER root
# --- Playwright + Chromium, for driving the app in a real browser -------------
# Self-contained under /opt — the member's own runtime is untouched.
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
xz-utils \
libxcomposite1 \
libxdamage1 \
libxfixes3 \
libxrandr2 \
libasound2 \
libatk1.0-0 \
libatk-bridge2.0-0 \
libatspi2.0-0 \
libcups2 \
libdbus-1-3 \
libgbm1 \
libnspr4 \
libnss3 \
libxkbcommon0 \
libpango-1.0-0 \
libcairo2 \
libxshmfence1 \
libx11-xcb1 \
libxcb-dri3-0 \
libdrm2 \
&& rm -rf /var/lib/apt/lists/*
RUN set -eux; \
arch="$(dpkg --print-architecture)"; \
case "$arch" in amd64) nodearch=x64;; arm64) nodearch=arm64;; *) echo "unsupported arch: $arch" >&2; exit 1;; esac; \
curl -fsSL "https://nodejs.org/dist/v20.19.5/node-v20.19.5-linux-${nodearch}.tar.xz" -o /tmp/pw-node.tar.xz; \
mkdir -p /opt/pw-node; \
tar -xJf /tmp/pw-node.tar.xz -C /opt/pw-node --strip-components=1; \
rm /tmp/pw-node.tar.xz; \
export npm_config_prefix=/opt/pw-node PATH="/opt/pw-node/bin:$PATH"; \
/opt/pw-node/bin/npm install -g playwright@1.56.0; \
test -d /opt/pw-node/lib/node_modules/playwright; \
/opt/pw-node/bin/node /opt/pw-node/lib/node_modules/playwright/cli.js install chromium
# `pw <script.js>` runs Node with `require("playwright")` resolvable (CommonJS).
RUN printf '#!/bin/sh\nNODE_PATH=/opt/pw-node/lib/node_modules exec /opt/pw-node/bin/node "$@"\n' > /usr/local/bin/pw \
&& chmod +x /usr/local/bin/pw
# Fail the build if Chromium cannot start.
RUN printf 'const{chromium}=require("playwright");(async()=>{const b=await chromium.launch();const p=await b.newPage();await p.setContent("<h1 id=t>ok</h1>");if(await p.textContent("#t")!=="ok")throw new Error("bad render");await b.close();console.log("chromium OK");})()\n' > /tmp/pw-check.js \
&& pw /tmp/pw-check.js \
&& rm -f /tmp/pw-check.js
# `ember test` resolves its browser via CHROME_BIN, falling back to `google-chrome` on PATH
# (frontend/testem.js). Point both at the Chromium Playwright just installed. The glob is
# resolved at build time so a Playwright bump can't strand a hardcoded chromium-<build> path.
RUN set -eux; \
chrome="$(echo /opt/ms-playwright/chromium-*/chrome-linux/chrome)"; \
test -x "$chrome"; \
printf '#!/bin/bash\nexec %s --no-sandbox --disable-dev-shm-usage "$@"\n' "$chrome" \
> /usr/local/bin/google-chrome; \
chmod +x /usr/local/bin/google-chrome; \
google-chrome --version
ENV CHROME_BIN=/usr/local/bin/google-chrome
ENV IS_SANDBOX=1
RUN mkdir -p /root/.claude && \
echo '{"permissions":{"deny":["WebFetch","WebSearch"]}}' > /root/.claude/settings.json
# Startup for a direct `docker run` (the devcontainer path uses post-start.sh instead, which
# starts the same services). Bring up Postgres + Redis + MongoDB, then hand off.
RUN cat > /usr/local/bin/start-services.sh <<'EOF'
#!/bin/bash
set -e
service postgresql start || true
service redis-server start >/dev/null 2>&1 || redis-server --daemonize yes >/dev/null 2>&1 || true
mkdir -p /data/db
mongod --dbpath /data/db --bind_ip 127.0.0.1 --fork --logpath /tmp/mongod.log >/dev/null 2>&1 || true
until pg_isready -h localhost -p 5432 -U postgres >/dev/null 2>&1; do sleep 0.5; done
exec "$@"
EOF
RUN chmod +x /usr/local/bin/start-services.sh
WORKDIR /workspace/repo
# Resolver for the DNS jail (.devcontainer/dns-jail-container.sh, applied by
# post-start.sh); if this does not land, Explore just runs unjailed.
RUN (command -v apk >/dev/null 2>&1 && apk add --no-cache dnsmasq bind-tools) \
|| (apt-get update && apt-get install -y --no-install-recommends dnsmasq-base dnsutils \
&& rm -rf /var/lib/apt/lists/*) \
|| true
ENTRYPOINT ["/usr/local/bin/start-services.sh"]
CMD ["sleep", "infinity"]

View File

@@ -1 +0,0 @@
/home/ericbell/workspaces/dataannotation/current-project/worker-toolkit-flaredown/repo

View File

@@ -1,11 +0,0 @@
{
"repo": "flaredown",
"defaultCommit": "b0605ff3",
"version": "7f40461c4d",
"explorePorts": {
"clientHost": 4000,
"serverHost": null,
"corpusHost": null,
"livereloadHost": 7020
}
}

View File

@@ -1,226 +0,0 @@
# Per-repo harbor task Dockerfile for flaredown (rubyforgood, GPL-3). Polyglot symptom tracker:
# a backend/ Rails 7.1 API (Ruby 3.2.3, Mongoid 8.1 on MongoDB + Postgres + Redis + Sidekiq)
# and an Ember frontend/ (Node 14). Mirrors the explore stack; bakes the workspace + Claude Code
# (grader), git-commits a baseline. The app lives in subdirs — gems install in /workspace/backend.
#
# MongoDB 7.0 (not compose's EOL, arm64-less 4.4.9): Mongoid 8.1.3 + driver 2.20.1 support up to
# 7.0, which has native amd64 + aarch64 builds. Same wire protocol; the app is version-agnostic.
FROM ruby:3.2.3
ARG TOOLKIT_BUILD_ID=dev
RUN apt-get update && apt-get install -y --no-install-recommends \
postgresql postgresql-client libpq-dev \
redis-server \
build-essential pkg-config libyaml-dev \
python3 \
git sudo curl ca-certificates gnupg xz-utils jq procps \
&& rm -rf /var/lib/apt/lists/*
# MongoDB 7.0 server binary (mongod), arch-aware ubuntu2204 build (runs on bookworm).
RUN set -eux; \
arch="$(dpkg --print-architecture)"; \
case "$arch" in amd64) marm=x86_64;; arm64) marm=aarch64;; *) echo "unsupported arch: $arch" >&2; exit 1;; esac; \
ver=7.0.14; \
curl -fsSL "https://fastdl.mongodb.org/linux/mongodb-linux-${marm}-ubuntu2204-${ver}.tgz" -o /tmp/mongo.tgz; \
tar -xzf /tmp/mongo.tgz -C /tmp; \
cp /tmp/mongodb-linux-${marm}-ubuntu2204-${ver}/bin/mongod /usr/local/bin/; \
rm -rf /tmp/mongo.tgz /tmp/mongodb-linux-*; \
mongod --version | head -1
# Node via nvm: 18 (default) + 14 (the Ember client; frontend/.nvmrc = v14.21.3). Pin npm 6
# in the v14 line — the frontend's .npmrc is engine-strict and requires npm 6.x (nvm's 14.21.3
# otherwise bundles npm 7, which fails engine-strict).
ENV NVM_DIR=/usr/local/nvm
RUN mkdir -p "$NVM_DIR" \
&& curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash \
&& bash -c '. "$NVM_DIR/nvm.sh" \
&& nvm install 18 \
&& nvm install 14.21.3 && nvm use 14.21.3 && npm install -g npm@6.14.18 \
&& nvm alias default 18' \
&& for b in node npm npx; do ln -sf "$NVM_DIR"/versions/node/v18.*/bin/"$b" /usr/local/bin/"$b"; done \
&& node --version
# phantomjs stub — the Ember client's phantomjs-prebuilt@2.1.16 (for `ember test`) has no arm64
# binary and is EOL; a version-reporting stub on PATH makes `npm install` skip the impossible
# download so the client's deps install and it can build/serve. `ember test` needs a real
# phantomjs (unavailable on arm64 upstream anyway); the rspec verifier doesn't touch the client.
RUN printf '#!/bin/bash\n[ "$1" = "--version" ] && { echo "2.1.1"; exit 0; }\nexit 0\n' > /usr/local/bin/phantomjs \
&& chmod +x /usr/local/bin/phantomjs
# Match backend/Gemfile.lock "BUNDLED WITH 2.5.6".
RUN gem install bundler -v 2.5.6
# Postgres trust auth (backend/config/database.yml connects as PG_DATABASE_USERNAME=postgres).
RUN PG_VERSION=$(ls /etc/postgresql) \
&& printf 'local all all trust\nhost all all 127.0.0.1/32 trust\nhost all all ::1/128 trust\nhost all all 0.0.0.0/0 trust\n' > "/etc/postgresql/${PG_VERSION}/main/pg_hba.conf" \
&& echo "listen_addresses='*'" >> "/etc/postgresql/${PG_VERSION}/main/postgresql.conf"
# Install Claude Code globally (grader runs `claude`); hard-gate on presence — a missing grader
# CLI silently zeros every reward, so a broken image must never be cached.
ARG CLAUDE_CODE_MIN=2.1.251
RUN for i in 1 2 3; do \
if curl -fsSL https://claude.ai/install.sh -o /tmp/claude-install.sh && bash /tmp/claude-install.sh; then break; fi; \
echo "WARNING: claude install attempt $i failed; retrying in 5s" >&2; sleep 5; \
done; \
rm -f /tmp/claude-install.sh; \
for p in /root/.claude-code/claude /root/.local/bin/claude "$(find /root -name claude -type f 2>/dev/null | head -1)"; do \
[ -n "$p" ] && [ -x "$p" ] && ln -sf "$p" /usr/local/bin/claude && break; \
done; \
command -v claude >/dev/null 2>&1 || { echo "FATAL: claude CLI not installed — the grader needs it" >&2; exit 1; }; \
_v="$(claude --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"; \
[ "$(printf '%s\n%s\n' "$CLAUDE_CODE_MIN" "$_v" | sort -V | head -1)" = "$CLAUDE_CODE_MIN" ] \
|| { echo "FATAL: claude $_v is older than $CLAUDE_CODE_MIN, the minimum the grader needs" >&2; exit 1; }; \
echo "claude $_v installed at $(command -v claude)"
USER root
# --- Playwright + Chromium, when the task opts in ----------------------------
# Installed only when task.toml sets `[metadata] browser = true`. A Dockerfile cannot read
# task.toml, so build-workspace.sh writes that answer to environment/browser-optin.
# Self-contained under /opt — the member's own runtime is untouched.
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright
COPY browser-optin /tmp/browser-optin
RUN set -eu; \
if [ "$(cat /tmp/browser-optin)" != "1" ]; then echo "browser: task did not opt in; skipping Playwright"; exit 0; fi; \
set -x; \
apt-get update -qq; \
apt-get install -y -qq --no-install-recommends \
xz-utils \
libxcomposite1 \
libxdamage1 \
libxfixes3 \
libxrandr2 \
libasound2 \
libatk1.0-0 \
libatk-bridge2.0-0 \
libatspi2.0-0 \
libcups2 \
libdbus-1-3 \
libgbm1 \
libnspr4 \
libnss3 \
libxkbcommon0 \
libpango-1.0-0 \
libcairo2 \
libxshmfence1 \
libx11-xcb1 \
libxcb-dri3-0 \
libdrm2; \
rm -rf /var/lib/apt/lists/*; \
arch="$(dpkg --print-architecture)"; \
case "$arch" in amd64) nodearch=x64;; arm64) nodearch=arm64;; *) echo "unsupported arch: $arch" >&2; exit 1;; esac; \
curl -fsSL "https://nodejs.org/dist/v20.19.5/node-v20.19.5-linux-${nodearch}.tar.xz" -o /tmp/pw-node.tar.xz; \
mkdir -p /opt/pw-node; \
tar -xJf /tmp/pw-node.tar.xz -C /opt/pw-node --strip-components=1; \
rm /tmp/pw-node.tar.xz; \
export npm_config_prefix=/opt/pw-node PATH="/opt/pw-node/bin:$PATH"; \
/opt/pw-node/bin/npm install -g playwright@1.56.0; \
test -d /opt/pw-node/lib/node_modules/playwright; \
/opt/pw-node/bin/node /opt/pw-node/lib/node_modules/playwright/cli.js install chromium; \
printf '#!/bin/sh\nNODE_PATH=/opt/pw-node/lib/node_modules exec /opt/pw-node/bin/node "$@"\n' > /usr/local/bin/pw; \
chmod +x /usr/local/bin/pw; \
printf 'const{chromium}=require("playwright");(async()=>{const b=await chromium.launch();const p=await b.newPage();await p.setContent("<h1 id=t>ok</h1>");if(await p.textContent("#t")!=="ok")throw new Error("bad render");await b.close();console.log("chromium OK");})()\n' > /tmp/pw-check.js; \
pw /tmp/pw-check.js; \
rm -f /tmp/pw-check.js
WORKDIR /workspace
COPY workspace/ .
RUN mkdir -p .claude && \
echo '{"permissions":{"deny":["WebFetch","WebSearch"]}}' > .claude/settings.json
# .env is gitignored; materialize from the committed backend/env-example (public dev secrets).
# env-example points PG at host `postgresql` (the compose service name) — rewrite to localhost
# (everything is on localhost in this single container). Redis is already localhost; Mongoid
# reads MONGODB_HOST (unset → localhost).
RUN if [ -f backend/env-example ] && [ ! -f backend/.env ]; then \
cp backend/env-example backend/.env && \
sed -i 's/^PG_DATABASE_HOST=.*/PG_DATABASE_HOST=localhost/' backend/.env; \
fi
RUN git init -q && \
git config user.email "dev@agent" && \
git config user.name "Dev" && \
git add -A && \
git commit -m "initial" --quiet
# Install backend gems (in backend/). Add linux platforms (host is typically darwin-arm64).
RUN cd backend \
&& bundle config set --local frozen false \
&& bundle lock --add-platform x86_64-linux \
&& bundle lock --add-platform aarch64-linux \
&& bundle install --jobs 4 --retry 3
# Install the Ember client deps (baked; non-fatal — the rspec verifier doesn't need them, and
# the Node-14/bower toolchain is fragile in a non-interactive build). OPENSSL_CONF=/dev/null
# for the old webpack md4 hashing on bookworm's OpenSSL 3.
# --unsafe-perm so npm (as root) runs the postinstall (patch-package + bower install) instead of
# skipping it; without it bower_components never populates and the client can't build.
RUN . "$NVM_DIR/nvm.sh" && nvm use 14.21.3 >/dev/null \
&& cd frontend && OPENSSL_CONF=/dev/null npm install --unsafe-perm --no-audit --no-fund \
|| echo "WARNING: frontend npm install failed (non-fatal — JS client isn't needed for grading)" >&2
# Fail loudly if any load-bearing tool is missing.
RUN for t in ruby bundle psql redis-server mongod node claude python3; do \
command -v "$t" >/dev/null 2>&1 || { echo "FATAL: required tool '$t' missing from image" >&2; exit 1; }; \
done; \
echo "toolchain OK: ruby=$(ruby --version) node=$(node --version) mongod=$(mongod --version | head -1)"
# Fold setup edits (.env, Gemfile.lock platform locks) into the baseline so the grader's
# working-tree diff attributes only the agent's changes.
RUN git add -A && git commit --amend --no-edit --quiet
# Startup: start Postgres + Redis + MongoDB, create the PG dev/test DBs, load the PG schema.
# Mongo collections are created lazily by Mongoid — nothing to load there.
RUN cat > /usr/local/bin/start-services.sh <<'EOF'
#!/bin/bash
set -e
service postgresql start
service redis-server start >/dev/null 2>&1 || redis-server --daemonize yes >/dev/null 2>&1 || true
mkdir -p /data/db && mongod --dbpath /data/db --bind_ip 127.0.0.1 --fork --logpath /tmp/mongod.log >/dev/null 2>&1 || true
until pg_isready -h localhost -p 5432 -U postgres >/dev/null 2>&1; do sleep 0.5; done
su postgres -c "psql -c \"CREATE DATABASE flaredown_development OWNER postgres;\"" >/dev/null 2>&1 || true
su postgres -c "psql -c \"CREATE DATABASE flaredown_test OWNER postgres;\"" >/dev/null 2>&1 || true
cd /workspace/backend && bundle exec rails db:schema:load >/tmp/schema-load-dev.log 2>&1 || echo "WARN: dev schema load failed - see /tmp/schema-load-dev.log" >&2
cd /workspace/backend && RAILS_ENV=test bundle exec rails db:schema:load >/tmp/schema-load-test.log 2>&1 || echo "WARN: test schema load failed - see /tmp/schema-load-test.log" >&2
exec "$@"
EOF
RUN chmod +x /usr/local/bin/start-services.sh
# Install the Codex CLI at BUILD time, for the same reason claude is: the agent-setup
# install needs the network, which the trial DNS jail blocks. Hard-fail rather than let a
# codex-less image cache and break every trial on that repo at agent-setup.
RUN for i in 1 2 3; do \
if curl -fsSL https://chatgpt.com/codex/install.sh -o /tmp/codex-install.sh \
&& CODEX_INSTALL_DIR=/usr/local/bin CODEX_NON_INTERACTIVE=true sh /tmp/codex-install.sh; then break; fi; \
echo "WARNING: codex install attempt $i failed; retrying in 5s" >&2; sleep 5; \
done; \
rm -f /tmp/codex-install.sh; \
if ! command -v codex >/dev/null 2>&1 && [ -x "$HOME/.local/bin/codex" ]; then \
ln -sf "$HOME/.local/bin/codex" /usr/local/bin/codex; \
fi; \
if ! command -v codex >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then \
npm install -g @openai/codex@latest || true; \
fi; \
command -v codex >/dev/null 2>&1 \
&& echo "codex installed at $(command -v codex)" \
|| echo "WARNING: codex CLI not installed (see the install output above)" >&2
# Restrict DNS to the model endpoint when DNSJAIL_ALLOW is set (the agent supplies it).
# Source: scripts/lib/dns-jail-container.sh, staged here by build-workspace.sh.
COPY dns-jail/ /opt/raccoon-dns-jail/
RUN if [ -f /opt/raccoon-dns-jail/dns-jail-container.sh ]; then \
install -m 0755 /opt/raccoon-dns-jail/dns-jail-container.sh /usr/local/bin/raccoon-dns-jail \
&& sh -n /usr/local/bin/raccoon-dns-jail; \
else echo "NOTE: no DNS jail script staged; trials on this image run unjailed" >&2; fi
ENTRYPOINT ["/usr/local/bin/start-services.sh"]
# Resolver for the trial DNS allowlist (scripts/lib/dns-jail.sh); if this
# does not land, trials just run unjailed.
RUN (command -v apk >/dev/null 2>&1 && apk add --no-cache dnsmasq bind-tools) \
|| (apt-get update && apt-get install -y --no-install-recommends dnsmasq-base dnsutils \
&& rm -rf /var/lib/apt/lists/*) \
|| true
CMD ["sleep", "infinity"]

View File

@@ -1,8 +0,0 @@
# Seeded from the shared checks config for repo `flaredown` — task-specific checks are
# expected here and are kept; the local build step won't touch this file.
# Sourced by tests/test.sh: each line is one deterministic-signal check.
# run_signal <label> <command> [baseline_known_failures]
# raccoon-sync-hash: a9dad0e681f41c05dc5690dffe7822d9dce4184842fb9f3b25570d42418eb3e3
# run_setup <command> — one-shot build/codegen before the checks (not scored, not counted)
run_setup 'until pg_isready -h localhost -p 5432 -U postgres >/dev/null 2>&1; do sleep 0.5; done; cd backend && RAILS_ENV=test bundle exec rails db:schema:load'
run_signal 'rspec' 'cd backend && RAILS_ENV=test bundle exec rspec --exclude-pattern '\''spec/system/**/*'\''' ''

View File

@@ -1,16 +0,0 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "bundler"
directory: "/backend"
schedule:
interval: "weekly"
- package-ecosystem: "npm"
directory: "/frontend"
schedule:
interval: "weekly"

View File

@@ -1,154 +0,0 @@
name: backend
on:
push:
branches:
- main
- master
pull_request:
branches:
- main
- master
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
backend:
- 'backend/**'
- '.github/workflows/**'
frontend:
- 'frontend/**'
- '.github/workflows/**'
standardrb:
needs: changes
if: ${{ needs.changes.outputs.backend == 'true' }}
runs-on: ubuntu-latest
defaults:
run:
working-directory: backend
steps:
- uses: actions/checkout@v4
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
working-directory: backend
bundler-cache: true
- name: Build & Run
run: |
bundle exec standardrb
erb-lint:
needs: changes
if: ${{ needs.changes.outputs.backend == 'true' }}
runs-on: ubuntu-latest
defaults:
run:
working-directory: backend
steps:
- uses: actions/checkout@v4
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
bundler-cache: true
- name: ERB lint
run: |
gem install erb_lint
erblint --lint-all --autocorrect
rspec:
needs: changes
if: ${{ needs.changes.outputs.backend == 'true' }}
runs-on: ubuntu-latest
defaults:
run:
working-directory: backend
env:
MONGODB_HOST: localhost
MONGODB_PORT: 27017
POSTGRES_HOST: localhost
DATABASE_HOST: localhost
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
POSTGRES_HOST_AUTH_METHOD: trust
POSTGRES_PORT: 5432
INTERCOM_SECRET: secret
BASE_URL: test.com
services:
redis:
image: redis:6.2.3-alpine
ports: ["6379:6379"]
options: --entrypoint redis-server
db:
image: postgres:12.8-alpine
env:
POSTGRES_PASSWORD: password
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Install PostgreSQL client
run: |
sudo apt-get -yqq install libpq-dev
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
working-directory: backend
bundler-cache: true
- name: Start MongoDB
uses: supercharge/mongodb-github-action@1.10.0
with:
mongodb-version: 4.4.9
- name: Load database schema
run: |
bundle exec rake db:create
bundle exec rake db:schema:load
- name: Run rspec
run: |
bundle exec rspec
brakeman:
name: Security Analysis
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
working-directory: backend
bundler-cache: true
- name: Brakeman
uses: reviewdog/action-brakeman@v2
with:
brakeman_version: gemfile
reporter: github-pr-review

View File

@@ -1,75 +0,0 @@
name: frontend
on:
push:
branches:
- main
- master
pull_request:
branches:
- main
- master
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
backend:
- 'backend/**'
- '.github/workflows/**'
frontend:
- 'frontend/**'
- '.github/workflows/**'
test-app:
name: Test app
needs: changes
if: ${{ needs.changes.outputs.frontend == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 14
cache: npm
cache-dependency-path: frontend/package-lock.json
- uses: browser-actions/setup-chrome@v2
id: setup-chrome
- run: npm install -g npm@6.14.18
- run: npm install
working-directory: ./frontend
- run: npm run test
working-directory: ./frontend
env:
CHROME_BIN: ${{ steps.setup-chrome.outputs.chrome-path }}
node-next-test:
strategy:
matrix:
node_version: ['16', '18', '20']
needs: changes
if: ${{ needs.changes.outputs.frontend == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node_version }}
- uses: browser-actions/setup-chrome@v2
id: setup-chrome
- run: npm install
working-directory: ./frontend
- run: npm run test
working-directory: ./frontend
env:
CHROME_BIN: ${{ steps.setup-chrome.outputs.chrome-path }}

View File

@@ -1,86 +0,0 @@
name: native
on:
push:
branches:
- main
- master
pull_request:
branches:
- main
- master
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
native: ${{ steps.filter.outputs.native }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
backend:
- 'backend/**'
- '.github/workflows/**'
native:
- 'native/**'
- '.github/workflows/**'
test-app:
name: Test app
needs: changes
if: ${{ needs.changes.outputs.native == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 18
cache: npm
cache-dependency-path: native/package-lock.json
- run: npm ci
working-directory: ./native
- run: npm run test
working-directory: ./native
lint:
name: Lint
needs: changes
if: ${{ needs.changes.outputs.native == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 18
cache: npm
cache-dependency-path: native/package-lock.json
- run: npm ci
working-directory: ./native
- run: npm run lint
working-directory: ./native
type-check:
name: Type check
needs: changes
if: ${{ needs.changes.outputs.native == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 18
cache: npm
cache-dependency-path: native/package-lock.json
- run: npm ci
working-directory: ./native
- run: npm run tsc
working-directory: ./native

View File

@@ -1,17 +0,0 @@
npm-debug.log
backend/dump.rdb
backend/dump
dump.rdb
.rbenv-gemsets
.idea/*
.bundle
frontend/.env
.DS_Store
TODO.md
docs/superpowers/

View File

@@ -1,2 +0,0 @@
flaredown

View File

@@ -1 +0,0 @@
3.2.3

View File

@@ -1,5 +0,0 @@
nodejs 12.22.6
ruby 3.2.3
postgres 12.8
mongodb 4.4.9
redis 6.2.3

View File

@@ -1,2 +0,0 @@
{
}

View File

@@ -1,70 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Flaredown is a chronic-illness symptom tracker. It is a monorepo with three deployable apps:
- `backend/` — Rails 7.1 API (Ruby 3.2.3), the only backend for all clients.
- `frontend/` — Ember.js 2.18 web app (the production web client at app.flaredown.com), proxies API calls to the backend.
- `native/` — Expo / React Native + TypeScript app (newer, in-progress replacement for the Ember client).
The root `app/` directory is a stray remnant (single `g-recaptcha.js`), not a fourth app.
## Commands
Everything is Dockerized; `make` wraps `docker compose`. Prefer these over running services natively.
- `make start` / `make stop` — run the full dev stack (backend + workers + Ember frontend) via the `dev` profile.
- `make startNative` / `make stopNative` — run backend + React Native (`native` profile).
- `make build` — rebuild the backend image. Do this before running specs if backend code/deps changed.
- `make seed` — seed databases (`rails app:setup`).
- `make console` — Rails console.
- Web app: http://localhost:4300 (Ember). Native: http://localhost:19006. Backend API: http://localhost:3000.
### Tests
- All backend specs: `make specs` (equivalently `script/backend rspec spec spec`).
- A single spec: `script/backend rspec spec/services/weather_retriever_spec.rb`. The `script/backend` wrapper runs any command inside the backend container (`docker compose --profile dev run --rm backend $@`).
- Add `debugger` to Ruby code to break into an interactive shell under rspec.
- Frontend (Ember): `cd frontend && npm test` (`ember test`).
- Native: `cd native && npm test` (jest), `npm run tsc` (typecheck).
### Lint (all enforced in CI; run before pushing)
- Ruby: `script/backend standardrb` (StandardRB, not RuboCop).
- ERB: `script/backend erb_lint --lint-all`.
- Native: `cd native && npm run lint` (eslint + prettier), `npm run lint:fix` to autofix.
CI (`.github/workflows/{backend,frontend,native}.yml`) uses path filters — backend jobs only run when `backend/**` changes, etc. StandardRB, ERB lint, rspec, and frontend build are required for merge.
## Architecture
### Dual database — the most important thing to understand
The backend uses **both PostgreSQL and MongoDB simultaneously**, split by data type:
- **PostgreSQL (ActiveRecord)** — relational/reference data: `User` (Devise auth), `Condition`, `Symptom`, `Treatment`, `Food`, `Tag`, `Profile`, `Weather`, and the `user_*` join tables. These models subclass `ActiveRecord::Base` and carry a `# == Schema Information` header. Schema lives in `db/schema.rb` + `db/structure.sql`; migrations in `db/migrate/`.
- **MongoDB (Mongoid 8)** — high-volume, user-generated, schemaless data: `Checkin` (the core daily symptom/treatment/tag log), `Comment`, `Reaction`, `Pattern`, `Notification`, `HarveyBradshawIndex`, `Feedback`, `PromotionRate`, `OracleRequest`. These `include Mongoid::Document`. Config in `config/mongoid.yml`.
The two stores are linked by an **encrypted foreign key**: Mongo documents store `encrypted_user_id` (symmetric-encryption gem, see `config/symmetric-encryption.yml`) rather than a plain `user_id`, and dereference it back to the Postgres `User`. When querying check-in data by user, filter on `encrypted_user_id`, not `user_id`. `Checkin` embeds condition/symptom/treatment sub-documents inline.
### API layer
Versioned JSON API under `app/controllers/api/v1/`, routed via `namespace :api { scope module: :v1 }` in `config/routes.rb`. Serialization uses `active_model_serializers` 0.9 (`app/serializers/`). Auth is Devise + `devise_invitable` + Facebook OmniAuth; authorization is CanCanCan with a Mongoid adapter (`app/models/ability.rb`). Business logic lives in `app/services/` (e.g. `weather_retriever`, `pattern_creator`, `chart_list_service`) — controllers should stay thin.
### Background work
Sidekiq (`config/sidekiq.yml`, `worker` process in `Procfile`) backed by Redis, with jobs in `app/jobs/` (check-in reminders, data exports, notification dispatch, top-posts mailers). Recurring schedules are defined in `config/cronotab.rb` (Crono) and rake tasks under `lib/tasks/` invoked by Heroku Scheduler.
### External integrations
Tomorrow.io (weather, via `tomorrowio_rb`), Pusher (realtime), Geocoder + `nearest_time_zone` (location → timezone for reminders), AWS SES (inbound/bounce handling in `aws_ses_controller`).
## Deployment
Heroku, via `rake` tasks in the root `Rakefile`. Frontend and backend are separate Heroku apps deployed with `git subtree split` (`rake production:deploy` / `rake staging:deploy`). Commits to `master` auto-deploy to staging. Postgres/Redis are Heroku addons; MongoDB is hosted at mongodb.com.
## Gotchas
- Node is pinned to **12.22.6** for the Ember frontend (`.tool-versions`); the native app uses a modern toolchain independently. Don't assume one Node version across the repo.
- Env files: `cp backend/env-example backend/.env` and `cp backend/env-example frontend/.env`. A `FACEBOOK_APP_ID` is needed in `frontend/.env` or the app renders a blank beige screen on first load (see README "Common Problems" for the workaround).

View File

@@ -1,33 +0,0 @@
## Contributing
We ♥ contributors! By participating in this project, you agree to abide by the Ruby for Good [code of conduct].
**First:** if you're unsure or afraid of *anything*, just ask or submit the issue or pull request anyways. You won't be yelled at for giving your best effort. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contributions, and don't want a wall of rules to get in the way of that.
[code of conduct]: https://github.com/rubyforgood/code-of-conduct
Here are the basic steps to submit a pull request. Make sure that you're working on an [open issue]–if the relevant issue doesn't exist, open it!
[open issue]: https://github.com/rubyforgood/r4g-github-provisioning/issues
1. Claim an issue on [our issue tracker][open issue] by assigning it to yourself (core team member) or commenting. If the issue doesn't exist yet, open it.
2. Fork the repo.
3. Run the tests. We only take pull requests with passing tests, and it's great to know that you have a clean slate: `bundle exec rake`
4. Add a test for your change. If you are adding functionality or fixing a bug, you should add a test!
5. Make the test pass.
6. Push to your fork and submit a pull request. Include the issue number (ex. `Resolves #1`) in the PR description.
7. For any changes, please create a feature branch and open a PR for it when you feel it's ready to merge. Even if there's no real disagreement about a PR, at least one other person on the team needs to look over a PR before merging. The purpose of this review requirement is to ensure shared knowledge of the app and its changes and to take advantage of the benefits of working together without anyone being a bottleneck.
At this point you're waiting on us–we'll try to respond to your PR quickly. We may suggest some changes or improvements or alternatives.
Some things that will increase the chance that your pull request is accepted:
* Use Rails idioms and helpers
* Include tests that fail without your code, and pass with it
* Update the documentation, the surrounding one, examples elsewhere, guides, whatever is affected by your contribution

View File

@@ -1,674 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{one line to give the program's name and a brief idea of what it does.}
Copyright (C) {year} {name of author}
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
{project} Copyright (C) {year} {fullname}
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

View File

@@ -1,26 +0,0 @@
start: ## Start the project
docker compose --profile dev up
stop: ## Stop the project
docker compose --profile dev down
startNative: ## Start the react native project
docker compose --profile native up
stopNative: ## Stop the react native project
docker compose --profile native down
build: ## Build the project
docker compose build backend
specs: ## Run the specs
docker compose --profile dev run --rm backend rspec spec spec
console: ## Open a rails console
docker compose --profile dev run --rm backend rails c
seed: ## Reset, migrate, load fixtures, and seed your database
docker compose --profile tools run --rm app-setup
help:
@sed -n -E "s/(^[^ ]+):.* ## (.*)/`printf "\033[32m"`\1|`printf "\033[0m"` \2/p" $(MAKEFILE_LIST) | sort | column -t -s '|'

View File

@@ -1,157 +0,0 @@
# Flaredown
[![rspec](https://github.com/rubyforgood/Flaredown/actions/workflows/rspec.yml/badge.svg)](https://github.com/rubyforgood/Flaredown/actions/workflows/rspec.yml)
[![frontend](https://github.com/rubyforgood/Flaredown/actions/workflows/frontend.yml/badge.svg)](https://github.com/rubyforgood/Flaredown/actions/workflows/frontend.yml)
[![ERB lint](https://github.com/rubyforgood/Flaredown/actions/workflows/erb_lint.yml/badge.svg)](https://github.com/rubyforgood/Flaredown/actions/workflows/erb_lint.yml)
[![standardrb lint](https://github.com/rubyforgood/Flaredown/actions/workflows/ruby_lint.yml/badge.svg)](https://github.com/rubyforgood/Flaredown/actions/workflows/ruby_lint.yml)
Flaredown makes it easy for people to track symptoms over time, and learn how to control them. Our goal is to analyze the aggregate data from users of this tool to understand the probable effects of treatments and environmental stressors on chronic illness.
Help would be appreciated! Please join us in [slack #flaredown](https://join.slack.com/t/rubyforgood/shared_invite/zt-3ej5oyume-_rhWjVi3bYi83RyS3nuxTg), raise a GitHub issue, or email <contact@flaredown>.
## Environment
* PostgreSQL 12.8
* MongoDB 4.4.9
* Redis 6.2.3
* Ruby 3.2.3
* Node 12.22.6
## Installation
You can run the application and its dependencies using `docker compose`, or run the app natively using the setup instructions below.
Alternatively, you can run the app using the `make` commands available: `make help`
If you want to run the application on your own machine see the next sections on dependency installations.
### Running with Docker
Populate the necessary environment parameters:
```bash
cp backend/env-example backend/.env
cp frontend/env-example frontend/.env
```
In `frontend/.env`, `PORT` is the backend API port used by the Ember app and `FRONTEND_PORT` is the local frontend port.
Set `FACEBOOK_APP_ID` in `frontend/.env` if you want to use Facebook login locally.
Set up the database:
```bash
docker compose --profile tools run --rm app-setup
```
This command is interactive and resets the local Docker development and test databases. Type `yes` when prompted to continue.
Start the application:
```bash
docker compose --profile dev up
```
Visit your app at [http://localhost:4300](http://localhost:4300).
Frontend dependency changes are handled automatically by Docker. For a full reset of all local Docker data, including databases and dependency volumes, run `docker compose down -v`, then run the database setup command again afterward.
### Running natively
#### Mac Prerequisites
_If you are running on an M1 mac, run the following command before you start the installation process:_
```bash
$env /usr/bin/arch -arm64 /bin/zsh ---login
```
_Remove all gems before you proceed_
```bash
gem uninstall -aIx
```
#### Backend
You can install the dependencies via [asdf-vm](https://asdf-vm.com/) declared in the `.tool-versions` file, or:
- [Ruby Version Manager](https://rvm.io/)
- [MongoDB installation on OSX](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-os-x/)
On macOS, you can install `libpq` by running `brew install libpq && brew link --force libpq && bundle config --local build.pg "--with-ldflags=-L$(brew --prefix libpq)/lib --with-pg-include=$(brew --prefix libpq)/include"`, which is required for `bundle install` to succeed.
```bash
cd backend
echo "gem: --no-ri --no-rdoc" > ~/.gemrc
bundle config set --local without 'production'
bundle config set --local jobs 5
bundle config set --local retry 10
bundle install
cp env-example .env # You may adjust it however you like
# RVM is going to autoload this on every 'cd' to the directory
bundle exec rake app:setup
gem install foreman
```
#### Frontend
```bash
cd frontend
npm install
```
#### React Native
```bash
cd native
npm install
```
## Development
### Prerequisites
- Populate the necessary environment parameters with `cp backend/env-example backend/.env && cp frontend/env-example frontend/.env`
- Create a [Facebook dev app](https://developers.facebook.com/docs/development/create-an-app) and paste your own ID into `frontend/.env` file's `FACEBOOK_APP_ID` parameter.
- Note: This is not necessary in `backend/.env` but we have not yet cleaned up these two files into the necessary components.
- Reset, migrate, load fixtures, and seed your database using `make seed` or `bundle exec rails app:setup`
### Running
If you are running the application natively, run the following to start your server. If you're using docker, this should be up and running already.
```bash
rake run
```
Visit your app at [http://localhost:4300](http://localhost:4300) for the current ember application, or [http://localhost:19006](http://localhost:19006) for the React Native version.
## Running tests locally
1. Run `make build` or `docker compose build backend` to ensure the latest backend is built and being run
2. To run all tests run `make specs` or `script/backend rspec spec spec`, or you can run a specific test suite such as `script/backend rspec spec spec/services/weather_retriever_spec.rb `
3. Debugging tip: in Ruby code you can add a line that says `debugger` and rspec will automatically break on that line and give you an interactive Ruby shell
## CI
Several checks are configured to run on all commits using GitHub Actions, including lint, build and test steps. Definitions can be found in [./.github/workflows](./.github/workflows). Those checks which always run are required to be successful for pull requests to be merged.
## Deployment
Deployments target [Heroku](https://heroku.com). The traditional deployment is manually configured and is composed of two distinct applications (frontend and api) in two environments (staging and production), with automatic deployments to staging of commits to master:
* [flaredown-staging-api](https://dashboard.heroku.com/apps/flaredown-staging-api)
* [flaredown-staging-webapp](https://dashboard.heroku.com/apps/flaredown-staging-webapp) (https://app.flaredown.com)
* [flaredown-api](https://dashboard.heroku.com/apps/flaredown-api)
* [flaredown-webapp](https://dashboard.heroku.com/apps/flaredown-webapp) (https://staging.flaredown.com) (Temporarily https://flaredown-staging-webapp.herokuapp.com/login due to https://github.com/rubyforgood/Flaredown/issues/506)
Addons are used for Heroku Postgres, Heroku Redis, Heroku Scheduler + Papertrail. MongoDB is provided by mongodb.com.
## Style Guide
### 🎨 [Figma Assets](https://www.figma.com/proto/MBVn73pD6JbBkxd65KSZHr/Flaredown-Guide?page-id=0%3A1&node-id=1%3A3&viewport=241%2C48%2C0.45&scaling=contain&starting-point-node-id=1%3A3)
## Common Problems
* On first load, the app displays a blank beige screen instead of the login screen. Temporary fix is to add `console.log(process.env.FACEBOOK_APP_ID)` right inside of the module.exports at the top of the `frontend/config/environment.js` file. You can then refresh the page (no need to kill Docker) and this should fix it. You can now remove the log.
## License
Copyright 2015-2024 Logan Merriam and contributors.
Flaredown is open source software made available under the GPLv3 License. For details see the LICENSE file.

View File

@@ -1,80 +0,0 @@
require "rake"
desc "run application"
task :run do
pids = [
spawn("cd backend && bundle install && foreman start -f Procfile.local"),
spawn("cd frontend && rm -rfd ./dist && ./node_modules/.bin/ember serve --port 4300")
]
trap "INT" do
Process.kill "INT", *pids
exit 1
end
pids.each do |pid|
Process.wait pid
end
end
{production: "flaredown", staging: "flaredown-staging"}.each do |env, application|
namespace env.to_sym do
desc "restart application"
task :restart do
log "Restart #{application}"
restart "#{application}-api"
end
desc "deploy application"
task :deploy do
Rake::Task["#{env}:deploy:backend"].invoke
Rake::Task["#{env}:deploy:frontend"].invoke
end
namespace :deploy do
desc "deploy frontend application"
task :frontend do
log "Deploy frontend #{application} with revision: #{revision}"
deploy_to "git@heroku.com:#{application}-webapp.git", "frontend"
end
desc "deploy backend application"
task :backend do
log "Deploy backend #{application} with revision: #{revision}"
deploy_to "git@heroku.com:#{application}-api.git", "backend"
migrate "#{application}-api"
end
end
desc "setup application"
task :setup do
system("heroku pg:reset DATABASE --app #{application}-api --confirm #{application}-api")
system("heroku run rake app:setup --app #{application}-api")
end
desc "invite user to join into application"
task :invite do
system("heroku run rake app:invite --app #{application}-api")
end
end
end
def deploy_to(remote, subtree)
system("git push #{remote} `git subtree split --prefix #{subtree} #{revision}`:master --force")
end
def migrate(application)
system("heroku run rake db:migrate --app #{application}")
end
def restart(application)
system("heroku restart --app #{application}")
end
def revision
ENV.fetch("REVISION") { "master" }
end
def log(message)
puts ">>> #{message}"
end

View File

@@ -1,9 +0,0 @@
# Security Policy
## Supported Versions
The current deployed version is eligible for security reports
## Reporting a Vulnerability
Please report vulterabilities to flaredown at rubyforgood dot org and they will be triaged as soon as we can and give you public credit for useful reports.

View File

@@ -1,78 +0,0 @@
/**
* This file has been copied from ember-g-recaptcha and altered to fix a bug as
* described in https://github.com/algonauti/ember-g-recaptcha/issues/12
*
* Once we upgrade this app to Ember 3+ we can remove this file and update the
* dependency on ember-g-recaptcha to at least 0.9.0 which fixes this race
* condition.
*/
import Ember from 'ember';
import Configuration from '../configuration';
export default Ember.Component.extend({
classNames: ['g-recaptcha'],
sitekey: Configuration.siteKey,
tabindex: Ember.computed.alias('tabIndex'),
renderReCaptcha() {
// this is the line that was causing a race condition
if (Ember.isNone(window.grecaptcha) || Ember.isNone(window.grecaptcha.render)) {
Ember.run.later(() => {
this.renderReCaptcha();
}, 500);
} else {
let container = this.$()[0];
let properties = this.getProperties(
'sitekey',
'theme',
'type',
'size',
'tabindex'
);
let parameters = Ember.merge(properties, {
callback: this.get('successCallback').bind(this),
'expired-callback': this.get('expiredCallback').bind(this)
});
let widgetId = window.grecaptcha.render(container, parameters);
this.set('widgetId', widgetId);
this.set('ref', this);
}
},
resetReCaptcha() {
if (Ember.isPresent(this.get('widgetId'))) {
window.grecaptcha.reset(this.get('widgetId'));
}
},
successCallback(reCaptchaResponse) {
let action = this.get('onSuccess');
if (Ember.isPresent(action)) {
action(reCaptchaResponse);
}
},
expiredCallback() {
let action = this.get('onExpired');
if (Ember.isPresent(action)) {
action();
} else {
this.resetReCaptcha();
}
},
// Lifecycle Hooks
didInsertElement() {
this._super(...arguments);
Ember.run.next(() => {
this.renderReCaptcha();
});
}
});

View File

@@ -1,30 +0,0 @@
# See https://help.github.com/articles/ignoring-files for more about ignoring files.
#
# If you find yourself ignoring temporary files generated by your text editor
# or operating system, you probably want to add a global ignore instead:
# git config --global core.excludesfile '~/.gitignore_global'
# Ignore bundler config.
/.bundle
# Ignore the default SQLite database.
/db/*.sqlite3
/db/*.sqlite3-journal
# Ignore all logfiles and tempfiles.
/log/*
!/log/.keep
/tmp
# Ignore env
/.env*
# Ignore idea's files
/.idea
/coverage
/public/uploads/tmp
# Ignore Claude Code files
.claude/

View File

@@ -1,7 +0,0 @@
Pry.config.pager = false
Pry.config.color = true
if defined?(Rails)
Pry.config.prompt_name = "#{Rails.application.class.module_parent_name.downcase.green}/#{Rails.env.red}"
end

View File

@@ -1,2 +0,0 @@
--color
--tag ~type:system

View File

@@ -1 +0,0 @@
../.ruby-version

View File

@@ -1,53 +0,0 @@
# Auto generated files with errors to ignore.
# Remove from this list as you refactor files.
---
ignore:
- app/controllers/api/v1/aws_ses_controller.rb:
- Security/Open
- app/controllers/api/v1/profiles_controller.rb:
- Style/SafeNavigation
- app/controllers/api/v1/sessions_controller.rb:
- Style/SafeNavigation
- app/jobs/group_top_posts_job.rb:
- Style/SafeNavigation
- app/jobs/merge_trackables/checkin_trackables.rb:
- Performance/StringIdentifierArgument
- Lint/SymbolConversion
- app/jobs/merge_trackables/dispatcher.rb:
- Lint/SymbolConversion
- app/jobs/merge_trackables/user_trackable_association.rb:
- Lint/SymbolConversion
- app/models/ability.rb:
- Lint/SymbolConversion
- app/models/concerns/topicable.rb:
- Performance/StringIdentifierArgument
- app/models/profile.rb:
- Performance/StringIdentifierArgument
- app/models/registration.rb:
- Layout/MultilineMethodCallIndentation
- app/services/charts_pattern.rb:
- Lint/DuplicateMethods
- Performance/StringIdentifierArgument
- app/services/checkin/updater.rb:
- Lint/SymbolConversion
- Style/RedundantParentheses
- app/services/trackable_creator.rb:
- Lint/SymbolConversion
- lib/tasks/app.rake:
- Lint/ConstantDefinitionInBlock
- Style/GlobalStdStream
- Lint/Loop
- lib/tasks/hbi_completeness.rake:
- Lint/ConstantDefinitionInBlock
- lib/tasks/oneoff.rake:
- Performance/StringIdentifierArgument
- Layout/MultilineMethodCallIndentation
- lib/tasks/trackables.rake:
- Lint/ConstantDefinitionInBlock
- Lint/UselessAssignment
- lib/tasks/usda.rake:
- Lint/ConstantDefinitionInBlock
- lib/tasks/utils.rake:
- Performance/StringIdentifierArgument
- spec/models/food_spec.rb:
- Lint/ConstantDefinitionInBlock

View File

@@ -1 +0,0 @@
../.tool-versions

View File

@@ -1,23 +0,0 @@
FROM ruby:3.2.3
# set working directory
WORKDIR /app
# install dependencies
RUN apt-get update -qq && \
apt-get install -y nodejs postgresql-client
# install bundler
RUN gem install bundler:2.5.6
# copy the Gemfile and Gemfile.lock to the container
COPY Gemfile Gemfile.lock ./
# install the gems
RUN bundle install --full-index
# copy the rest of the application files to the container
COPY . .
# start the server
CMD ["bundle", "exec", "puma", "-C", "config/puma.rb"]

View File

@@ -1,109 +0,0 @@
source "https://rubygems.org"
ruby "3.2.3"
# Configuration management. keep on top of Gemfile
gem "dotenv-rails", groups: %i[development test]
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem "rails", "~> 7.1.0"
gem "rake"
gem "sprockets-rails"
# JSON serializer
gem "active_model_serializers", "~> 0.9"
# Use postgresql and mongo as the database for Active Record
gem "mongoid", "8.1.3" # https://www.mongodb.com/docs/mongoid/current/reference/compatibility/#rails-compatibility
gem "pg"
# Use Puma as the app server
gem "puma", "5.6.8"
# Authentication libraries
gem "cancancan", "~> 3.6.1"
gem "cancancan-mongoid", "~> 2.0"
gem "devise", "~> 4.8"
gem "devise_invitable", "~> 2.0"
gem "omniauth", "~> 1.8"
gem "omniauth-facebook", "~> 3.0"
# Colored output to console
gem "colored"
# Background jobs
gem "sidekiq", "~> 7.3"
# Structured seed data
gem "seedbank"
# ISO 3166 standard countries
gem "countries", require: "countries/global"
# Pusher Client
gem "pusher"
# ActiveRecord data translations
gem "globalize"
# Abort requests that are taking too long
gem "rack-timeout"
# wrapper for tomorrow.io API
gem "tomorrowio_rb", "~>0.0.3"
gem "geocoder"
gem "nearest_time_zone"
gem "symmetric-encryption"
gem "ruby-progressbar", require: false
gem "kaminari-actionview"
gem "kaminari-mongoid"
gem "rack-cors", "2.0.1", require: "rack/cors" # freezing to gemfile.lock version because heroku is not respecting lockfile
gem "simplecov", require: false, group: :test
group :development, :test do
# Call 'byebug' anywhere in the code to stop execution and get a debugger console
gem "bullet"
gem "byebug"
gem "database_cleaner"
gem "database_cleaner-mongoid"
gem "erb_lint", require: false
gem "factory_bot_rails"
# Generate Fake data
gem "ffaker"
gem "pry-byebug"
gem "pry-doc"
gem "pry-rails"
gem "rspec-rails"
gem "standardrb"
end
group :development do
gem "annotate"
gem "awesome_print"
gem "better_errors"
gem "brakeman"
gem "foreman", require: false
gem "letter_opener"
end
group :test do
gem "capybara"
gem "cuprite"
gem "mongoid-rspec"
gem "shoulda-matchers"
gem "vcr"
gem "webmock"
end
group :production do
gem "rails_12factor"
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[mingw mswin x64_mingw jruby]
gem "bugsnag"

View File

@@ -1,581 +0,0 @@
GEM
remote: https://rubygems.org/
specs:
actioncable (7.1.5.2)
actionpack (= 7.1.5.2)
activesupport (= 7.1.5.2)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
zeitwerk (~> 2.6)
actionmailbox (7.1.5.2)
actionpack (= 7.1.5.2)
activejob (= 7.1.5.2)
activerecord (= 7.1.5.2)
activestorage (= 7.1.5.2)
activesupport (= 7.1.5.2)
mail (>= 2.7.1)
net-imap
net-pop
net-smtp
actionmailer (7.1.5.2)
actionpack (= 7.1.5.2)
actionview (= 7.1.5.2)
activejob (= 7.1.5.2)
activesupport (= 7.1.5.2)
mail (~> 2.5, >= 2.5.4)
net-imap
net-pop
net-smtp
rails-dom-testing (~> 2.2)
actionpack (7.1.5.2)
actionview (= 7.1.5.2)
activesupport (= 7.1.5.2)
nokogiri (>= 1.8.5)
racc
rack (>= 2.2.4)
rack-session (>= 1.0.1)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
actiontext (7.1.5.2)
actionpack (= 7.1.5.2)
activerecord (= 7.1.5.2)
activestorage (= 7.1.5.2)
activesupport (= 7.1.5.2)
globalid (>= 0.6.0)
nokogiri (>= 1.8.5)
actionview (7.1.5.2)
activesupport (= 7.1.5.2)
builder (~> 3.1)
erubi (~> 1.11)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
active_model_serializers (0.9.8)
activemodel (>= 3.2)
concurrent-ruby (~> 1.0)
activejob (7.1.5.2)
activesupport (= 7.1.5.2)
globalid (>= 0.3.6)
activemodel (7.1.5.2)
activesupport (= 7.1.5.2)
activerecord (7.1.5.2)
activemodel (= 7.1.5.2)
activesupport (= 7.1.5.2)
timeout (>= 0.4.0)
activestorage (7.1.5.2)
actionpack (= 7.1.5.2)
activejob (= 7.1.5.2)
activerecord (= 7.1.5.2)
activesupport (= 7.1.5.2)
marcel (~> 1.0)
activesupport (7.1.5.2)
base64
benchmark (>= 0.3)
bigdecimal
concurrent-ruby (~> 1.0, >= 1.0.2)
connection_pool (>= 2.2.5)
drb
i18n (>= 1.6, < 2)
logger (>= 1.4.2)
minitest (>= 5.1)
mutex_m
securerandom (>= 0.3)
tzinfo (~> 2.0)
addressable (2.8.7)
public_suffix (>= 2.0.2, < 7.0)
andand (1.3.3)
annotate (3.2.0)
activerecord (>= 3.2, < 8.0)
rake (>= 10.4, < 14.0)
ast (2.4.2)
awesome_print (1.9.2)
base64 (0.3.0)
bcrypt (3.1.20)
benchmark (0.5.0)
better_errors (2.10.1)
erubi (>= 1.0.0)
rack (>= 0.9.0)
rouge (>= 1.0.0)
better_html (2.1.1)
actionview (>= 6.0)
activesupport (>= 6.0)
ast (~> 2.0)
erubi (~> 1.4)
parser (>= 2.4)
smart_properties
bigdecimal (3.3.1)
brakeman (6.1.2)
racc
bson (4.15.0)
bugsnag (6.27.1)
concurrent-ruby (~> 1.0)
builder (3.3.0)
bullet (7.2.0)
activesupport (>= 3.0.0)
uniform_notifier (~> 1.11)
byebug (11.1.3)
cancancan (3.6.1)
cancancan-mongoid (2.0.0)
cancancan (>= 2.0, < 4)
capybara (3.40.0)
addressable
matrix
mini_mime (>= 0.1.3)
nokogiri (~> 1.11)
rack (>= 1.6.0)
rack-test (>= 0.6.3)
regexp_parser (>= 1.5, < 3.0)
xpath (~> 3.2)
coderay (1.1.3)
coercible (1.0.0)
descendants_tracker (~> 0.0.1)
colored (1.2)
concurrent-ruby (1.3.5)
connection_pool (2.5.5)
countries (4.0.1)
i18n_data (~> 0.13.0)
sixarm_ruby_unaccent (~> 1.1)
crack (1.0.1)
bigdecimal
rexml
crass (1.0.6)
csv (3.3.0)
cuprite (0.15)
capybara (~> 3.0)
ferrum (~> 0.14.0)
database_cleaner (2.1.0)
database_cleaner-active_record (>= 2, < 3)
database_cleaner-active_record (2.2.2)
activerecord (>= 5.a)
database_cleaner-core (~> 2.0)
database_cleaner-core (2.0.1)
database_cleaner-mongoid (2.0.1)
database_cleaner-core (~> 2.0.0)
mongoid
date (3.5.0)
descendants_tracker (0.0.4)
thread_safe (~> 0.3, >= 0.3.1)
devise (4.9.4)
bcrypt (~> 3.0)
orm_adapter (~> 0.1)
railties (>= 4.1.0)
responders
warden (~> 1.2.3)
devise_invitable (2.0.11)
actionmailer (>= 5.0)
devise (>= 4.6)
diff-lcs (1.6.2)
docile (1.4.0)
dotenv (3.1.0)
dotenv-rails (3.1.0)
dotenv (= 3.1.0)
railties (>= 6.1)
drb (2.2.3)
erb (6.0.0)
erb_lint (0.5.0)
activesupport
better_html (>= 2.0.1)
parser (>= 2.7.1.4)
rainbow
rubocop
smart_properties
erubi (1.13.1)
factory_bot (6.4.6)
activesupport (>= 5.0.0)
factory_bot_rails (6.4.3)
factory_bot (~> 6.4)
railties (>= 5.0.0)
faraday (1.8.0)
faraday-em_http (~> 1.0)
faraday-em_synchrony (~> 1.0)
faraday-excon (~> 1.1)
faraday-httpclient (~> 1.0.1)
faraday-net_http (~> 1.0)
faraday-net_http_persistent (~> 1.1)
faraday-patron (~> 1.0)
faraday-rack (~> 1.0)
multipart-post (>= 1.2, < 3)
ruby2_keywords (>= 0.0.4)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
ferrum (0.14)
addressable (~> 2.5)
concurrent-ruby (~> 1.1)
webrick (~> 1.7)
websocket-driver (>= 0.6, < 0.8)
ffaker (2.23.0)
foreman (0.88.1)
geocoder (1.8.3)
base64 (>= 0.1.0)
csv (>= 3.0.0)
globalid (1.3.0)
activesupport (>= 6.1)
globalize (6.3.0)
activemodel (>= 4.2, < 7.2)
activerecord (>= 4.2, < 7.2)
request_store (~> 1.0)
hashdiff (1.2.1)
hashie (3.5.7)
httpclient (2.8.3)
i18n (1.14.7)
concurrent-ruby (~> 1.0)
i18n_data (0.13.0)
io-console (0.8.1)
irb (1.15.3)
pp (>= 0.6.0)
rdoc (>= 4.0.0)
reline (>= 0.4.2)
json (2.7.1)
jwt (2.3.0)
kaminari-actionview (1.2.1)
actionview
kaminari-core (= 1.2.1)
kaminari-core (1.2.1)
kaminari-mongoid (1.0.2)
kaminari-core (~> 1.0)
mongoid
kdtree (0.4)
language_server-protocol (3.17.0.3)
launchy (2.5.2)
addressable (~> 2.8)
letter_opener (1.10.0)
launchy (>= 2.2, < 4)
lint_roller (1.1.0)
logger (1.7.0)
loofah (2.24.1)
crass (~> 1.0.2)
nokogiri (>= 1.12.0)
mail (2.9.0)
logger
mini_mime (>= 0.1.1)
net-imap
net-pop
net-smtp
marcel (1.0.4)
matrix (0.4.2)
method_source (1.1.0)
mini_mime (1.1.5)
mini_portile2 (2.8.9)
minitest (5.26.2)
mongo (2.20.1)
bson (>= 4.14.1, < 6.0.0)
mongoid (8.1.3)
activemodel (>= 5.1, < 7.2, != 7.0.0)
concurrent-ruby (>= 1.0.5, < 2.0)
mongo (>= 2.18.0, < 3.0.0)
ruby2_keywords (~> 0.0.5)
mongoid-compatibility (0.6.0)
activesupport
mongoid (>= 2.0)
mongoid-rspec (4.2.0)
mongoid (>= 3.0, < 10.0)
mongoid-compatibility (>= 0.5.1)
multi_json (1.15.0)
multi_xml (0.6.0)
multipart-post (2.1.1)
mutex_m (0.3.0)
nearest_time_zone (0.0.4)
andand
kdtree
require_all
net-imap (0.5.12)
date
net-protocol
net-pop (0.1.2)
net-protocol
net-protocol (0.2.2)
timeout
net-smtp (0.5.1)
net-protocol
nio4r (2.7.3)
nokogiri (1.18.10)
mini_portile2 (~> 2.8.2)
racc (~> 1.4)
oauth2 (1.4.7)
faraday (>= 0.8, < 2.0)
jwt (>= 1.0, < 3.0)
multi_json (~> 1.3)
multi_xml (~> 0.5)
rack (>= 1.2, < 3)
omniauth (1.8.1)
hashie (>= 3.4.6, < 3.6.0)
rack (>= 1.6.2, < 3)
omniauth-facebook (3.0.0)
omniauth-oauth2 (~> 1.2)
omniauth-oauth2 (1.5.0)
oauth2 (~> 1.1)
omniauth (~> 1.2)
orm_adapter (0.5.0)
parallel (1.24.0)
parser (3.3.0.5)
ast (~> 2.4.1)
racc
pg (1.5.6)
pp (0.6.3)
prettyprint
prettyprint (0.2.0)
pry (0.14.2)
coderay (~> 1.1)
method_source (~> 1.0)
pry-byebug (3.10.1)
byebug (~> 11.0)
pry (>= 0.13, < 0.15)
pry-doc (1.5.0)
pry (~> 0.11)
yard (~> 0.9.11)
pry-rails (0.3.11)
pry (>= 0.13.0)
psych (5.2.6)
date
stringio
public_suffix (6.0.2)
puma (5.6.8)
nio4r (~> 2.0)
pusher (2.0.3)
httpclient (~> 2.8)
multi_json (~> 1.15)
pusher-signature (~> 0.1.8)
pusher-signature (0.1.8)
racc (1.8.1)
rack (2.2.21)
rack-cors (2.0.1)
rack (>= 2.0.0)
rack-session (1.0.2)
rack (< 3)
rack-test (2.2.0)
rack (>= 1.3)
rack-timeout (0.7.0)
rackup (1.0.1)
rack (< 3)
webrick
rails (7.1.5.2)
actioncable (= 7.1.5.2)
actionmailbox (= 7.1.5.2)
actionmailer (= 7.1.5.2)
actionpack (= 7.1.5.2)
actiontext (= 7.1.5.2)
actionview (= 7.1.5.2)
activejob (= 7.1.5.2)
activemodel (= 7.1.5.2)
activerecord (= 7.1.5.2)
activestorage (= 7.1.5.2)
activesupport (= 7.1.5.2)
bundler (>= 1.15.0)
railties (= 7.1.5.2)
rails-dom-testing (2.3.0)
activesupport (>= 5.0.0)
minitest
nokogiri (>= 1.6)
rails-html-sanitizer (1.6.2)
loofah (~> 2.21)
nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
rails_12factor (0.0.3)
rails_serve_static_assets
rails_stdout_logging
rails_serve_static_assets (0.0.5)
rails_stdout_logging (0.0.5)
railties (7.1.5.2)
actionpack (= 7.1.5.2)
activesupport (= 7.1.5.2)
irb
rackup (>= 1.0.0)
rake (>= 12.2)
thor (~> 1.0, >= 1.2.2)
zeitwerk (~> 2.6)
rainbow (3.1.1)
rake (13.2.1)
rdoc (6.16.0)
erb
psych (>= 4.0.0)
tsort
redis-client (0.26.1)
connection_pool
regexp_parser (2.9.0)
reline (0.6.3)
io-console (~> 0.5)
request_store (1.5.0)
rack (>= 1.4)
require_all (3.0.0)
responders (3.2.0)
actionpack (>= 7.0)
railties (>= 7.0)
rexml (3.4.4)
rouge (4.2.1)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
rspec-expectations (3.13.5)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-mocks (3.13.7)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-rails (7.1.1)
actionpack (>= 7.0)
activesupport (>= 7.0)
railties (>= 7.0)
rspec-core (~> 3.13)
rspec-expectations (~> 3.13)
rspec-mocks (~> 3.13)
rspec-support (~> 3.13)
rspec-support (3.13.6)
rubocop (1.62.1)
json (~> 2.3)
language_server-protocol (>= 3.17.0)
parallel (~> 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 1.8, < 3.0)
rexml (>= 3.2.5, < 4.0)
rubocop-ast (>= 1.31.1, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 3.0)
rubocop-ast (1.31.2)
parser (>= 3.3.0.4)
rubocop-performance (1.20.2)
rubocop (>= 1.48.1, < 2.0)
rubocop-ast (>= 1.30.0, < 2.0)
ruby-progressbar (1.13.0)
ruby2_keywords (0.0.5)
securerandom (0.4.1)
seedbank (0.5.0)
rake (>= 10.0)
shoulda-matchers (6.2.0)
activesupport (>= 5.2.0)
sidekiq (7.3.9)
base64
connection_pool (>= 2.3.0)
logger
rack (>= 2.2.4)
redis-client (>= 0.22.2)
simplecov (0.22.0)
docile (~> 1.1)
simplecov-html (~> 0.11)
simplecov_json_formatter (~> 0.1)
simplecov-html (0.12.3)
simplecov_json_formatter (0.1.4)
sixarm_ruby_unaccent (1.2.0)
smart_properties (1.17.0)
sprockets (4.2.2)
concurrent-ruby (~> 1.0)
logger
rack (>= 2.2.4, < 4)
sprockets-rails (3.5.2)
actionpack (>= 6.1)
activesupport (>= 6.1)
sprockets (>= 3.0.0)
standard (1.35.1)
language_server-protocol (~> 3.17.0.2)
lint_roller (~> 1.0)
rubocop (~> 1.62.0)
standard-custom (~> 1.0.0)
standard-performance (~> 1.3)
standard-custom (1.0.2)
lint_roller (~> 1.0)
rubocop (~> 1.50)
standard-performance (1.3.1)
lint_roller (~> 1.1)
rubocop-performance (~> 1.20.2)
standardrb (1.0.1)
standard
stringio (3.1.8)
symmetric-encryption (4.6.0)
coercible (~> 1.0)
thor (1.4.0)
thread_safe (0.3.6)
timeout (0.4.4)
tomorrowio_rb (0.0.3)
tsort (0.2.0)
tzinfo (2.0.6)
concurrent-ruby (~> 1.0)
unicode-display_width (2.5.0)
uniform_notifier (1.16.0)
vcr (6.3.1)
base64
warden (1.2.9)
rack (>= 2.0.9)
webmock (3.26.1)
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
webrick (1.9.2)
websocket-driver (0.7.6)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
xpath (3.2.0)
nokogiri (~> 1.8)
yard (0.9.36)
zeitwerk (2.7.3)
PLATFORMS
ruby
DEPENDENCIES
active_model_serializers (~> 0.9)
annotate
awesome_print
better_errors
brakeman
bugsnag
bullet
byebug
cancancan (~> 3.6.1)
cancancan-mongoid (~> 2.0)
capybara
colored
countries
cuprite
database_cleaner
database_cleaner-mongoid
devise (~> 4.8)
devise_invitable (~> 2.0)
dotenv-rails
erb_lint
factory_bot_rails
ffaker
foreman
geocoder
globalize
kaminari-actionview
kaminari-mongoid
letter_opener
mongoid (= 8.1.3)
mongoid-rspec
nearest_time_zone
omniauth (~> 1.8)
omniauth-facebook (~> 3.0)
pg
pry-byebug
pry-doc
pry-rails
puma (= 5.6.8)
pusher
rack-cors (= 2.0.1)
rack-timeout
rails (~> 7.1.0)
rails_12factor
rake
rspec-rails
ruby-progressbar
seedbank
shoulda-matchers
sidekiq (~> 7.3)
simplecov
sprockets-rails
standardrb
symmetric-encryption
tomorrowio_rb (~> 0.0.3)
tzinfo-data
vcr
webmock
RUBY VERSION
ruby 3.2.3p157
BUNDLED WITH
2.5.6

View File

@@ -1,2 +0,0 @@
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml

View File

@@ -1,2 +0,0 @@
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml

View File

@@ -1,5 +0,0 @@
# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
require File.expand_path("../config/application", __FILE__)
Rails.application.load_tasks

View File

@@ -1,3 +0,0 @@
//= link_tree ../images
//= link_directory ../javascripts .js
//= link_directory ../stylesheets .css

View File

@@ -1,28 +0,0 @@
module Api
module V1
class AwsSesController < ApplicationController
skip_authorize_resource only: [:mail_it, :notification]
skip_before_action :authenticate_user!, only: [:mail_it, :notification]
def notification
message_type = request.headers["x-amz-sns-message-type"]
# sns_topic = request.headers['x-amz-sns-topic-arn']
raw_post = request.raw_post
if message_type.include? "Confirmation"
send_subscription_confirmation(raw_post)
elsif message_type.include? "Notification"
EmailRejectDispatcher.perform_async(raw_post)
end
render nothing: true, status: 200
end
def send_subscription_confirmation(raw_post)
json = JSON.parse(raw_post)
open(json["SubscribeURL"])
end
end
end
end

View File

@@ -1,9 +0,0 @@
module Api
module V1
class ChartListsController < ApplicationController
def show
render json: ChartListService.new(current_user: current_user).as_json
end
end
end
end

View File

@@ -1,32 +0,0 @@
module Api
module V1
class ChartsController < ApplicationController
def show
chart = Chart.new(chart_params)
# FIXME
# rubocop:disable Style/SignalException
fail(ActiveRecord::RecordInvalid, chart) if chart.invalid?
# rubocop:enable Style/SignalException
render json: chart
end
def chart_params
includes_params = {
tags: [],
foods: [],
symptoms: [],
conditions: [],
treatments: [],
weathersMeasures: [],
harveyBradshawIndices: []
}
params.permit(:id, :start_at, :end_at, includes: includes_params).tap do |whitelist|
whitelist[:user] = current_user
end
end
end
end
end

View File

@@ -1,31 +0,0 @@
module Api
module V1
class ChartsPatternController < ApplicationController
skip_before_action :authenticate_user!, only: [:index]
def index
offset = charts_pattern_params[:offset].to_i
start_at = (charts_pattern_params[:start_at].to_date - offset.days).to_s
end_date = charts_pattern_params[:end_at].to_date
end_at = ((Time.current.to_date == end_date) ? end_date : (end_date + offset.days)).to_s
@patterns = Pattern.where(id: {"$in": charts_pattern_params[:pattern_ids] || []})
@extended_patterns = @patterns.map do |pattern|
pattern.extend(PatternExtender).form_chart_data(start_at: start_at,
end_at: end_at,
pattern: pattern)
end
render json: @extended_patterns, meta: {color_ids: Flaredown::Colorable::IDS}
end
private
def charts_pattern_params
params.permit(:start_at, :end_at, :offset, pattern_ids: [])
end
end
end
end

View File

@@ -1,40 +0,0 @@
module Api
module V1
class CheckinsController < ApplicationController
def index
date = params[:date]
if date.blank? && params.require(:page)
render json: current_user.checkins.where(:note.nin => [nil, ""]).order_by(date: :desc).page(params[:page]).per(10)
else
render json: current_user.checkins.includes([:harvey_bradshaw_index, :promotion_rate, :conditions, :symptoms, :treatments]).select { |x|
x.date.to_date == Date.parse(date)
}
end
end
def show
render json: Checkin.find(id)
end
def create
date = params.require(:checkin).require(:date)
parsed = DateTime.parse(date)
now = DateTime.current
save_date = DateTime.new(parsed.year, parsed.month, parsed.day, now.hour, now.minute, now.second)
checkin = Checkin::Creator.new(current_user.id, save_date).create!
render json: checkin
end
def update
render json: Checkin::Updater.new(current_user, params).update!
end
private
def id
params.require(:id)
end
end
end
end

View File

@@ -1,45 +0,0 @@
module Api
module V1
class CommentsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:index]
def index
render json: @comments.where(:id.in => params[:ids]).order_by(created_at: :asc)
end
def show
render json: @comment
end
def create
@comment.encrypted_user_id = current_user.encrypted_id
if @comment.save
UpdatePostCountersJob.perform_async(parent_id: create_params[:post_id], parent_type: "Post")
unless @comment.encrypted_user_id == @comment.post.encrypted_user_id
Notification.create(
kind: :comment,
notificateable: @comment,
encrypted_user_id: @comment.encrypted_user_id,
encrypted_notify_user_id: @comment.post.encrypted_user_id
)
end
DiscussionMention.perform_async(current_user.encrypted_id, @comment.id.to_s)
render json: @comment, status: :created
else
render json: {errors: @comment.errors}, status: :unprocessable_entity
end
end
private
def create_params
params.require(:comment).permit(:body, :post_id)
end
end
end
end

View File

@@ -1,33 +0,0 @@
module Api
module V1
class ConditionsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:show]
def index
@conditions = @conditions.includes(:translations)
@conditions = ids.present? ? @conditions.where(id: ids) : @conditions.order(:name).limit(50)
render json: @conditions
end
def show
render json: @condition
end
def create
render json: TrackableCreator.new(@condition, current_user).create!
end
private
def create_params
params.require(:condition).permit(:name)
end
def ids
@ids ||= params[:ids] if params[:ids].is_a?(Array)
end
end
end
end

View File

@@ -1,32 +0,0 @@
module Api
module V1
class CountriesController < ApplicationController
skip_before_action :authenticate_user!
def index
render json: Country.all, each_serializer: CountrySerializer
end
def show
country = Country.find_country_by_alpha2(alpha2)
# FIXME
# rubocop:disable Style/SignalException
fail ActiveRecord::RecordNotFound if country.nil?
# rubocop:enable Style/SignalException
render json: country, serializer: CountrySerializer
end
private
def alpha2
id = params.require(:id)
match_data = /^[[:alpha:]]{2}$/.match(id)
# FIXME
# rubocop:disable Style/SignalException
fail(ActionController::BadRequest, "id param must be a 2 alphabetic characters string") if match_data.nil?
# rubocop:enable Style/SignalException
match_data[0]
end
end
end
end

View File

@@ -1,11 +0,0 @@
module Api
module V1
class DataExportSchedulesController < ApplicationController
def create
DataExportJob.perform_later(current_user.id)
head :created
end
end
end
end

View File

@@ -1,27 +0,0 @@
module Api
module V1
class DayHabitsController < ApplicationController
skip_before_action :authenticate_user!
def index
render json: DayHabit.all
end
def show
day_habit = DayHabit.find(day_habit_id)
render json: day_habit
end
private
def day_habit_id
id = params.require(:id)
# FIXME
# rubocop:disable Style/SignalException
fail(ActionController::BadRequest, "id param is not a valid day_habit id") unless DayHabit.all_ids.include?(id)
# rubocop:enable Style/SignalException
id
end
end
end
end

View File

@@ -1,9 +0,0 @@
module Api
module V1
class DiscoursesController < ApplicationController
def create
render json: {url: DiscourseClient.new(current_user, params).generate_url}
end
end
end
end

View File

@@ -1,29 +0,0 @@
module Api
module V1
class EducationLevelsController < ApplicationController
skip_before_action :authenticate_user!
def index
render json: EducationLevel.all
end
def show
education_level = EducationLevel.find(education_level_id)
render json: education_level
end
private
def education_level_id
id = params.require(:id)
# FIXME
# rubocop:disable Style/SignalException
unless EducationLevel.all_ids.include?(id)
fail(ActionController::BadRequest, "id param is not a valid education_level id")
end
# rubocop:enable Style/SignalException
id
end
end
end
end

View File

@@ -1,27 +0,0 @@
module Api
module V1
class EthnicitiesController < ApplicationController
skip_before_action :authenticate_user!
def index
render json: Ethnicity.all
end
def show
ethnicity = Ethnicity.find(ethnicity_id)
render json: ethnicity
end
private
def ethnicity_id
id = params.require(:id)
# FIXME
# rubocop:disable Style/SignalException
fail(ActionController::BadRequest, "id param is not a valid ethnicity id") unless Ethnicity.all_ids.include?(id)
# rubocop:enable Style/SignalException
id
end
end
end
end

View File

@@ -1,42 +0,0 @@
module Api
module V1
class FoodsController < ApplicationController
load_and_authorize_resource
def index
@foods = @foods.includes(:translations)
foods =
if ids.present?
@foods.where(id: ids)
elsif scope.present?
CollectionRetriever.new(Food, scope, current_user).retrieve
end
render json: foods
end
def show
render json: @food
end
def create
render json: TrackableCreator.new(@food, current_user).create!
end
private
def create_params
{long_desc: params.require(:food).require(:name)}
end
def ids
@ids ||= params[:ids]
end
def scope
@scope ||= params[:scope]&.to_sym
end
end
end
end

View File

@@ -1,28 +0,0 @@
module Api
module V1
class HarveyBradshawIndicesController < ApplicationController
load_and_authorize_resource
def show
render json: @harvey_bradshaw_index
end
def create
@harvey_bradshaw_index.save
render json: @harvey_bradshaw_index
end
private
def create_params
params.require(:harvey_bradshaw_index).permit(
:abdominal_mass, :abdominal_pain, :abscess,
:anal_fissure, :aphthous_ulcers, :arthralgia,
:checkin_id, :erythema_nodosum, :new_fistula,
:pyoderma_gangrenosum, :stools, :uveitis, :well_being
)
end
end
end
end

View File

@@ -1,19 +0,0 @@
module Api
module V1
class InvitationsController < ApplicationController
skip_before_action :authenticate_user!
def show
render json: Invitation.find(params[:id])
end
def update
invitation = Invitation.find(params[:id])
invitation.accept!(
params.require(:invitation).permit(:email, :password, :password_confirmation)
)
render json: invitation
end
end
end
end

View File

@@ -1,52 +0,0 @@
module Api
module V1
class NotificationsController < ApplicationController
def index
notifications = Notification.where(encrypted_notify_user_id: current_user.encrypted_id)
authorize_collection :index, notifications
render json: {notifications: notifications.aggregated_by_kind_and_subject}
end
def update
notifications = Notification.where(notification_params)
authorize_collection :update, notifications
if notifications.update_all(unread: false)
render json: {notifications: notifications.aggregated_by_kind_and_subject}
else
render json: {errors: notifications.map(&:errors).compact}, status: :unprocessable_entity
end
end
def destroy
notifications = Notification.where(notification_params)
authorize_collection :destroy, notifications
if notifications.destroy
head :no_content
else
render json: {errors: notifications.map(&:errors).compact}, status: :unprocessable_entity
end
end
private
def notification_params
parameters = params.permit(:notificateable_id, :notificateable_type)
parameters[:notificateable_type] = parameters[:notificateable_type].titleize
parameters[:encrypted_notify_user_id] = current_user.encrypted_id
parameters
end
def authorize_collection(name, collection)
collection.each { |element| authorize! name, element }
end
end
end
end

View File

@@ -1,35 +0,0 @@
module Api
module V1
class OmniauthCallbacksController < Devise::OmniauthCallbacksController
Devise.omniauth_providers.each do |provider|
define_method provider do
handle_omniauth
end
end
def failure
Rails.logger.warn("Api::V1::OmniauthCallbacksController#failure: #{failure_message}".yellow)
render json: {errors: failure_message}, status: 401
end
private
def handle_omniauth
user = User.find_for_database_authentication(email: email_param)
if user && user.invitation_token.nil?
render json: user, root: false, serializer: SessionSerializer
else
render json: {errors: "User not found"}, status: 401
end
end
def oauth_params
@oauth_params ||= ActionController::Parameters.new(request.env["omniauth.auth"])
end
def email_param
oauth_params.fetch(:info).fetch(:email)
end
end
end
end

View File

@@ -1,60 +0,0 @@
module Api
module V1
class OracleRequestsController < ApplicationController
skip_before_action :authenticate_user!
serialization_scope :oracle_token
load_resource
def show
render json: @oracle_request
end
def create
if oracle_token.present?
@oracle_request.token = oracle_token
else
loop do
@oracle_request.token = SecureRandom.uuid
break unless OracleRequest.where(token: @oracle_request.token).exists?
end
end
@oracle_request.save
render json: @oracle_request, serializer: OracleRequestWithTokenSerializer
end
def update
if @oracle_request.can_edit?(oracle_token)
@oracle_request.update!(create_params)
render json: @oracle_request
else
render json: {errors: "Unauthorized"}, status: :unauthorised
end
end
private
def create_params
params.require(:oracle_request).permit(
:age,
:sex_id,
responce: [
:name,
:confidence,
:correction
],
symptom_ids: []
)
end
def oracle_token
request.headers["X-Oracle-Token"]
end
end
end
end

View File

@@ -1,56 +0,0 @@
module Api
module V1
class PasswordsController < ApplicationController
skip_before_action :authenticate_user!
def show
user = user_signed_in? ? current_user : User.with_reset_password_token(params[:id])
if user.blank?
raise ActiveRecord::RecordNotFound, "User not found"
else
render json: user, token: params[:id], serializer: PasswordSerializer
end
end
def create
user = User.find_by!(email: email_param.downcase)
return unless user.send_reset_password_instructions
render json: user, serializer: PasswordSerializer
end
def update
if user_signed_in?
if current_user.update_with_password(update_password_params)
render json: current_user, token: params[:id], serializer: PasswordSerializer
else
render json: {errors: current_user.errors}, status: :unprocessable_entity
end
else
user = User.reset_password_by_token(update_password_by_token_params)
if user.errors.empty?
render json: user, token: params[:id], serializer: PasswordSerializer
else
render json: {errors: user.errors}, status: :unprocessable_entity
end
end
end
private
def email_param
params.require(:password).fetch(:email)
end
def update_password_params
params.require(:password).permit(:current_password, :password, :password_confirmation)
end
def update_password_by_token_params
params.require(:password).permit(:reset_password_token, :password, :password_confirmation)
end
end
end
end

View File

@@ -1,68 +0,0 @@
module Api
module V1
class PatternsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:index]
def index
page = params[:page] || 1
pattern_ids = params[:pattern_ids]
@patterns =
if pattern_ids.present?
Pattern.where(id: {"$in" => pattern_ids})
else
Pattern.accessible_by(current_ability).where(encrypted_user_id: encrypted_user_id)
end
render json: @patterns.page(page).per(10)
end
def show
pattern = Pattern.find_by(id: pattern_params[:id])
render json: pattern
end
def create
@pattern = PatternCreator.new(pattern_params.to_h).create
render json: @pattern
end
def update
@pattern.update(pattern_params)
render json: @pattern
end
def destroy
pattern = Pattern.find_by(id: params[:id])
authorize! :destroy, pattern
if pattern.destroy
head :no_content
else
render json: {errors: pattern.errors}, status: :unprocessable_entity
end
end
private
def pattern_params
params.require(:pattern)
.permit(:name, :start_at, :end_at, includes: [:id, :category, :label])
.merge(user_id: current_user.id)
end
def current_ability
@current_ability ||= Ability.new(current_user)
end
def encrypted_user_id
@encrypted_user_id ||= SymmetricEncryption.encrypt(current_user.id)
end
end
end
end

View File

@@ -1,18 +0,0 @@
module Api
module V1
class PostablesController < ApplicationController
load_and_authorize_resource
def index
render json: PostableSerializer.new(
@postables
.where(encrypted_user_id: current_user.encrypted_id)
.order_by(created_at: :desc)
.page(params[:page])
.per(20),
current_user
)
end
end
end
end

View File

@@ -1,46 +0,0 @@
module Api
module V1
class PostsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:index, :show]
def index
if params[:summary]
render json: SummaryPosts.new(current_user).show_list
else
@posts = DiscussionPosts.new(params, current_user).show_list
results = @posts
.includes([:comments, :notifications, :reactions])
.order(last_commented: :desc, created_at: :desc)
.page(params[:page])
.per(10)
render json: results
end
end
def show
render json: @post
end
def create
@post.encrypted_user_id = current_user.encrypted_id
if @post.save
render json: @post, status: :created
else
render json: {errors: @post.errors}, status: :unprocessable_entity
end
end
private
def create_params
params.require(:post).permit(
:title, :body,
tag_ids: [], symptom_ids: [], condition_ids: [], treatment_ids: []
)
end
end
end
end

View File

@@ -1,75 +0,0 @@
module Api
module V1
class ProfilesController < ApplicationController
require "sidekiq/api"
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:index]
def index
post = Post.find(params[:post_id])
encrypted_user_ids =
(post.comments.distinct(:encrypted_user_id) << post.encrypted_user_id).uniq.map do |encrypted_id|
SymmetricEncryption.decrypt(encrypted_id)
end
@profiles = Profile.where(user_id: encrypted_user_ids).where.not(slug_name: nil)
render json: @profiles.map { |profile| profile.attributes.slice("screen_name", "slug_name") }
end
def show
render json: @profile
end
def update
initial_onboarding_reminder = params.dig(:profile, :onboarding_reminder)
@profile.assign_attributes(update_params.merge(transform_hash_time))
time_changed = @profile.checkin_reminder_at_changed? || @profile.time_zone_name_changed?
@profile.save!
if time_changed || initial_onboarding_reminder
delete_old_job(@profile.reminder_job_id)
job_id = CheckinReminderJob.perform_in(get_reminder_time.minutes, @profile.id, @profile.checkin_reminder_at)
@profile.update_column(:reminder_job_id, job_id)
end
current_user.profile.reload
set_locale
render json: @profile
end
private
def update_params
params.require(:profile).permit(
:country_id, :sex_id, :onboarding_step_id, :birth_date,
:day_habit_id, :education_level_id, :day_walking_hours,
:pressure_units, :temperature_units, :screen_name, :notify,
:checkin_reminder, :time_zone_name, :notify_top_posts, ethnicity_ids: []
)
end
def transform_hash_time
checkin_reminder_at = params.require(:profile)[:checkin_reminder_at]
user_time = checkin_reminder_at && checkin_reminder_at.values.join(":")
{checkin_reminder_at: user_time.try(:to_time, :utc)}
end
def get_reminder_time
time_zone_name = @profile.time_zone_name
checkin_at_timezone = @profile.checkin_reminder_at.strftime("%H:%M").in_time_zone(time_zone_name)
# Select minutes
(checkin_at_timezone - Time.current.in_time_zone(time_zone_name)).divmod(1.day)[1].divmod(1.minute)[0]
end
def delete_old_job(enqueued_job_id)
Sidekiq::ScheduledSet.new.find_job(enqueued_job_id)&.delete
end
end
end
end

View File

@@ -1,35 +0,0 @@
module Api
module V1
class PromotionRatesController < ApplicationController
load_and_authorize_resource
def show
render json: @promotion_rate
end
def create
@promotion_rate.save
render json: @promotion_rate
end
def update
@promotion_rate.update(resource_params.merge(additional_params))
render json: @promotion_rate
end
private
def resource_params
params.require(:promotion_rate).permit(:checkin_id, :score, :feedback)
end
def additional_params
user = @promotion_rate.checkin.user
{user_created_at: user.created_at}
end
end
end
end

View File

@@ -1,17 +0,0 @@
module Api
module V1
class PushersController < ApplicationController
def create
render json: Flaredown.pusher.authenticate!(current_user, socket_id)
rescue
render json: {errors: "Bad authentication"}, status: "403"
end
private
def socket_id
params.require(:socket_id)
end
end
end
end

View File

@@ -1,77 +0,0 @@
module Api
module V1
class ReactionsController < ApplicationController
def create
react(__method__)
end
def update
react(__method__)
end
def destroy
reaction = Reaction.where(reaction_params).first
authorize! :destroy, reaction
if reaction.destroy
UpdatePostCountersJob.perform_async(parent_id: reaction_params[:reactable_id],
parent_type: reaction_params[:reactable_type])
head :no_content
else
render json: {errors: reaction.errors}, status: :unprocessable_entity
end
end
private
def react(method_name)
reaction = Reaction.find_or_initialize_by(reaction_params)
authorize! method_name, reaction
if reaction.save
UpdatePostCountersJob.perform_async(parent_id: reaction_params[:reactable_id],
parent_type: reaction_params[:reactable_type])
unless reaction.encrypted_user_id == reaction.reactable.encrypted_user_id
Notification.create(
kind: :reaction,
notificateable: reaction.reactable,
encrypted_user_id: reaction.encrypted_user_id,
encrypted_notify_user_id: reaction.reactable.encrypted_user_id
)
end
reaction.id = params[:id] if params[:id].present?
render json: serialized_reaction(reaction), status: :created
else
render json: {errors: reaction.errors}, status: :unprocessable_entity
end
end
def reaction_params
reaction = params.require(:reaction)
{
value: reaction[:value],
reactable_id: reaction[:reactable_id],
reactable_type: reaction[:reactable_type].titleize,
encrypted_user_id: current_user.encrypted_id
}
end
def serialized_reaction(reaction)
ReactionSerializer
.new(
Reaction.similar_to(reaction).values_count_with_participated(current_user.encrypted_id),
reaction.reactable_id.to_s,
reaction.reactable_type
)
.serialize_one
end
end
end
end

View File

@@ -1,15 +0,0 @@
module Api
module V1
class RegistrationsController < ApplicationController
skip_before_action :authenticate_user!
def create
render json: Registration.create!(params)
end
def destroy
render json: Registration.delete!(params)
end
end
end
end

View File

@@ -1,34 +0,0 @@
module Api
module V1
class SearchesController < ApplicationController
SEARCH_MAPPER = {
"dose" => Search::ForDose,
"food" => Search::ForFood,
"topic" => Search::ForTopic
}.freeze
skip_before_action :authenticate_user!, only: :show
def show
search = (SEARCH_MAPPER[resource_param] || Search).new(search_params)
# FIXME
# rubocop:disable Style/SignalException
fail(ActiveRecord::RecordInvalid, search) if search.invalid?
# rubocop:enable Style/SignalException
render json: search, serializer: SearchSerializer
end
def search_params
params.permit(:resource, :scope, query: [:name, :treatment_id]).tap do |params|
params[:user] = current_user
end
end
def resource_param
params[:resource]
end
end
end
end

View File

@@ -1,29 +0,0 @@
module Api
module V1
class SessionsController < ApplicationController
skip_before_action :authenticate_user!
def create
# FIXME
# rubocop:disable Style/SignalException
fail "missing information" if params[:user].nil?
fail "invalid email or password" if user.nil?
# rubocop:enable Style/SignalException
render json: user, root: false, serializer: SessionSerializer
rescue => e
render json: {errors: Array(e.message)}, status: 401
end
private
def user
@user ||=
begin
user = User.find_for_database_authentication(email: params[:user][:email])
user if user && user.valid_password?(params[:user][:password])
end
end
end
end
end

View File

@@ -1,27 +0,0 @@
module Api
module V1
class SexesController < ApplicationController
skip_before_action :authenticate_user!
def index
render json: Sex.all
end
def show
sex = Sex.find(sex_id)
render json: sex
end
private
def sex_id
id = params.require(:id)
# FIXME
# rubocop:disable Style/SignalException
fail(ActionController::BadRequest, "id param is not a valid sex id") unless Sex.all_ids.include?(id)
# rubocop:enable Style/SignalException
id
end
end
end
end

View File

@@ -1,33 +0,0 @@
module Api
module V1
class SymptomsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:show]
def index
@symptoms = @symptoms.includes(:translations)
@symptoms = ids.present? ? @symptoms.where(id: ids) : @symptoms.order(:name).limit(50)
render json: @symptoms
end
def show
render json: @symptom
end
def create
render json: TrackableCreator.new(@symptom, current_user).create!
end
private
def create_params
params.require(:symptom).permit(:name)
end
def ids
@ids ||= params[:ids] if params[:ids].is_a?(Array)
end
end
end
end

View File

@@ -1,40 +0,0 @@
module Api
module V1
class TagsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:show]
def index
@tags = @tags.includes(:translations)
if ids.present?
@tags = @tags.where(id: ids)
elsif scope.present?
@tags = CollectionRetriever.new(Tag, scope, current_user).retrieve
end
render json: @tags
end
def show
render json: @tag
end
def create
render json: TrackableCreator.new(@tag, current_user).create!
end
private
def create_params
params.require(:tag).permit(:name)
end
def ids
@ids ||= params[:ids]
end
def scope
@scope ||= params[:scope].try(:to_sym)
end
end
end
end

View File

@@ -1,32 +0,0 @@
module Api
module V1
class TopicFollowingsController < ApplicationController
load_and_authorize_resource
def show
render json: @topic_following
end
def update
if @topic_following.update(update_params)
render json: @topic_following, status: :ok
else
render json: {errors: @topic_following.errors}, status: :unprocessable_entity
end
end
private
def update_params
empty_params = {
"tag_ids" => [],
"symptom_ids" => [],
"condition_ids" => [],
"treatment_ids" => []
}
empty_params.merge(params.require(:topic_following).permit(empty_params))
end
end
end
end

View File

@@ -1,46 +0,0 @@
module Api
module V1
class TrackingsController < ApplicationController
load_and_authorize_resource except: :create
def index
render json: @trackings.by_trackable_type(trackable_type).active_at(at)
end
def show
render json: @tracking
end
def create
tracking = Tracking.new(create_params.merge(start_at: Time.zone.today, user: current_user))
authorize! :create, tracking
tracking.save!
current_user.topic_following.add_topic("#{tracking.trackable_type.downcase}_ids", tracking.trackable_id)
render json: tracking
end
def destroy
TrackingDestroyer.new(current_user, @tracking, Time.zone.today).destroy
head :no_content
end
private
def at
Time.zone.parse(params.require(:at))
end
def trackable_type
params.require(:trackable_type)
end
def create_params
params.require(:tracking).permit(:trackable_id, :trackable_type, :color_id)
end
end
end
end

View File

@@ -1,33 +0,0 @@
module Api
module V1
class TreatmentsController < ApplicationController
load_and_authorize_resource
skip_before_action :authenticate_user!, only: [:show]
def index
@treatments = @treatments.includes(:translations)
@treatments = ids.present? ? @treatments.where(id: ids) : @treatments.order(:name).limit(50)
render json: @treatments
end
def show
render json: @treatment
end
def create
render json: TrackableCreator.new(@treatment, current_user).create!
end
private
def create_params
params.require(:treatment).permit(:name)
end
def ids
@ids ||= params[:ids] if params[:ids].is_a?(Array)
end
end
end
end

View File

@@ -1,30 +0,0 @@
module Api
module V1
class UnsubscribesController < ApplicationController
skip_before_action :authenticate_user!
def update
@profile = Profile.find_by(notify_token: activation_params[:notify_token])
@profile&.update_column(attribute_dispatcher_key, false)
render json: @profile
end
private
def activation_params
params.permit(:notify_token, :notify_top_posts, :stop_remind)
end
def attribute_dispatcher_key
if activation_params[:stop_remind]
:checkin_reminder
elsif activation_params[:notify_top_posts]
:notify_top_posts
else
:notify
end
end
end
end
end

Some files were not shown because too many files have changed in this diff Show More