eric/project-work
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

stuff

Browse Source
This commit is contained in:
Eric Bell
2026-09-11 10:54:37 -04:00
parent 6850368c2d
commit 989deccb63
4 changed files with 295 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
explore.md Normal file
View File

@@ -0,0 +1,112 @@
Deeply explore the current working directory (or a path the user specifies), extract the most salient facts about the codebase, and write them to **OVERVIEW.md** in the project root.
The goal is a document a new developer could read on day one to understand *what the app does*, *how it's structured*, *what it connects to*, and *where the interesting parts are*. Be specific and factual — avoid vague summaries. If you find a concrete detail (a database URL format, an API endpoint, a notable architectural pattern), include it.
## Exploration strategy
Use the tools available to you to explore in parallel where possible. Here's what to look for:
**Start with the high-level anchors:**
- `package.json` / `Cargo.toml` / `pyproject.toml` / `go.mod` — dependencies, scripts, metadata
- `README.md` if it exists — stated purpose
- Main entry point (e.g. `src/main.tsx`, `app.py`, `cmd/main.go`, `index.js`)
- Build/config files (e.g. `vite.config.*`, `webpack.config.*`, `docker-compose.yml`, `.env.example`)
**File and directory structure:**
- Walk the top 2–3 levels of the directory tree
- Identify major groupings (e.g. `routes/`, `components/`, `api/`, `db/`, `services/`)
- Note any monorepo structure (workspaces, `packages/`, `apps/`)
**Tech stack:**
- Framework(s) and runtime
- Language(s)
- Build tooling
- Test framework
**Integrations:**
- Third-party APIs and SDKs (look for imports, env var names, config keys)
- Authentication providers
- Analytics, monitoring, feature flags
- Payment processors, messaging services, etc.
**Database and data layer:**
- ORM or query library in use
- Database type (Postgres, MySQL, SQLite, MongoDB, etc.)
- Schema files or migration directories
- Connection config (env var names, config files)
**Connectivity and configuration:**
- `.env.example` or similar — what env vars are expected
- API proxy config (e.g. Vite's `server.proxy`, nginx config)
- Port numbers, base URLs, service addresses
- Any hardcoded endpoints or service URLs in source
**Architecture patterns:**
- State management approach
- Routing strategy
- Notable design patterns (e.g. provider pattern, command/event bus, repository pattern)
- Anything non-obvious that would trip up a new developer
## OVERVIEW.md format
Write the file to the project root. Use this structure, but adapt section depth and detail to what's actually present — don't include empty sections:
```markdown
# [App/Project Name] — Overview
> One-sentence description of what this app does and who uses it.
## Purpose
2–4 sentences on the domain, user-facing purpose, and any important context
(e.g. "phase 0 of a migration from Preact to React").
## Tech Stack
| Layer | Technology |
|-------|-----------|
| ... | ... |
## Directory Structure
Brief annotated tree of the top 2–3 levels. Only include directories and files
that are meaningful — skip `node_modules`, lockfiles, build output, etc.
## Architecture
Key architectural patterns, data flow, and anything non-obvious. This section
is where you explain the *how* rather than just listing what exists.
## Integrations
For each external service or API: what it is, what it's used for, and where
in the codebase it appears.
## Database & Data Layer
ORM/library, database type, schema location, migration approach, connection config.
If there's no database, say so (e.g. "Frontend-only — no database layer").
## Connectivity & Configuration
Expected environment variables, API proxy setup, service endpoints, ports.
Use a table or list with variable name + purpose.
## Key Entry Points
The files a new developer should read first to understand how the app boots
and how requests/events flow through it.
## Notes & Gotchas
Anything that would surprise a new developer: non-standard patterns, in-progress
migrations, known tech debt worth knowing about, Preact internals being used, etc.
```
## Quality bar
- Be specific. "Uses Postgres via Drizzle ORM, schema defined in `packages/db/schema.ts`" is better than "uses a database."
- If something is unclear (e.g. you can see a dependency but can't find where it's used), say so briefly rather than omitting it.
- Keep the file readable — a developer should be able to scan it in 5 minutes.
- Don't reproduce large code blocks; reference file paths instead.
- After writing the file, confirm to the user what was created and where.

217
sources/260903_instructions.md
View File

@@ -32,29 +32,8 @@ monitored. Abusing model access will be subject to penalties, including but not
*“*👋 ***New to the project?*** *Start with a* [*2-minute overview of what we're doing*](https://images-for-tasks.s3.amazonaws.com/eec61925-34b8-486c-bf6b-1ee417b2004a/theProject/theProject-v2.pdf)*, put together by J.D Nichols.”* *“*👋 ***New to the project?*** *Start with a* [*2-minute overview of what we're doing*](https://images-for-tasks.s3.amazonaws.com/eec61925-34b8-486c-bf6b-1ee417b2004a/theProject/theProject-v2.pdf)*, put together by J.D Nichols.”*
**Quick Navigation:**
[Confidentiality](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aconfidentiality) **|** [Start Here](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Astart_here_ref) **|** [Toolkit](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atoolkit_ref) **|** [Repositories](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Arepo_context_ref) **|** [Agents](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Achoosing_agent_ref) **|** [Meaningful Failure](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ameaningful_failure_ref) **|** [Snapshots](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Asnapshot_ref) **|** [Workspace](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aworkspace_patch_ref) **|** [instruction.md](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Awriting_instruction_ref) **|**
[Grading](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Ahow_grading_works_ref) **|** [Holistic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Agrader_guidance_ref) **|** [Detectors](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Adetectors_ref) **|** [Reference Runs](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Areference_runs_ref) **|** [Atomic Rubric](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aatomic_rubric_ref) **|** [Regrade](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aregrade_sanity_ref) **|** [Submitting & Feedback](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Afeedback_loop_ref) **|** [Versions &](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref)
[Migration](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Amigration_ref) **|** [Examples](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Aexamples_ref) **|** [Troubleshooting](https://app.example.tech/workers/projects/427d-9324-758a6a6ee01f/instructions_popout?jumpTo=bookmark%3Atroubleshooting_ref)
**Quick Links****:**
[Submission Overview](https://app.example.tech/projects/a87d6c37-964a-4e44-89b6-8f00213b0dfd)**:** View your current standing, reported-hours progress, review-queue outlook, and submission
history.
[**Grading Standard**](https://app.example.tech/publish/s/6d6183f7-e335-4e8a-94d2-ece2d1d5ab86.html)
[Workflows](https://app.example.tech/publish/g/Y2IYEXllIxVDSl87an9wPgcnMyVVL3pkOwAxUg4ABAgnEmxSfQAkcwQPCSE=)
[FAQ](https://docs.google.com/document/d/NwUMvGLxSc6OTgkK6fAexQQhcUySD-I/edit?tab=t.27m5l6dtp5q)
[Searching for model failures: hardness ladder](https://app.example.tech/publish/g/Y31AWnZyL2RWZhcRdBhDCS5EKRIvDylgFW5UCD1OF1QwLVNgDRMuMAs0A0w=)
[~20min video of Nick talking about tasks that he likes](https://app.example.tech/publish/s/b6f62716-30d3-4dca-95e4-d4aa67e9f799.mp4)
[Learning Exercise refresher](https://app.example.tech/publish/s/541aba88-70ad-4d51-867a-1fbb78a89d32.html)
Task Catalog ([JSON](https://app.example.tech/publish/s/95301d04-ac07-4bd3-be80-5ee468c74fd4.json)) ([Viewer](https://app.example.tech/publish/s/4cee7539-070f-41de-98c8-e09377c2023f.html))
# 🗞️ **Recent Changes** # 🗞️ **Recent Changes**
Dated project updates, newest first. Each entry points you to the section with the full details. Dated project updates, newest first. Each entry points you to the section with the full details.
@@ -400,202 +379,6 @@ the codebase with the tools for building tasks in it. The **Your Toolkit** secti
interesting tasks. A domain you know well beats guessing in one you do not. interesting tasks. A domain you know well beats guessing in one you do not.
The Setup page asks which repository you chose. Reviewers use your answer to route your task. The Setup page asks which repository you chose. Reviewers use your answer to route your task.
## **ZenBill (ZenBill-006)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Zenbill-Onboarding.pdf)
A **B2B payment and invoice platform** built on Rails 7 + React 18. Businesses use ZenBill to send and receive
money via ACH transfers and credit cards, manage invoices, and sync with QuickBooks Online.
| **Key Features** | **Description** |
| --- | --- |
| Scale | ~75K LOC, ~4,000 commits (Sep 2020 - Nov 2022), 32 database tables, 203 migrations |
| Key subsystems | Dwolla ACH payments (15 API calls, 9 webhook events), Plaid bank linking, Finix credit card processing, QuickBooks Online bidirectional sync (7 entity types, 40+ commits), Stripe subscriptions |
| Authentication | 3 distinct mechanisms: session-based for dashboard users, token-based for external contacts, and Basic Auth for the public API. Authorization is initialized from `UsersOrganization`, not `User`. |
Architecture
65 `ActiveInteraction` classes encapsulating business logic, 7 AASM state machines, 100+ Jbuilder templates, subdomain routing across 6 subdomains, polymorphic funding sources (4 types)
## **Palolo (Palolo-031)**
🔒 **Private**
📺 [2-min codebase tour](https://images-for-tasks.s3.amazonaws.com/89b02a42-f308-4c6d-aeda-429d79cced60/theProject/Palolo-Onboarding.pdf)
An **employee financial wellness platform** built as a TypeScript monorepo (pnpm, 9 packages). Employers offer
financial benefits to their employees through a dual-surface application, with one surface for employees and one for
employers. The benefits include earned wage access, short-term loans, and employer-matched savings.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~172K LOC of TypeScript, ~45 Prisma models, 15+ external service providers |
| Products | Earned Wage Access, short-term loans with underwriting, employer-matched savings with vesting, payroll integration via Atomic/Finch/Argyle |
| Architecture | Express API with SQS background jobs, dual-surface app (consumer banking for employees + HQ admin for employers), MFA auth state machine ( `Unauthenticated → AwaitingOtp →` `AwaitingPin → Authenticated` ), multi-provider BaaS abstraction layer with mock providers for development |
## **zeta-platform**
🔒 **Private**
A large legacy-Ruby banking monorepo. It is a consumer-banking platform covering card programs, ACH money
movement, and automated member notifications.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~11,400 commits, 3,463 files; ~1,700 RSpec examples across models/controllers/GraphQL/services/jobs/queries |
| Domain | Consumer banking: card issuing & decline logic, ACH risk scoring, virtual-card issuance, automation notifications |
| Stack | Rails 5.1 / Ruby 2.6.6 (EOL, era-matched) · PostgreSQL + Redis (Sidekiq) · sprockets asset pipeline · in-repo React frontend (the API + specs run without it) |
| Integrations | Stripe, Plaid, Twilio, and Slack, all lazy and ENV-gated; the app boots and runs the suite with blank placeholder keys |
| Reference data | Supplementary data corpus mounted at `/data/zeta-corpus`; see the zeta reference corpus notes below |
## **zeta-polyglot**
🔒 **Private**
A single toolkit bundling 38 repositories from the wider zeta ecosystem behind one Explore container, the container
you use to explore the codebase. Each bundled repository is called a member. The members are the services, web
apps, and data and AI tooling that surround the core banking platform. Pick a member to run with `run-app <repo>`.
Each member's setup is deferred to its first use. Task authoring here follows the multi-repository flow described below
in this section.
**Key** **Features**
**Description**
| Scale | 38 member repos in one image: 11 Ruby/Rails services, 8 Node/React web apps, 9 Python AI/ML/data projects, and 10 read-only repos (docs, infra, coding challenges) |
| --- | --- |
| Domain | The broader consumer-banking ecosystem: money-movement & webhook services, card/back-office services, customer-facing web & content sites, chatbot/agent tooling, and transaction-anomaly & prediction ML |
| Stack | One Explore image carrying every member's runtime (rbenv Ruby 3.1/3.2 · nvm Node 14/16/18/19 · pyenv Python 3.10) · PostgreSQL + Redis · per-member frameworks (Rails, React/Next/Gatsby, Flask/FastAPI, dbt) |
| Integrations | Per-member, all lazy and ENV-gated; each boots and runs its suite with blank placeholder keys |
| Reference data | Supplementary data corpus mounted at `/data/zeta-corpus`; see the zeta reference corpus notes below |
## **Breezy (breezy-complete)**
🔒 **Private**
An **AI phone-receptionist platform** for home-service professionals. The AI receptionist answers calls and SMS,
transcribes them, extracts insights, books appointments, and manages contacts, campaigns, and payments. The
codebase is a monorepo with a Rails 7 API in `backend/` and a Next.js 14 frontend in `frontend/`.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | ~10,000 commits (2016–2026), 254 database tables, 868 migrations; ~170K LOC of Ruby + ~300K LOC of TypeScript/JS |
| Key subsystems | Inbound/outbound call handling with transcripts, contact threads (calls/SMS/email), AI notes & insights, appointment scheduling with a native calendar, structured AI-prompt configuration (FAQs/intents), Stripe subscriptions/billing, website builder |
| Stack | Ruby 3.2.0 / Rails 7.0 (Bullet Train–derived) · Puma + Sidekiq · Next.js 14 / React 18 (Node 22) · PostgreSQL 14 + Redis 6.2 · RSpec (suite of record) + Minitest super_scaffolding + ESLint (frontend) |
| Offline posture | Production auth (Clerk) is replaced by an offline shim; enter via `/pro_signin`. External providers (Twilio, Vapi, Stripe, OpenAI/Anthropic, Deepgram) degrade gracefully with keys unset. |
## **Potion (potion-polyglot)**
🔒 **Private**
An **AI personalised-video platform for sales and marketing outreach**. Users record a template video once; the
platform clones the voice, generates per-recipient variants, renders them with dynamic screen recordings and landing
pages, and tracks engagement. This is a **52-repo estate**, containing a Nuxt 2 + Express flagship ( `potion-app` )
surrounded by the lambdas, GPU inference services, video-processing workers and infrastructure it depends on.
| **Key** **Features** | **Description** |
| --- | --- |
| Scale | 52 repos in one toolkit; flagship `potion-app` ~8,900 commits (2021–2026), 30 MongoDB models across 36 schemas, ~167K LOC application code (~92K Vue + ~75K JS) |
| Estate shape | 25 Node members (14->20, each pinned to its own dependency era), 17 Python (ML/inference: voice cloning, wav2lip, MODNet, sentence-split), 10 no-runtime infra/Terraform repos |
Key
subsystems
Video generation & rendering pipeline (ffmpeg workers, job producer/consumer, watchers),
voice cloning (ElevenLabs + in-house training repos), dynamic screen recording lambdas,
custom-domain landing pages, website builder, Stripe billing, AppSumo redemption, GCP/AWS
media storage
| Stack | Node 16 · Nuxt 2.14 / Vue · Express 4.16 · Mongoose 6.8 / MongoDB · socket.io 4.7 · Jest (suite of record: 13 suites, 27 tests) · sibling members on Python 3.8–3.11 |
| --- | --- |
| Offline posture | Own JWT auth with a seeded verified user ( `dev@example.com` ); enter at `/auth/login`. Local MongoDB. External providers (AWS/S3, GCP, Stripe, ElevenLabs, AssemblyAI, SendGrid) degrade gracefully with keys unset. |
## **human-essentials**
🌐 **Open source**
Inventory management for **diaper banks & essentials banks** serving 200+ non-profits. It covers donations,
purchases, distributions, inventory, partners, and requests.
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Multi-tenant inventory & distribution management for essentials banks |
| Stack | Rails 8.0 / Ruby 3.4.3 · PostgreSQL · importmap (no Node build for the app) |
| Tests | RSpec + Capybara + **Cuprite** (headless Chrome); external HTTP stubbed via WebMock |
| Notable | Multi-tenant (everything scoped to `Organization` ); **event-sourced inventory** ( `Event` STI + `InventoryAggregate` ); business logic in `app/services/` |
## **casa**
🌐 **Open source**
Case management for **Court Appointed Special Advocates** (every CASA in Maryland, plus WA/MO/KS).
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Volunteer & case management for court-appointed child advocates |
| Stack | Rails 8.0 / Ruby 4.0.3 · PostgreSQL · jsbundling (esbuild) + sass (Node 24) · imagemagick |
| Tests | RSpec, with ~3,580 examples across ~452 files; system specs via Selenium headless Chrome |
| Notable | Largest and most integrated of the six: cases, contacts, court dates, reports; heavy system-spec coverage |
## **awbw**
🌐 **Open source**
**A Window Between Worlds** is an art-program platform helping 140k+ people per year through trauma-recovery
workshops.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Art-program, workshop, and site management for a national non-profit network |
| Stack | Rails 8.1 / Ruby 4.0.1 · **MySQL 8** (Trilogy adapter) · Vite (Node 22) |
| Tests | RSpec, with 328 spec files (21 system); system specs via Selenium headless Chrome |
| Notable | The only MySQL repo; Stripe/Pay payments + Geocoder (stubbed in tests); JSON columns |
## **stocks-in-the-future**
🌐 **Open source**
**Stocks in the Future** is a financial-literacy app teaching students across ~20 Baltimore schools via simulated
portfolios.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Classroom financial-literacy platform (students, teachers, portfolios, stocks) |
| Stack | Rails 8.1 / Ruby 3.4.4 · PostgreSQL + Redis (background jobs) · importmap (no Node build) |
| Tests | **Minitest**, with ~733 tests across 87 files; system specs via Selenium headless Chrome |
| Notable | Postgres + Redis; classroom/teacher/student domain; Minitest rather than RSpec |
## **community-foundation**
🌐 **Open source**
**Community Foundation** helps community foundations plan and allocate funds.
| **Key** **Features** | **Description** |
| --- | --- |
| Purpose | Fund planning & allocation for community foundations |
| Stack | Rails 8.1 / Ruby 4.0.2 · **SQLite** (no DB service) · importmap + Tailwind (no Node for the app) |
| Tests | Minitest; system specs via Selenium headless Chrome |
| Notable | Lightweight (SQLite, no external services); encrypted credentials; CI enforces a 90% coverage gate |
## **endsideout**
🌐 **Open source**
**End Side Out** supports student programs in Baltimore and Monrovia, Liberia, serving 6,000+ students.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Program management for a sports-and-education non-profit |
| Stack | Rails 8.1 / Ruby 4.0.0 · **SQLite** (no DB service) · importmap + Tailwind (no Node for the app) |
| Tests | Minitest, with ~106 runs; system specs via Selenium headless **Firefox** (+ axe accessibility) |
## **flaredown**
🌐 **Open source**
**Flaredown** is a symptom tracker for people with chronic illness. Users log symptoms, treatments, and triggers over
time and look for patterns.
| **Key Features** | **Description** |
| --- | --- |
| Purpose | Track symptoms, treatments, and triggers for chronic illnesses |
| Stack | Ruby 3.2.3 / Rails 7.1 API with Mongoid on MongoDB 7.0 as the primary store, plus PostgreSQL for a small relational slice, Redis and Sidekiq · Ember client on Node 14, served with a proxy to the API |
| Tests | RSpec, with 315 examples run from `backend/` (96% coverage); Ember client suite via `ember test`, with 452 tests on headless Chrome. Browser/acceptance specs are excluded from the verifier |

182
sources/OVERVIEW.md Normal file
View File

@@ -0,0 +1,182 @@
# Potion Voice — Overview
> An asynchronous voice-cloning and text-to-speech service for Potion's personalized-video pipeline, combining Node.js queue workers with a GPU-oriented Coqui VITS training and inference toolkit.
## Purpose
Potion Voice has no HTTP server or user interface. It provides two continuously running workers: one fine-tunes a per-user voice model from uploaded recordings, and one uses that model to synthesize a personalized greeting and enqueue downstream video-compositing work. The repository also contains Python command-line tools for preparing speech datasets, training the shared multi-speaker baseline, cloning and minimizing individual voices, synthesizing speech, and scoring model or salutation quality.
## Tech Stack
| Layer | Technology |
| --- | --- |
| Worker runtime | Node.js, CommonJS modules; no Node version is declared |
| Process management | PM2, one process per worker |
| ML runtime | Python 3 (the guide targets 3.10), PyTorch, Coqui TTS/Trainer |
| Speech model | VITS with 512-dimensional speaker d-vectors; 22,050 Hz training/inference output |
| Audio processing | Coqui resampling/embedding tools, `ffmpeg` for 48 kHz output, `espeak-ng` as the documented phoneme backend |
| Database | MongoDB through Mongoose 6.x |
| Queue and object storage | AWS SDK v2, SQS, S3, CloudFront-hosted source audio |
| Compute and filesystem | GPU-backed EC2 is the documented target; trained assets and logs are placed on an EFS mount |
| Monitoring | Bugsnag for worker exceptions; TensorBoard/TensorBoardX for training runs |
| Evaluation | Resemblyzer speaker similarity, `textdistance`, and Potion's internal transcription API |
| Tests | No automated test framework, test files, lint command, or CI configuration is present |
Python dependency sets are split across `requirements*.txt`: development pins PyTorch 1.12.1/CUDA 11.6, the legacy/default set pins PyTorch 1.9.1/CUDA 11.1, production has separate CPU and unpinned-GPU variants, and local development leaves PyTorch unpinned. Every set also installs a private `potion-voice-utils` Git dependency, although this checkout has no direct import from it.
## Directory Structure
```text
.
├── app/services/ Shared Node.js helpers
│ ├── s3/ S3 upload/download wrapper
│ ├── sqs/ SQS receive/delete/send wrapper
│ ├── utils/ Error serialization, Bugsnag helper, file deletion
│ └── voice_cloning/ Older duplicate VoiceCloning model/service
├── voice-cloning-job-handler/ Per-user model-training worker
│ ├── index.js Queue loop and end-to-end orchestration
│ ├── user_audio_profile/ Mongoose schema and CRUD service
│ ├── voice_cloning/ Mongoose schema and CRUD service
│ └── pm2-{development,production}.yml
├── voice-synthsizer-job-handler/ Greeting-synthesis worker (directory typo is historical)
│ ├── index.js Queue loop, synthesis, upload, downstream job creation
│ ├── job/ Downstream AI job schema/service
│ ├── recording/ Large shared Recording schema
│ ├── recording_salutation/ Dynamic-video salutation schema
│ ├── salutation/ Reusable generated-salutation schema/service
│ ├── user_audio_profile/ Duplicate profile schema/service
│ └── pm2-{development,production}.yml
├── voice-cloning/ Python ML and audio toolkit
│ ├── assets/ Speaker encoder and World Gender Name Dictionary data
│ ├── docs/ EC2 setup and command examples
│ ├── utils/ Synthesis, similarity, name matching, transcription helpers
│ ├── prepare_datasets.py Archive extraction, resampling, d-vector generation
│ ├── train_multispeaker_baseline_model.py
│ ├── clone_voice.py Fine-tunes the baseline for one speaker
│ ├── minimize_cloned_voice_model.py Removes training-only model state
│ ├── synthesize_speech.py Generates and resamples a WAV
│ └── score_*.py Manual model/salutation evaluation tools
├── requirements*.txt Python environment variants
├── package.json Shared/root Node dependencies
└── README.md One-line project description
```
This is not configured as an npm workspace. There are three package manifests with largely duplicated dependencies; the worker code resolves shared modules and, depending on installation layout, dependencies from the repository root.
## Architecture
### Queue contracts
| Worker | Expected SQS message body |
| --- | --- |
| Voice cloning | JSON with `job._doc._id`, `job._doc.userAudioProfileId`, `job._doc.metadata.directoryName`, `job._doc.input[]`, and top-level `job.env`. Each input item contains `waveUrl` and `originalText`. |
| Synthesis | JSON with `userAudioProfileId`, `text`, `firstName`, `salutationId`, `recordingId`, `baseUrlForPotionAi`, and `env`. |
In both workers, the message's `env` selects the Mongo URI and environment-specific storage resources. This is separate from the process-level environment used to configure PM2 and Bugsnag.
### Voice-cloning flow
1. `voice-cloning-job-handler/index.js` short-polls one message from the configured SQS FIFO queue and immediately deletes it.
2. It selects a MongoDB connection and CloudFront base URL from the message environment, then marks both the `VoiceCloning` and `UserAudioProfile` documents as `processing`.
3. It rewrites each recording URL's host to the selected CloudFront host, downloads WAV files over HTTPS, and writes a VCTK-style dataset under `/tmp/<directoryName>/{wav48,txt}/1/`. Files are numbered `1_001`, `1_002`, and so on.
4. It archives the dataset and invokes three Python programs as child processes:
- `prepare_datasets.py` computes speaker embeddings at 16 kHz, then restores and resamples the training audio to 22,050 Hz.
- `clone_voice.py` fine-tunes the hard-coded `pretrained-models/checkpoint_365000.pth` VITS baseline. Defaults are batch size 96, 200 epochs, mixed precision, two evaluation samples, and checkpoints every 200 steps.
- `minimize_cloned_voice_model.py` reloads `checkpoint_365200.pth`, drops the discriminator and optimizer state, and creates `_light.pth` plus `config_light.json` inference assets.
5. Generated datasets, checkpoints, configs, embeddings, and command logs live under `/mnt/efs/potion-voice/<env>/<directoryName>/`. Mongo status moves to `completed`, and `UserAudioProfile.training_model_path` records five local paths (full/light model, full/light config, and speaker embeddings).
6. The same five files are uploaded through S3 and their returned locations are stored in `training_model_s3_path`. The code constructs the bucket argument as `potion-voice-users-training-model/<env>` and object keys as `<directoryName>/<basename>`.
An exception after Mongo connects marks both records `error` and reports to Bugsnag. There is no compensating queue retry because receipt deletion happens before processing.
### Greeting-synthesis flow
1. `voice-synthsizer-job-handler/index.js` receives and immediately deletes one SQS message, connects to the Mongo database selected by `job.env`, and finds a completed `UserAudioProfile`.
2. It reads the **local EFS paths** from `training_model_path`; `training_model_s3_path` is not used for inference. `synthesize_speech.py` loads the light VITS model and the profile's single-speaker embeddings, writes a native-rate WAV, and runs `ffmpeg` to create the default 48,000 Hz WAV.
3. The resampled file is uploaded to bucket `recordings-<env>` with a generated key ending in `_salutation_<firstName>.wav`.
4. The worker upserts a reusable `Salutations` record keyed by user, audio profile, and first name; updates the requested `recording_salutations` record; and loads the associated `Recordings` document.
5. It inserts a new `Job` (default type `ai-job`) containing the original video/greeting, crop timestamp, synthesized greeting URL, request origin, environment, recording IDs, and dynamic-video type. Another service is expected to consume this Mongo-backed job and composite the final personalized video.
Both workers run serially in an infinite loop. Empty polls sleep for two seconds; active queues are processed without that delay. They open and close Mongoose around each message rather than maintaining a process-wide connection.
### Python toolkit
The Python scripts are also usable independently from `voice-cloning/`:
- Baseline training combines VCTK 0.92, LibriTTS train-clean-360, and Potion salutation recordings into a multi-speaker VITS model. The checked-in configuration targets 22,050 Hz audio and 512-dimensional d-vectors. The guide estimates 5–7 days for 100 epochs on an AWS `g5.2xlarge`.
- Per-user cloning expects matching transcripts and recordings in `txt/1/` and `wav48/1/`; the guide recommends 30 samples and says a default clone takes about one hour on `g5.2xlarge`.
- `score_cloned_voice.py` and `score_models.py` synthesize fixed sentences and compare Resemblyzer embeddings against real recordings; the latter ranks checkpoint files and reports a top five.
- `score_salutation.py` transcribes a WAV, extracts candidate names, validates them against the included World Gender Name Dictionary, and combines transcription confidence with Jaro-Winkler, Levenshtein, and Match Rating Approach similarity.
## Integrations
| Integration | Use and code location |
| --- | --- |
| AWS SQS (`us-west-2`) | Environment-specific FIFO queues feed both workers. Shared wrappers are in `app/services/sqs/`; queue URLs are supplied by PM2 configuration. |
| AWS S3 | `app/services/s3/index.js` uploads trained model assets and synthesized greetings. AWS credentials are not explicit variables; the AWS SDK's normal credential chain is assumed. |
| CloudFront/HTTPS | The cloning worker replaces the host of every supplied `waveUrl` with an environment-specific CloudFront base and downloads it using Node's `https` module. |
| Amazon EFS | `/mnt/efs/potion-voice/<env>/<directoryName>` is the durable model/data/log location and the coupling point between training and synthesis. |
| MongoDB | MongoDB Atlas-style `mongodb+srv://...` URIs are selected per message environment. Models represent cloning jobs, profiles, greetings, recordings, and downstream jobs. |
| Bugsnag | Both worker entry points initialize Bugsnag with package version, app environment, backend key, and Node release stage. |
| Coqui TTS/Trainer | VITS training and inference implementation. The install guide requires a separate editable checkout of Coqui TTS v0.10.2 under ignored `voice-cloning/TTS/`. |
| Potion transcription API | `voice-cloning/utils/transcription_utils.py` posts a WAV with a bearer token, then optionally polls for up to 60 seconds. It is used only by the salutation-scoring CLI. Commented examples point at `/api/transcript` on development and staging Potion hosts. |
| Dataset sources | Baseline-training instructions retrieve VCTK, LibriTTS, and Potion salutation archives from the private `potion-datasets` S3 bucket. |
## Database & Data Layer
Mongoose schemas are defined beside each worker; there is no separate schema package, migration system, repository abstraction, or declared indexes. Most service modules are higher-order factories that bind a Mongoose model and expose basic CRUD methods. Reads commonly add `deleted: false`, while removes are soft deletes.
| Model | Role and notable fields |
| --- | --- |
| `VoiceCloning` | Tracks `userId`, `userAudioProfileId`, `status`, raw `input`, `training_model`, `metadata`, and `deleted`. |
| `UserAudioProfile` | Tracks profile `name`, clone `status`, local `training_model_path`, S3 `training_model_s3_path`, and soft deletion. Its schema/service is duplicated in both workers. |
| `Salutations` | Caches synthesized audio by `userId`, `userAudioProfileId`, and `firstName`; stores the S3 URL in the historically named `salutationVideo` field. |
| `recording_salutations` | Connects a generated greeting to master/dynamic recordings and tracks processing state and derived media URLs. |
| `Recordings` | A broad schema shared with the video product. This worker mainly reads original/master video URLs, crop timestamp, user, and dynamic-video type. |
| `Job` | Creates the downstream `ai-job` record with recording/user/salutation IDs and a mixed `metadata` payload. |
All schemas enable timestamps. Several cross-service payloads and model-asset maps use `Schema.Types.Mixed`, so MongoDB does not enforce their internal shape.
## Connectivity & Configuration
The PM2 YAML files are the only environment templates. In this checkout sensitive values are redacted; production values should remain secret rather than being committed.
| Variable | Purpose |
| --- | --- |
| `SQS_URL` | Queue consumed by the current worker. Checked-in examples use environment-specific FIFO queues in `us-west-2`. |
| `MONGODB_URI_DEV`, `MONGODB_URI_STAGING`, `MONGODB_URI_PROD` | MongoDB URI selected from the **message's** `env`. Not every PM2 file supplies all three. |
| `POTION_APP_ENV` | Used by worker code in the Bugsnag app-version string and by the shared Bugsnag helper. |
| `NODE_ENV` | Bugsnag `releaseStage`; PM2 sets it to `production` even in the synthesis development config. |
| `BUGSNAG_BACKEND_KEY` | Bugsnag API key. |
| `CLOUDFRONT_URL_DEV`, `CLOUDFRONT_URL_STAGING`, `CLOUDFRONT_URL_PROD` | Cloning worker's replacement host for input WAV downloads. |
| `APP_ENV` | Present in synthesis PM2 files, but the JavaScript reads `POTION_APP_ENV` instead. |
| `TRANSCRIPTION_API_ENDPOINT`, `TRANSCRIPTION_API_TOKEN` | Required only by `score_salutation.py`; token is sent as bearer authentication. |
There is no listening application port. TensorBoard is optional and documented on port 6006. Runtime AWS access relies on SDK/CLI credentials or an instance role. Shell tools include `python3`, `tar`, `ffmpeg`, and, for setup, `git`, `unzip`, and `aws`.
## Key Entry Points
1. `voice-cloning-job-handler/index.js` — complete training-worker control flow and its SQS message shape.
2. `voice-synthsizer-job-handler/index.js` — inference worker and handoff to the video job pipeline.
3. `voice-cloning/prepare_datasets.py` — exact input archive layout, sampling conversion, and embedding generation.
4. `voice-cloning/clone_voice.py` — per-speaker VITS fine-tuning configuration.
5. `voice-cloning/synthesize_speech.py` and `voice-cloning/utils/synthesize_utils.py` — inference and 48 kHz WAV production.
6. `voice-cloning/train_multispeaker_baseline_model.py` plus `train_config.py` — shared baseline datasets and model hyperparameters.
7. `voice-cloning/docs/potion-voice-cloning_Installation_Guide.md` — machine sizing, CUDA/system packages, dataset setup, and CLI examples.
8. `app/services/sqs/sqs_service.js` and `app/services/s3/index.js` — shared cloud I/O behavior.
## Notes & Gotchas
- A clean clone is not runnable end to end. `voice-cloning/TTS/`, `voice-cloning/pretrained-models/`, generated results, and deployment `app-scripts/` referenced by npm scripts are absent/ignored. The training worker specifically assumes `checkpoint_365000.pth`, then assumes cloning creates `checkpoint_365200.pth` in a directory whose name contains `vits_potion_clone`.
- Queue delivery is effectively **at most once**: both workers delete an SQS message before Mongo access, Python execution, or S3 upload. A crash or processing error cannot be retried from that receipt, and no dead-letter handling appears here.
- Inference reads EFS-local paths from Mongo, not the uploaded S3 asset map. Training and synthesis hosts therefore need the same `/mnt/efs/potion-voice` mount and path layout.
- Training uploads pass `potion-voice-users-training-model/<env>` as the S3 `Bucket` value. Standard S3 bucket names cannot contain `/`; verify whether the environment was intended as a key prefix before relying on this path.
- Several commands are assembled as shell strings from message values (`directoryName`, paths, and especially `text`). Quotes or shell metacharacters can break execution and untrusted input would create command-injection risk.
- Child-process paths are relative to the worker's current directory (`../voice-cloning/...`), while some Python assets are also opened by relative path. Starting PM2 from a different working directory can therefore break script, encoder, or checkpoint discovery.
- Temporary data is only partially cleaned: training archives/extracted files remain under `/tmp`, and synthesis removes the selected 48 kHz file but leaves the original WAV and UUID directory.
- Mongo connection retries recursively call `connectDB` without settling the original promise; after an initial connection failure a worker can remain stuck. The selected full Mongo URI is also printed to logs.
- `UserAudioProfile.find()` returns an array, but the synthesis worker tests only whether the array is truthy before dereferencing element zero. An empty result follows the exception path rather than the intended “model not found” branch.
- PM2 configuration and code use inconsistent environment names (`APP_ENV` versus `POTION_APP_ENV`); the synthesis development file also targets a staging queue while labeling `APP_ENV` as development. The cloning staging CloudFront value is blank in the checked-in example.
- Dataset configuration has drift: `train_config.py` overwrites the `POTION_SALUT_*` constants with voice-cloning values, `prepare_datasets.py` advertises a `DAPS` preset but does not implement its branch, and the guide shows some argument values that no longer match argparse choices.
- The root manifest declares `index.js` as its main file, but no root `index.js` exists. Worker deployment scripts reference an absent `app-scripts/` tree, and there is no standard `start` or `test` script.
- Shared/duplicated code has stale paths: `app/services/voice_cloning/` duplicates the handler implementation, the shared Bugsnag and delete-file utilities are not used by the worker entry points, and `fetchS3Object()` references an undefined `stringifyObj` logger if called.
- The install guide pins Coqui TTS v0.10.2 while the Python requirement variants and CUDA guidance span multiple PyTorch/CUDA combinations. Reproduce the intended image deliberately; do not assume the latest packages are compatible.

2
tools

Submodule tools updated: 2f41548859...0ca98f2b71