convert pdf to md, add OVERVIEW and docs

worker-toolkit-stocks-in-the-future/repo/OVERVIEW.md

@@ -0,0 +1,197 @@
+# Stocks in the Future — Overview
+
+> A Rails 8 web app used by middle-school students, teachers, and admins to run a financial-literacy program: students earn "SIF dollars" from grades and attendance, then buy and sell real-ticker stocks in a simulated portfolio.
+
+## Purpose
+
+[Stocks in the Future](https://sifonline.org/) (SIF) pairs classroom incentives with an investing curriculum. Students are rewarded with virtual cash for attendance and for math/reading grades, and they invest that cash in a simulated brokerage backed by real daily stock prices from Alpha Vantage. Teachers manage classrooms, enter quarterly grade books, and finalize earnings; admins manage schools, school years, stocks, users, and manual portfolio adjustments.
+
+This is a [Ruby for Good](https://rubyforgood.org/) volunteer project (`rubyforgood/stocks-in-the-future`). It is a server-rendered Rails monolith — Hotwire/Turbo with a sprinkle of Stimulus, no SPA front end.
+
+## Tech Stack
+
+| Layer | Technology |
+|-------|-----------|
+| Language / runtime | Ruby 3.4.4 (`.ruby-version`) |
+| Framework | Rails 8.1.2 (`config.load_defaults 8.0`) |
+| Database | PostgreSQL 15 via `pg ~> 1.6` |
+| Web server | Puma (`config/puma.rb`), nginx + unix socket in prod |
+| Background jobs | Solid Queue 1.4 (DB-backed), `config/queue.yml` + `config/recurring.yml` |
+| Auth | Devise 5.0 — **login is by `username`, not email** |
+| Authorization | Pundit 2.5 (`app/policies/`) |
+| Soft deletes | `discard ~> 2.0` on `User` |
+| Assets | Propshaft + importmap-rails (no JS bundler), Tailwind via `tailwindcss-rails` |
+| UI components | `shadcn-ui` gem (+ `tailwind_merge`), `lucide-rails` icons, `font-awesome-rails` |
+| Front end | Hotwire (Turbo + Stimulus), Trix/Action Text, Chart.js 4.5 (CDN-pinned) |
+| Rich text / files | Action Text, Active Storage |
+| Tests | Minitest + FactoryBot, Capybara + Selenium (system), WebMock, Mocha, SimpleCov |
+| Lint / security | RuboCop (+ rubocop-rails), erb_lint, i18n-tasks, Brakeman, bundler-audit |
+| Migrations safety | `strong_migrations ~> 2.8` |
+| Deploy | Capistrano 3 → AWS Lightsail (Ubuntu), Terraform for infra |
+
+## Directory Structure
+
+```
+app/
+  controllers/          # student/teacher-facing controllers
+    admin/              # /admin namespace, all inherit Admin::BaseController
+    concerns/           # SoftDeletableFiltering (discarded/all/kept param scoping)
+  models/               # 24 models; User STI -> Student, Teacher
+    concerns/url_helpers.rb
+  services/             # ExecuteOrder, DistributeEarnings, TransactionFeeProcessor,
+                        # AlphaVantageApiClient, StockAttributeUpdate,
+                        # ImportStudentService, BulkStudentImportService,
+                        # MemorablePasswordGenerator
+  jobs/                 # OrderExecutionJob, StockPricesUpdateJob,
+                        # StockAttributeUpdateJob, MonthlyPortfolioSnapshotJob
+  policies/             # Pundit: application, classroom, grade_book, order, portfolio, stock
+  facades/              # ClassroomFacade (student list + classroom stats)
+  presenters/           # AttendanceEntryPresenter, ClassroomPresenter, SchoolYearPresenter
+  form_builders/admin/  # Admin::FormBuilder
+  components/shadcn/    # Shadcn::FormBuilder
+  helpers/components/   # render_button / render_input / etc. -> app/views/components/ui/*
+  javascript/controllers/ # 8 Stimulus controllers (order form, portfolio chart, modal,
+                          # admin sidebar, autosave, clickable row, filters, navbar toggle)
+  views/                # ERB; layouts/application.html.erb and layouts/admin.html.erb
+  assets/tailwind/      # application.css + admin/buttons/forms/navbar/shadcn/tables partials
+config/
+  routes.rb  application.rb  recurring.yml  queue.yml  storage.yml
+  environments/{development,test,staging,production}.rb
+  deploy.rb  deploy/{production,staging}.rb   # Capistrano
+  initializers/api_keys.rb                    # global API_KEY constant
+db/
+  schema.rb  migrate/  seeds.rb  seeds/{development,staging,production,test}.rb + partials/
+docs/                   # scheduling, orders-and-transactions, gradebook-earnings, seeds, schema
+terraform/{production,staging}/   # Lightsail infra + bootstrap.sh
+test/                   # 87 test files: models, controllers, services, policies, jobs, system
+docker/  Dockerfile  Dockerfile.dev  docker-compose.yml
+```
+
+## Architecture
+
+### Users are STI with a separate admin flag
+
+`User` (table `users`) has `type` in `%w[User Student Teacher]` plus a boolean `admin` column. So there are three effective roles: **student**, **teacher**, and **admin** (`admin` is a flag on any user, not an STI subclass). `Student` auto-creates a `Portfolio` and an initial `ClassroomEnrollment` after create; `Teacher` syncs `username` from `email`.
+
+### Money is a ledger, always in cents
+
+`portfolios` has **no balance column**. Cash on hand is derived in `Portfolio#cash_on_hand_in_cents` as
+`(credits + deposits) - (debits + withdrawals + fees + pending buy orders + pending $1 fee)`.
+`PortfolioTransaction#transaction_type` is `deposit | withdrawal | credit | debit | fee`, where deposit/withdrawal are classroom earnings and admin adjustments and credit/debit are stock sales/purchases. Transactions are meant to be immutable ledger rows (see `docs/orders-and-transactions.md`). All monetary columns are `*_cents` integers (`amount_cents`, `price_cents`, `worth_cents`).
+
+### Order lifecycle (deferred execution)
+
+1. A student creates an `Order` (`buy`/`sell`, whole shares) from a stock page. It is saved `pending` — nothing settles immediately. Validations at this point check trading is enabled for the classroom, the stock isn't archived, sufficient shares to sell, and sufficient funds including a single `$1.00` fee (`PortfolioTransaction::TRANSACTION_FEE_CENTS = 1_00`).
+2. Students may edit or `cancel` pending orders; edits re-run funds validation with a refund of the previous cost.
+3. `OrderExecutionJob` (recurring) calls `ExecuteOrder` for each pending order: creates the debit/credit `PortfolioTransaction`, creates a `PortfolioStock` row (**negative `shares` for sells**), and flips the order to `completed`. If funds/shares are insufficient at execution time the order is canceled instead.
+4. `TransactionFeeProcessor` then charges **one $1.00 fee per user per run**, regardless of order count.
+
+Holdings are therefore an append-only set of `portfolio_stocks` rows; current positions are computed in `PortfolioPosition.for_portfolio` with a grouped SQL query (`HAVING SUM(portfolio_stocks.shares) > 0`) that also derives change and total-return amounts.
+
+### Grade book → earnings
+
+`SchoolYear` auto-creates 4 `Quarter`s on create; `Classroom` auto-creates a `GradeBook` per quarter on create. Teachers fill `GradeEntry` rows (attendance days, perfect-attendance flag, math grade, reading grade). `GradeBooksController#finalize` marks the book `verified!` and runs `DistributeEarnings`, which creates `deposit` transactions and marks the book `completed`. Rates live in `GradeEntry` (cents): `$0.20`/day attended, `$1.00` perfect attendance, `$3.00` for an A-range grade, `$2.00` for a B-range grade, `$2.00` per subject for improving over the previous quarter (via `Quarter#previous`). Statuses: `draft → verified → completed`.
+
+### Classroom membership is mid-migration
+
+There are two membership mechanisms in the codebase at once: the legacy `users.classroom_id` foreign key, and the newer `classroom_enrollments` join table (supports multiple/historical enrollments, one `primary` per student, `enrolled_at`/`unenrolled_at`). `ClassroomFacade#students` unions both. Some scopes (e.g. `Order.for_teacher`, `Classroom.order_by_student_count`) still join only on the legacy `users.classroom_id`.
+
+### Authorization
+
+`ApplicationController` runs `authenticate_user!` for everything, sets `@navbar_stocks = policy_scope(Stock).active`, and rescues `Pundit::NotAuthorizedError` by redirecting students to their portfolio and everyone else to root. `Admin::BaseController` additionally requires `current_user&.admin?` and uses the `admin` layout. Policy scopes are role-shaped, e.g. `OrderPolicy::Scope` resolves to all / teacher's classrooms / own orders.
+
+### Trading gates
+
+`Classroom#trading_enabled` (toggled by `PATCH /classrooms/:id/toggle_trading`) blocks order creation when false; `Classroom#archived` hides classrooms from teachers and blocks grade book access for non-admins. `Stock#archived` blocks new purchases.
+
+## Integrations
+
+| Service | Use | Where |
+|---------|-----|-------|
+| **Alpha Vantage** (`GLOBAL_QUOTE`) | Daily stock price refresh; sleeps 1.1s between symbols to respect the rate limit | `app/services/alpha_vantage_api_client.rb`, `app/jobs/stock_prices_update_job.rb` |
+| **Alpha Vantage** (`OVERVIEW`) | Weekly company metadata (name, description, exchange, industry, website, profit margin) | `app/services/stock_attribute_update.rb`, `app/jobs/stock_attribute_update_job.rb` |
+| **Amazon SES (SMTP)** | Devise password-reset / account-setup mail in staging + production, `us-east-1`, DKIM on `sifonline.org` | `config/environments/production.rb`, `staging.rb` |
+| **AWS Lightsail + SSM** | Hosting (`production_web` = `mi-059a7bcb37754c44d`, `staging_web` = `mi-0c65ce3a1a596c81c`), keyless ops via SSM | `terraform/`, README "Operations" |
+| **AWS Secrets Manager** | Stores SES SMTP creds at `stocks-in-the-future/ses-smtp` | README |
+| **Chart.js (jsDelivr CDN)** | Portfolio value chart from monthly snapshots | `config/importmap.rb`, `portfolio_chart_controller.js` |
+| **GitHub Actions** | CI, lint, auto-deploy to staging, stale-issue cleanup | `.github/workflows/` |
+
+Recurring schedules (`config/recurring.yml`, cron in UTC, app `time_zone` is Eastern in staging/production):
+
+| Job | Schedule |
+|-----|----------|
+| `OrderExecutionJob` | `*/15 * * * *` (every 15 minutes) |
+| `StockPricesUpdateJob` | `0 2 * * 2-6` (Tue–Sat 02:00 UTC ≈ weekday evenings ET) |
+| `StockAttributeUpdateJob` | `0 4 * * 6` (Saturdays) |
+| `MonthlyPortfolioSnapshotJob` | `0 23 L * *` (last day of month) |
+
+## Database & Data Layer
+
+- **Postgres** via Active Record; schema at `db/schema.rb` (version `2026_06_09_141805`), migrations in `db/migrate/`. `strong_migrations` guards unsafe migrations.
+- Connection config: `config/database.yml` (copy from `config/database.yml.sample`). Local dev DBs are `stocks_in_the_future_development` / `_test`; production uses `STOCKS_IN_THE_FUTURE_DATABASE_PASSWORD`, and `DATABASE_URL` overrides everything (Docker uses `postgresql://sif:password@db/`).
+- Core domain tables: `schools → school_years → classrooms` (with `years`, `quarters`, `grades`, `classroom_grades`), `users` (STI) + `classroom_enrollments` + `teacher_classrooms`, `portfolios → portfolio_stocks / portfolio_transactions / portfolio_snapshots`, `stocks`, `orders`, `grade_books → grade_entries`, `announcements`.
+- Solid Queue owns 11 `solid_queue_*` tables in the **same** database (no Redis).
+- Action Text (`action_text_rich_texts`) backs `Announcement#content`; Active Storage tables are present.
+- Notable indexes/constraints: unique `stocks.ticker`, unique `users.username`, **partial** unique index on `users.email` (`WHERE email IS NOT NULL AND email <> ''`) so username-only students can share a null email, unique `(quarter_id, classroom_id)` on grade books, unique `(portfolio_id, date)` on snapshots, partial unique-ish index on primary enrollments.
+- Seeds are environment-split: `db/seeds.rb` loads `db/seeds/#{Rails.env}.rb`, which loads ordered partials from `db/seeds/partials/`. After `bin/rails db:setup` you get logins `Teacher` / `Student` / `Admin`, all with password `password`.
+
+## Connectivity & Configuration
+
+| Variable | Purpose |
+|----------|---------|
+| `DATABASE_URL` | Full Postgres URL; used by Docker and CI |
+| `STOCKS_IN_THE_FUTURE_DATABASE_PASSWORD` | Production DB password when not using `DATABASE_URL` |
+| `RAILS_MAX_THREADS` | Puma threads / AR pool size |
+| `WEB_CONCURRENCY`, `PORT`, `PIDFILE`, `PUMA_SOCKET` | Puma process/binding config |
+| `SOLID_QUEUE_IN_PUMA` | If set, runs Solid Queue as a Puma plugin instead of a separate process |
+| `JOB_CONCURRENCY` | Solid Queue worker processes (default 1) |
+| `ALPHA_VANTAGE_API_KEY` | Stock price/overview API key |
+| `APP_HOST` | Mailer host (`app.sifonline.org` / `staging.sifonline.org`) |
+| `MAILER_SENDER` | Devise sender, default `no-reply@sifonline.org` |
+| `SES_SMTP_USERNAME`, `SES_SMTP_PASSWORD` | **Required** in staging/production (`ENV.fetch` with no default — boot fails without them) |
+| `SES_SMTP_ADDRESS`, `SES_SMTP_PORT` | Default `email-smtp.us-east-1.amazonaws.com`, `587` |
+| `RAILS_LOG_LEVEL` | Production log level (default `info`) |
+| `PRODUCTION_SERVER_IP`, `STAGING_SERVER_IP` | Capistrano deploy targets |
+| `APP_PORT` | Docker Compose host/container port (default 3000) |
+
+On the servers these are read from `/etc/stocks/env`; Capistrano sources that file for `assets:precompile` and `db:migrate`.
+
+Ports and endpoints: app on `localhost:3000`, Postgres `5432`, Redis `6379` (compose only). Health check at `GET /up` (silenced in logs). Production terminates TLS at a Lightsail load balancer, so `assume_ssl = true` and `force_ssl = false`.
+
+## Key Entry Points
+
+| File | Why it matters |
+|------|----------------|
+| `config/routes.rb` | Complete surface area: `root home#index`, `devise_for :users`, `resources :classrooms` (nested grade books, students, enrollments), `resources :orders`, `namespace :admin` |
+| `app/controllers/application_controller.rb` | Global auth, Pundit wiring, navbar stock scope, role-aware redirect on authorization failure |
+| `app/controllers/admin/base_controller.rb` | Admin gate + shared sorting helper |
+| `app/models/order.rb` | The densest file in the app — all trading validations and sort scopes |
+| `app/services/execute_order.rb` + `app/jobs/order_execution_job.rb` | How a pending order actually settles |
+| `app/models/portfolio.rb` + `app/models/portfolio_position.rb` | Balance derivation and holdings aggregation SQL |
+| `app/models/grade_entry.rb` + `app/services/distribute_earnings.rb` | Earnings math |
+| `config/recurring.yml`, `config/queue.yml` | Everything scheduled |
+| `docs/orders-and-transactions.md`, `docs/gradebook-earnings.md` | Domain rules in prose — read these before touching money code |
+
+## Development, Testing, Deployment
+
+- **Run locally:** `bin/setup` then `bin/dev` (Procfile.dev = rails server + `tailwindcss:watch` + `solid_queue:start`). Docker: `docker compose up`, with `bin/dc <cmd>` as a shortcut for `docker compose run stocks`.
+- **Tests:** `bin/rails test` and `bin/rails test:system` (87 test files). Minitest with FactoryBot factories in `test/factories/`, parallelized by processor count (override with `PARALLEL_WORKERS`), `WebMock.disable_net_connect!`, coverage via `COVERAGE=true` (forces 1 worker).
+- **Lint:** `bin/lint` runs i18n-tasks normalization, RuboCop, erb_lint, Brakeman (`--exit-on-warn`), bundler-audit, and `importmap audit`. CI enforces this.
+- **Deploy:** pushes to `main` run tests then `bundle exec cap staging deploy` (`.github/workflows/deploy-staging.yml`); production is a manual `cap production deploy`. Capistrano deploys to `/home/ubuntu/stocks-in-the-future` on Lightsail with rbenv Ruby 3.4.4, links `config/database.yml`, runs a custom `db:migrate` after publishing, restarts the `stocks` systemd unit, and re-chmods the Puma socket path for nginx.
+
+## Notes & Gotchas
+
+- **Hard deletes of users raise outside production.** `User#destroy`/`destroy!` are overridden to `discard`, and `soft_delete_guard` raises a loud error in dev/test. Use `really_destroy!` only if you truly mean it.
+- **Devise quirks:** `config.authentication_keys = [:username]`, and `User#email_changed?` is hard-coded to `false` so Devise never demands re-confirmation. Students are created with `email = nil`; teachers get `username = email`.
+- **Passwords for students are generated, not chosen** — `MemorablePasswordGenerator` builds `Superhero + number + Superhero` from Faker (marked "TODO: more robust solution later") and the plaintext is surfaced once in a flash message.
+- **`API_KEY` is a global constant** defined in `config/initializers/api_keys.rb` with a default of `"test-api-key"`. `StockAttributeUpdate` uses that constant, while `AlphaVantageApiClient` reads `ENV` directly and returns `nil` when unset — so missing keys fail quietly in two different ways.
+- **Docs drift from `config/recurring.yml`.** `docs/scheduling.md` says `OrderExecutionJob` runs at 1:00 AM ET on weekdays and then *triggers* `StockPricesUpdateJob`; in the code the job is scheduled every 15 minutes and the price update is an independent cron entry. Trust `config/recurring.yml` and the job source.
+- `docs/README.md` links to `docs/architecture/index.md`, which does not exist in the repo.
+- **Production Active Storage is `:heroku`, which is a `Disk` service rooted at `tmp/storage`** (`config/storage.yml`). Uploads are effectively ephemeral and not shared across instances.
+- **Redis is vestigial.** `docker-compose.yml` starts Redis and CI sets `REDIS_URL`, but there is no `redis` gem and Solid Queue is entirely Postgres-backed.
+- **Other leftovers:** `bin/delayed_job` exists although Delayed Job isn't in the Gemfile (the `daemons` gem is still there), and `.standard.yml` is present although `standard` isn't a dependency — RuboCop is the real linter.
+- **Dual form-builder stacks:** `app/components/shadcn/form_builder.rb` and `app/form_builders/admin/form_builder.rb`, plus a hand-rolled component layer in `app/helpers/components/*` rendering `app/views/components/ui/*`. Check which one a view uses before adding fields.
+- The `/admin` namespace is the in-house rewrite that used to live at `/admin-new` (see the comment in `config/routes.rb`); older non-admin controllers still serve overlapping teacher-facing screens (e.g. both `ClassroomsController` and `Admin::ClassroomsController`).
+- `config.load_defaults 8.0` while running Rails 8.1 — new 8.1 framework defaults are not enabled.
+- `Order` includes `ApplicationHelper` (a view helper) just to call `format_money` inside validation messages.
+- Repo state note: the working tree is on a **detached HEAD**, `app/.DS_Store` files show as deleted, and an untracked 15 MB `GITFOLDER.zip` sits in the project root.