diff --git a/README.md b/README.md index 0640422..3886e35 100644 --- a/README.md +++ b/README.md @@ -1,94 +1,261 @@ -# Tool Catalog - -This README catalogs the tools and resources in this directory. It is intended for humans and LLM agents to quickly identify what is available and when to use it. - -## Catalog - -| Tool / Folder | Purpose | Key Files | When to Use | -| --- | --- | --- | --- | -| `workflows/tool-catalog-maintainer/` | Creates and maintains catalog READMEs for directories of tools/resources. | `SKILL.md`, `CONTEXT.md`, `references/catalog-format.md` | Use when asked to catalog, index, or update documentation for a folder of tools. | -| `workflows/research-dfns/` | DFNS research folder — alerts logs, chat log, and TODO tracking. | `TODO.md`, `AlertsLog_*.txt`, `ChatLog_*.txt` | Use when reviewing or continuing DFNS research. | -| `sources/260731-1330-credit-spread.md` | SPY Bull Put Credit Spread trade plan (0DTE, July 31). | `260731-1330-credit-spread.md` | Use for today's SPY options trade — bullish slow grind thesis. | -| `sources/iron-condor-45dte.md` | SPX Iron Condor trade plan — 45DTE entry, 21DTE time exit, Schwab broker. | `iron-condor-45dte.md` | Use when setting up or managing SPX iron condor spreads. | -| `sources/cha-martin-watchlist.ms.csv` | Cha Martin watchlist export (CSV). | `cha-martin-watchlist.ms.csv` | Use for ticker watchlist reference or scan filtering. | - -## Tools - -### `workflows/tool-catalog-maintainer/` - -**Purpose:** -Creates and maintains a `README.md` catalog for a directory containing subfolders of tools, prompts, scripts, docs, workflows, skills, or other reusable resources. The README is a living catalog that improves over time. - -**Contents:** -- `SKILL.md` — skill definition and workflow -- `CONTEXT.md` — full context: inputs, process, outputs, verification -- `references/catalog-format.md` — catalog README structure and cataloging rules -- `output/` — generated artifacts - -**Use when:** -You need to catalog, index, summarize, or update documentation for a folder of tools so humans or LLM agents can quickly choose the right resource. - -**Setup / dependencies:** -None noted. Works with file read/write tools. - -**Notes:** -Always use this tool when asked to catalog a directory — inspect subfolders, read their key files, and write the README using `references/catalog-format.md`. - -### `workflows/research-dfns/` - -**Purpose:** -Research folder for DFNS (Digital Frontier Acquisition Corp.) — contains alert logs, chat transcripts, and tracking notes from July 27, 2026. - -**Contents:** -- `TODO.md` — research tracking tasks -- `AlertsLog_Mon Jul 27 2026*.txt` — alert logs (3 files) -- `ChatLog_Mon Jul 27 2026.txt` — chat transcript - -**Use when:** -Reviewing or continuing DFNS research. Note: per ticker tracking rule, DFNS should not be actively tracked unless a fresh positive reason arises. - -**Setup / dependencies:** -None noted. - -### `sources/260731-1330-credit-spread.md` - -**Purpose:** -Active trade plan for a SPY Bull Put Credit Spread (0DTE, July 31, 2026). Sell $740 put / Buy $738 put spread, targeting 20%+ profit with bullish slow grind thesis. - -**Use when:** -Executing or monitoring today's SPY options trade. - -**Setup / dependencies:** -None noted. Trade-specific, not reusable. - -### `sources/iron-condor-45dte.md` - -**Purpose:** -Trade plan for SPX Iron Condor through Schwab broker. 45DTE entry, 30-point wings, forced close at 21DTE. Includes profit target (25% of net credit), stop loss (50% of max loss), and monitoring cadence. - -**Use when:** -Setting up or managing SPX iron condor spreads with Schwab. - -**Setup / dependencies:** -Schwab broker session required. - -### `sources/cha-martin-watchlist.ms.csv` - -**Purpose:** -CSV export of a Cha Martin watchlist — likely contains tickers for scanning or monitoring. - -**Use when:** -Referencing tickers for scans, short squeeze candidates, or watchlist filtering. - -**Setup / dependencies:** -None noted. - -## Maintenance Notes - -When adding or updating a tool folder, update this README with: - -- purpose -- key files and entry points -- usage guidance -- setup requirements -- notable changes or cautions +# Tool Catalog + +This README catalogs the tools and resources in this directory. It is intended for humans and LLM agents to quickly identify what is available and when to use it. + +## Catalog + +| Tool / Folder | Purpose | Key Files | When to Use | +| --- | --- | --- | --- | +| `workflows/tool-catalog-maintainer/` | Creates and maintains catalog READMEs for directories of tools/resources. | `SKILL.md`, `CONTEXT.md`, `references/catalog-format.md` | Use when asked to catalog, index, or update documentation for a folder of tools. | +| `workflows/research-dfns/` | DFNS research folder — alerts logs, chat log, and TODO tracking from July 27, 2026. | `TODO.md`, `AlertsLog_*.txt`, `ChatLog_*.txt` | Use when reviewing or continuing DFNS research. | +| `workflows/backtest-strategy/` | Orchestrates rules-based strategy backtesting for core styles using TTG data and a pure-Python engine. | `CLAUDE.md`, `CONTEXT.md`, `_config/`, `stages/`, `shared/`, `pyproject.toml` | Use when backtesting trading strategies needing review-gated workflow, data caching, and honest reporting. | +| `workflows/scan-watchlist-for-equities/` | Scans equities for trade setups with confluence using a mechanical data-fetch script. | `CLAUDE.md`, `CONTEXT.md`, `_config/`, `shared/scripts/fetch_scan_data.py`, `stages/` | Use when scanning equities for confluence, generating mechanical data briefs, then interpreting for trade plans. | +| `workflows/scan-watchlist-for-options/` | Scans options for trade setups with confluence using a mechanical data-fetch script. | `CLAUDE.md`, `CONTEXT.md`, `_config/`, `shared/` (CSV watchlists, `scan-watchlist.md`), `stages/` | Use when scanning options for confluence, generating mechanical data briefs, then interpreting for trade plans. | +| `sources/260731-1330-credit-spread.md` | SPY Bull Put Credit Spread trade plan (0DTE, July 31, 2026). Sell $740 put / Buy $738 put spread. | `260731-1330-credit-spread.md` | Use for today's SPY options trade — bullish slow grind thesis. | +| `sources/iron-condor-45dte.md` | SPX Iron Condor trade plan — 45DTE entry, 21DTE time exit, Schwab broker. | `iron-condor-45dte.md` | Use when setting up or managing SPX iron condor spreads. | +| `sources/cha-martin-watchlist.ms.csv` | Cha Martin watchlist export (CSV) for ticker reference or scan filtering. | `cha-martin-watchlist.ms.csv` | Use for ticker watchlist reference or scan filtering. | +| `sources/2026-08-11-SFY-IC-Debrief.md` | Debrief of a SPY Iron Condor trade on 2026-08-11 with observations and key takeaways. | `2026-08-11-SFY-IC-Debrief.md` | Use for reviewing past iron condor trades to learn and improve. | +| `sources/HANDOFF_scan_workflow_scripts.md` | Documentation of the mechanical data-fetch script for scan workflows (API, MCP, output format). | `HANDOFF_scan_workflow_scripts.md` | Use when understanding or implementing the fetch_scan_data.py script. | +| `sources/servers.md` | List of MCP servers available in the MARI environment (ttg-stocks, ttg-benzinga, etc.). | `servers.md` | Use when referencing which data servers are available for MCP tool calls. | +| `sources/xpon-squeeze-2026-08-24.md` | XPON Short Squeeze — Premarket signal reconstruction (2026-08-24) with volume spike and squeeze mechanics. | `xpon-squeeze-2026-08-24.md` | Use for studying a micro-cap short squeeze setup and premarket signals. | +| `tos-short-squeeze/` | Contains a TSV file for short squeeze scanning. | `ShortSqueezeScanner.tsv` | Use for scanning short squeeze opportunities using the provided TSV data. | +| `backtest-workflow-PLAN.md` | Draft plan for a backtesting workflow (approved and built) describing architecture and stages. | `backtest-workflow-PLAN.md` | Use for understanding the design of the backtest-strategy workspace. | + +## Tools + +### `workflows/tool-catalog-maintainer/` + +**Purpose:** +Creates and maintains a `README.md` catalog for a directory containing subfolders of tools, prompts, scripts, docs, workflows, skills, or other reusable resources. The README is a living catalog that improves over time. + +**Contents:** +- `SKILL.md` — skill definition and workflow +- `CONTEXT.md` — full context: inputs, process, outputs, verification +- `references/catalog-format.md` — catalog README structure and cataloging rules +- `output/` — generated artifacts + +**Use when:** +You need to catalog, index, summarize, or update documentation for a folder of tools so humans or LLM agents can quickly choose the right resource. + +**Setup / dependencies:** +None noted. Works with file read/write tools. + +**Notes:** +Always use this tool when asked to catalog a directory — inspect subfolders, read their key files, and write the README using `references/catalog-format.md`. + +### `workflows/research-dfns/` + +**Purpose:** +Research folder for DFNS (Digital Frontier Acquisition Corp.) — contains alert logs, chat transcripts, and tracking notes from July 27, 2026. + +**Contents:** +- `TODO.md` — research tracking tasks +- `AlertsLog_Mon Jul 27 2026*.txt` — alert logs (3 files) +- `ChatLog_Mon Jul 27 2026.txt` — chat transcript + +**Use when:** +Reviewing or continuing DFNS research. Note: per ticker tracking rule, DFNS should not be actively tracked unless a fresh positive reason arises. + +**Setup / dependencies:** +None noted. + +### `workflows/backtest-strategy/` + +**Purpose:** +Orchestrates rules-based strategy backtesting for core styles (SPY 0DTE credit spreads, single-leg equity/options day trades) using TTG data servers as the only market-data source and a local pure-Python engine. + +**Contents:** +- `CLAUDE.md` — workspace identity and entry point +- `CONTEXT.md` — workspace-level routing and stage description +- `_config/` — stable references: `risk-params.md`, `strategy-spec-TEMPLATE.md` +- `stages/` — five stage folders (00_clarify_strategy through 04_report) each with review gates +- `shared/` — data cache (`shared/data/`) and engine scripts (`shared/scripts/`) +- `pyproject.toml` — zero dependencies (stdlib-only) + +**Use when:** +Backtesting trading strategies using TTG data, needing a review-gated workflow with data caching, honest reporting, and no look-ahead. + +**Setup / dependencies:** +None noted (stdlib-only, zero installs; the operator manages packaging with `uv`). Market data comes ONLY from TTG data servers. + +**Notes:** +Follows ICM.md — every stage ends at a review gate. Output is read and optionally edited before the next stage runs. Pure Python 3.12 stdlib (json, csv, sqlite3, statistics, math, datetime, urllib). No historical greeks/IV series; backtests are price/levels-driven. + +### `workflows/scan-watchlist-for-equities/` + +**Purpose:** +Scans tickers for trade setups with confluence, using a mechanical data-fetch script to gather price/volume/bars/S-R data from TTG, producing a data brief for interpretation. + +**Contents:** +- `CLAUDE.md` — workspace identity +- `CONTEXT.md` — workspace routing and stage description +- `_config/` — references such as ticker list, default parameters +- `shared/scripts/fetch_scan_data.py` — stdlib-only Python script for fetching and computing mechanical data +- `shared/` — watchlist CSVs (e.g., `cha-martin-watchlist.ms.csv`, date-named CSVs) +- `stages/` — five stage folders (00_clarify through 04_summary_card) + +**Use when:** +Scanning equities for trade setups with confluence, generating mechanical data briefs (no interpretation), then interpreting the brief for trade plans. + +**Setup / dependencies:** +Python 3.11+, access to TTG data API via `MARI_MCP_CONFIG` environment variable. The fetch script is stdlib-only (urllib, json, etc.). + +**Notes:** +The workflow includes a mechanical script (`fetch_scan_data.py`) that fetches and computes data without interpretation. Stages: 00_clarify (ask clarifying questions), 01_broad_filter (runs the script), 02_confluence_analysis, 03_trade_plan, 04_summary_card. The script writes `data_brief.md` and `raw_data.json` to `stages/01_broad_filter/output/`. + +### `workflows/scan-watchlist-for-options/` + +**Purpose:** +Scans options tickers for trade setups with confluence using a mechanical data-fetch script similar to the equities version. + +**Contents:** +- `CLAUDE.md` — workspace identity +- `CONTEXT.md` — workspace routing and stage description (identical to equities version) +- `_config/` — `trade_plan_template.md` +- `shared/` — CSV watchlists (e.g., `cha-martin-watchlist.ms.csv`, date-named CSVs) and `scan-watchlist.md` +- `stages/` — five stage folders (00_clarify through 04_summary_card) + +**Use when:** +Scanning options for trade setups with confluence, generating mechanical data briefs, then interpreting the brief for trade plans. + +**Setup / dependencies:** +Python 3.11+, access to TTG data API via `MARI_MCP_CONFIG` environment variable. The fetch script is stdlib-only. + +**Notes:** +Very similar to the equities workflow but focused on options. The shared folder contains CSV watchlists and a `scan-watchlist.md` file. The mechanical script (if present) would fetch options data; verify the exact script name and location. + +### `sources/260731-1330-credit-spread.md` + +**Purpose:** +Active trade plan for a SPY Bull Put Credit Spread (0DTE, July 31, 2026). Sell $740 put / Buy $738 put spread, targeting 20%+ profit with bullish slow grind thesis. + +**Contents:** +- `260731-1330-credit-spread.md` — trade plan with entry, exit, profit target, stop loss, and monitoring notes. + +**Use when:** +Executing or monitoring today's SPY options trade (July 31, 2026). + +**Setup / dependencies:** +None noted. Trade-specific, not reusable. + +### `sources/iron-condor-45dte.md` + +**Purpose:** +Trade plan for SPX Iron Condor through Schwab broker. 45DTE entry, 30-point wings, forced close at 21DTE. Includes profit target (25% of net credit), stop loss (50% of max loss), and monitoring cadence. + +**Contents:** +- `iron-condor-45dte.md` — trade plan with entry, exit, risk management, and monitoring details. + +**Use when:** +Setting up or managing SPX iron condor spreads with Schwab broker. + +**Setup / dependencies:** +Schwab broker session required. + +### `sources/cha-martin-watchlist.ms.csv` + +**Purpose:** +CSV export of a Cha Martin watchlist — likely contains tickers for scanning or monitoring. + +**Contents:** +- `cha-martin-watchlist.ms.csv` — plain CSV with one ticker per line (or columns). + +**Use when:** +Referencing tickers for scans, short squeeze candidates, or watchlist filtering. + +**Setup / dependencies:** +None noted. + +### `sources/2026-08-11-SFY-IC-Debrief.md` + +**Purpose:** +Debrief of a SPY Iron Condor trade on 2026-08-11, summarizing trade details, observations, key takeaways, and next times. + +**Contents:** +- `2026-08-11-SFY-IC-Debrief.md` — trade summary, plan, observations, asymmetric structure notes, NL3 feed issue, key takeaways. + +**Use when:** +Reviewing past iron condor trades to learn from observations and improve future trades. + +**Setup / dependencies:** +None noted. + +### `sources/HANDOFF_scan_workflow_scripts.md` + +**Purpose:** +Documentation of a mechanical data-fetch script for the scan-watchlist-for-equities workflow, detailing API facts, script design, environment, MCP protocol, verified responses, output format, and integration notes. + +**Contents:** +- `HANDOFF_scan_workflow_scripts.md` — detailed handoff covering goal, design, environment facts, MCP-over-HTTP protocol, verified response shapes, data brief format, stitching into CONTEXT.md, verification targets, pitfalls, resume checklist, and current state. + +**Use when:** +Understanding or implementing the `fetch_scan_data.py` script, or integrating mechanical data fetching into scan workflows. + +**Setup / dependencies:** +None noted. + +### `sources/servers.md` + +**Purpose:** +List of MCP servers available in the MARI environment for tool calls (e.g., market data, news, chat, etc.). + +**Contents:** +- `servers.md` — plain list of server names: `chrome-devtools`, `mari-cell`, `ttg-benzinga`, `ttg-chat`, `ttg-crypto`, `ttg-economy`, `ttg-finviz-elite`, `ttg-forex`, `ttg-futures`, `ttg-holygrail`, `ttg-indices`, `ttg-options`, `ttg-platform`, `ttg-stocks`, `ttg-uw`. + +**Use when:** +Referencing which data servers are available for MCP tool calls in workflows or scripts. + +**Setup / dependencies:** +None noted. + +### `sources/xpon-squeeze-2026-08-24.md` + +**Purpose:** +XPON Short Squeeze — Premarket signal reconstruction (2026-08-24) documenting a micro-cap short squeeze setup with volume spike, range break, premarket gap, and squeeze mechanics. + +**Contents:** +- `xpon-squeeze-2026-08-24.md` — detailed analysis of the setup, premarket signal, squeeze mechanics, why it was flagged, what went wrong, risk assessment, and key takeaways. + +**Use when:** +Studying a micro-cap short squeeze setup and premarket signals for scanning or education. + +**Setup / dependencies:** +None noted. + +### `tos-short-squeeze/` + +**Purpose:** +Contains a TSV file for short squeeze scanning, likely a precomputed list of candidates. + +**Contents:** +- `ShortSqueezeScanner.tsv` — tab-separated values with columns likely including ticker, metrics, etc. + +**Use when:** +Scanning for short squeeze opportunities using the provided TSV data as input or reference. + +**Setup / dependencies:** +None noted. + +### `backtest-workflow-PLAN.md` + +**Purpose:** +Draft plan for a backtesting workflow (approved and built) describing the architecture, stages, and rules for backtesting core styles using TTG data and a pure-Python engine. + +**Contents:** +- `backtest-workflow-PLAN.md` — goal, architecture (data, engine, reporting layers), proposed workspace layout, stage contracts, decision points, known limitations, credit/cost profile, and out-of-scope items. + +**Use when:** +Understanding the design of the `backtest-strategy` workspace or as a reference for building similar review-gated backtesting workflows. + +**Setup / dependencies:** +None noted. + +## Maintenance Notes + +When adding or updating a tool folder, update this README with: + +- purpose +- key files and entry points +- usage guidance +- setup requirements +- notable changes or cautions \ No newline at end of file diff --git a/README.md.backup b/README.md.backup new file mode 100644 index 0000000..0640422 --- /dev/null +++ b/README.md.backup @@ -0,0 +1,94 @@ +# Tool Catalog + +This README catalogs the tools and resources in this directory. It is intended for humans and LLM agents to quickly identify what is available and when to use it. + +## Catalog + +| Tool / Folder | Purpose | Key Files | When to Use | +| --- | --- | --- | --- | +| `workflows/tool-catalog-maintainer/` | Creates and maintains catalog READMEs for directories of tools/resources. | `SKILL.md`, `CONTEXT.md`, `references/catalog-format.md` | Use when asked to catalog, index, or update documentation for a folder of tools. | +| `workflows/research-dfns/` | DFNS research folder — alerts logs, chat log, and TODO tracking. | `TODO.md`, `AlertsLog_*.txt`, `ChatLog_*.txt` | Use when reviewing or continuing DFNS research. | +| `sources/260731-1330-credit-spread.md` | SPY Bull Put Credit Spread trade plan (0DTE, July 31). | `260731-1330-credit-spread.md` | Use for today's SPY options trade — bullish slow grind thesis. | +| `sources/iron-condor-45dte.md` | SPX Iron Condor trade plan — 45DTE entry, 21DTE time exit, Schwab broker. | `iron-condor-45dte.md` | Use when setting up or managing SPX iron condor spreads. | +| `sources/cha-martin-watchlist.ms.csv` | Cha Martin watchlist export (CSV). | `cha-martin-watchlist.ms.csv` | Use for ticker watchlist reference or scan filtering. | + +## Tools + +### `workflows/tool-catalog-maintainer/` + +**Purpose:** +Creates and maintains a `README.md` catalog for a directory containing subfolders of tools, prompts, scripts, docs, workflows, skills, or other reusable resources. The README is a living catalog that improves over time. + +**Contents:** +- `SKILL.md` — skill definition and workflow +- `CONTEXT.md` — full context: inputs, process, outputs, verification +- `references/catalog-format.md` — catalog README structure and cataloging rules +- `output/` — generated artifacts + +**Use when:** +You need to catalog, index, summarize, or update documentation for a folder of tools so humans or LLM agents can quickly choose the right resource. + +**Setup / dependencies:** +None noted. Works with file read/write tools. + +**Notes:** +Always use this tool when asked to catalog a directory — inspect subfolders, read their key files, and write the README using `references/catalog-format.md`. + +### `workflows/research-dfns/` + +**Purpose:** +Research folder for DFNS (Digital Frontier Acquisition Corp.) — contains alert logs, chat transcripts, and tracking notes from July 27, 2026. + +**Contents:** +- `TODO.md` — research tracking tasks +- `AlertsLog_Mon Jul 27 2026*.txt` — alert logs (3 files) +- `ChatLog_Mon Jul 27 2026.txt` — chat transcript + +**Use when:** +Reviewing or continuing DFNS research. Note: per ticker tracking rule, DFNS should not be actively tracked unless a fresh positive reason arises. + +**Setup / dependencies:** +None noted. + +### `sources/260731-1330-credit-spread.md` + +**Purpose:** +Active trade plan for a SPY Bull Put Credit Spread (0DTE, July 31, 2026). Sell $740 put / Buy $738 put spread, targeting 20%+ profit with bullish slow grind thesis. + +**Use when:** +Executing or monitoring today's SPY options trade. + +**Setup / dependencies:** +None noted. Trade-specific, not reusable. + +### `sources/iron-condor-45dte.md` + +**Purpose:** +Trade plan for SPX Iron Condor through Schwab broker. 45DTE entry, 30-point wings, forced close at 21DTE. Includes profit target (25% of net credit), stop loss (50% of max loss), and monitoring cadence. + +**Use when:** +Setting up or managing SPX iron condor spreads with Schwab. + +**Setup / dependencies:** +Schwab broker session required. + +### `sources/cha-martin-watchlist.ms.csv` + +**Purpose:** +CSV export of a Cha Martin watchlist — likely contains tickers for scanning or monitoring. + +**Use when:** +Referencing tickers for scans, short squeeze candidates, or watchlist filtering. + +**Setup / dependencies:** +None noted. + +## Maintenance Notes + +When adding or updating a tool folder, update this README with: + +- purpose +- key files and entry points +- usage guidance +- setup requirements +- notable changes or cautions diff --git a/backtest-workflow-PLAN.md b/backtest-workflow-PLAN.md new file mode 100644 index 0000000..14e2131 --- /dev/null +++ b/backtest-workflow-PLAN.md @@ -0,0 +1,107 @@ +# Backtest Workflow — DRAFT PLAN (for review — nothing built yet) + +**Status:** APPROVED (2026-08-27 ~11 PM ET) and BUILT — workspace live at +`workflows/backtest-strategy/`. No packages installed (zero-dep pyproject.toml +included; operator manages packaging with uv). No data pulled yet. Engine is +authored at stage-03 exec time, not pre-built. +**Date drafted:** Thu 2026-08-27, ~11 PM ET +**Proposed workspace:** `workflows/backtest-strategy/` (follows ICM.md — folder structure as orchestrator) + +--- + +## Goal + +Rules-based backtesting for Gump's core styles, using TTG data servers as the only market-data source and a local pure-Python engine: + +1. **SPY 0DTE credit put spreads** (bread and butter — premium-selling, % profit targets) +2. **Single-leg equity/options day trades** (VWAP-pullback longs, 50MA-fade shorts) +3. Multi-target management exactly as traded live: T1/T2/T3 partials, stop tightening, hard time exit (15:45 ET), optional breakeven-after-T1 + +## Architecture — three layers + +### 1. Data layer (TTG servers only — never scraped) + +| Need | Source (verified 08/27) | Notes | +|---|---|---| +| Equity bars | custom OHLC bars, daily summaries, grouped-daily (all tickers) | minute-level intraday | +| Options bars | per-contract OHLC bars | minute→month timespans, **back to 2014-06-02**, ≤50,000 bars/pull | +| Contract enumeration | contract list + specs | which strikes/expiries existed on date X | +| Historical quotes | per-contract quote history | **depth unverified — stage 01 verifies before any promise** | +| Historical ticks | per-contract trade history | fallback/verification for quote data | +| Snapshots (live) | chain w/ greeks+IV, per-contract, unified | current-tape only — not for backtests | + +**Cache design:** all pulls land in `shared/data/` as SQLite (stdlib `sqlite3`), keyed by (contract, timespan, window). Re-runs read cache first — pulls happen once per window ever. Cache caps enforced so the folder doesn't balloon. + +### 2. Engine layer — pure Python 3.12 stdlib (v1: zero installs) + +- Bar-by-bar event loop over cached data (`json`, `csv`, `sqlite3`, `statistics`, `math`, `datetime` only) +- Strategies are **declarative spec files** (`_config/strategy-*.md` + params block) — human-readable, reviewable, diffable +- Position model: + - Single leg (long/short equity or option) + - **Two-leg credit spread** = short leg + long leg bars stitched; credit = leg diff at entry; P&L tracked per bar +- Exit model: stop, T1/T2/T3 partial scale-outs, hard time exit, optional breve-after-T1, EOD flat (15:45 ET per house rule) +- Costs: configurable slippage (default **conservative**: mid ± half-spread per fill), optional commissions; RTH default / extended optional flag per session-language rule +- Validation: in-sample/out-of-sample split by date; no look-ahead (signals computed on bars ≤ current bar only) + +### 3. Reporting layer + +- `trades.csv` — every simulated trade: entry/exit, legs, MAE/MFE, time-in-trade, exit reason +- `metrics.json` — win rate, expectancy, avg win/loss, profit factor, max drawdown, per-hour-of-day and per-weekday breakdowns +- Final stage renders a readable report + summary card + +## Proposed ICM workspace layout + +``` +workflows/backtest-strategy/ +├── CLAUDE.md # entry point + exec protocol (read ICM.md fresh, clear output/ on re-run) +├── CONTEXT.md # workspace context: routing + data rules (TTG-only, no fabrication) +├── _config/ +│ ├── strategy-spec-TEMPLATE.md +│ └── risk-params.md # default slippage, session, sizing +├── shared/ +│ └── data/ # SQLite cache (built during runs) +└── stages/ + ├── 00_clarify_strategy/ # capture rules in plain English → strategy_spec.md + ├── 01_verify_data/ # enumerate contracts, verify quote-history depth → data_manifest.md + ├── 02_fetch_cache/ # pull bars/quotes → shared/data/ → fetch_log.md + ├── 03_run_backtest/ # engine executes spec → trades.csv + metrics.json + └── 04_report/ # human-readable report + summary card → report.md +``` + +Every stage ends at a **review gate** — output/ is read and (if needed) edited before the next stage runs, per ICM. + +## Stage contracts (summary) + +| Stage | Reads | Does | Writes | +|---|---|---|---| +| 00_clarify | _config templates, member Q&A | freeze strategy rules + params | strategy_spec.md | +| 01_verify_data | strategy_spec.md | enumerate contracts; **test-pull quote history depth**; flag gaps | data_manifest.md | +| 02_fetch_cache | data_manifest.md | pull bars/quotes → cache (single pass, no repeats) | fetch_log.md | +| 03_run_backtest | cache + spec | run engine, no look-ahead, costs applied | trades.csv, metrics.json | +| 04_report | trades + metrics | render findings honestly (incl. small-sample warnings) | report.md | + +## Decision points — need Gump's call + +1. **v1 compute:** pure stdlib, zero installs (recommended) — or approve `pip install pandas numpy` now? +2. **First strategy to backtest:** (a) SPY 0DTE credit put spread *(recommended — core style)*, (b) VWAP-pullback equity long, (c) 50MA fade short? +3. **Data window:** propose **last 6 months** of SPY 0DTE contracts for v1 (cache stays lean; extendable later)? +4. **Slippage default:** conservative mid ± half-spread, tunable in `_config/risk-params.md` — OK? +5. **Sizing model:** fixed 1-contract (clean signal measurement) vs fixed-dollar risk? Propose fixed 1-contract for v1. + +## Known limitations (stated up front) + +- **No historical greeks/IV series** — backtests are price/levels-driven; IV-rank conditions are not testable +- **No order-book replay** — fills modeled at bar close with slippage knob; inherently approximate, slightly optimistic +- **Quote-history depth TBD** — stage 01 measures it before we rely on it +- **0DTE dailies** only exist for the era they traded; earlier "0DTE" = nearest weekly +- Small-sample honesty: 6 months of 0DTE ≈ ~125 trading days — stage 04 will flag low-N results as such + +## Credit/cost profile + +- All pulls are read-only TTG market-data calls (conservation mode is OFF per member). +- Heavy stage is 02; the cache means any re-run/re-parameterization costs **zero** additional pulls. +- Rough v1 pull count: ~125 contracts × 2 legs × 1 window, each well under the 50k-bar cap. + +## Explicitly out of scope for v1 + +- Portfolio-level / multi-strategy simulation, options greeks modeling, intrabar stop sequencing (stops checked bar-by-bar, close-based), live-paper forwarding (backtest ≠ trade plan — any live trade still goes through the normal cockpit path) diff --git a/spy-0dte-scalp-runbook.md b/spy-0dte-scalp-runbook.md new file mode 100644 index 0000000..eb946cc --- /dev/null +++ b/spy-0dte-scalp-runbook.md @@ -0,0 +1,53 @@ +# SPY 0DTE Scalp Runbook — E2E Fast Path +*Built from live execution 2026-08-28 (spy-1787942969793, closed un-filled at 3:00 wall). Companion lessons live in MEMORY.md: record_plan v2 schema, arm-card flow, ttg-options endpoint patterns.* + +## The speed rule +**Never discover inside a trade window.** Design + record the plan BEFORE the intended entry window; arm = one line. First-ever run took ~7 min for design→record (3 schema validation iterations) + unresolved arm-card surfacing = whole window lost. With this runbook: **~2 min to recorded plan, one line to arm.** + +## 1. Pre-stage (do this 5+ min before the window) +- Fresh SPY snapshot (built-in `snapshot` tool) — price, day OHLC, VWAP. +- Direction call: price vs VWAP + day-low/high structure. Trend-day tape → trade the break (continuation), not the bounce. Mid-range chop → stand down. +- Liquidity check (both expiries, Friday rule): + - `ttg-options` → `options_snapshot_chain`, params: `underlyingAsset: "SPY"`, `contract_type: "put"|"call"`, `expiration_date: "2026-08-28"` (EXACT string; range objects `{"gte":...}` = HTTP 500), `limit: 250`, `sort: "strike_price"`. + - 0DTE near-money: 1–2¢ spreads, 300–600K vol = elite. Next-week: 2–3¢ spreads but premium ~$4 → +20% in 15 min needs ~$1.60 SPY move vs ~$0.30 on 0DTE. 0DTE is the 15-min vehicle; next-week is for plays/swings. +- Vehicle pick: ATM-ish strike, delta −0.4 to −0.5 at expected trigger price, spread ≤ 2¢. + +## 2. Record the plan (exact verified v2 shape) +`ttg_trade_workflow_record_plan` — template (from the call that validated): +``` +brokerId: "tos-paper", connectedBrokerPackId: "tos-paper", +accountCode: "D-67336185" // 185; "D-67336186" = 186 +assetClass: "option", symbol: "SPY", side: "buy", +tradeStyle: "scalp", +instrumentRef: {symbol: ".SPY260828P769", strike: 769, type: "put", + expiration: "2026-08-28", underlying: "SPY", + occ: "O:SPY260828P00769000"}, // MUST be object, not string +levels: { + entry: {zone: {low: 0.76, high: 0.92}, type: "limit", minBidSize: 50}, // premium zone + stop: {offset: 0.18}, // points from fill (premium pts for options) + targets: [{offset: 0.20, portionPct: 67}, {offset: 0.40, portionPct: 33}], // max 2 + timeStopSec: 900, // REQUIRED for scalp/day; in-trade from fill + breakevenAfterT1: true +}, +risk: {maxDollars: 70}, // engine computes qty = maxDollars/(offset×100) — NEVER send qty +timeLimitSec: 780, // ENTRY WINDOW seconds (NOT the time stop) +rationale: "≥40 chars — put SPY-level triggers, invalidations, hard walls, order-safety directives here" +``` +Validation gotchas (each cost an iteration): `tradeStyle` not `style`; no top-level entry/stop/target/qty; `instrumentRef` object not string; engine enforces **R:R > 1 — every target offset must exceed the stop offset**. +Response: planId `spy-`, status draft in PLANS/. SPY-level triggers go in rationale. + +## 3. Arm (the one unverified link — TEST FIRST, outside a live window) +Record response says: "Propose an arm trade-action card — the app runs the pack preflight, verifies the stream, starts the preview belt. Do NOT call record_preflight (cockpit packs), do NOT start belt jobs yourself." +- Arm card does NOT auto-render in the cockpit UI (confirmed 2026-08-28: cockpit open+connected, no card). +- **Working hypothesis:** arm via the Live Trading window's own MARI chat — type `arm `. Cockpit agent (locked to the pack) proposes the Arm card → tap Arm (preflight auto-runs) → tap Enter. +- NEVER fake arming from main chat (`update_plan status:"open"` skips preflight/stream verify — dangerous). Main-window MARI cannot arm; cockpit orchestrator owns execution. + +## 4. Manage (standing protocol — member directives, encode in rationale every time) +1. On fill: verify BOTH working and filled orders. Stop must read **SELL TO CLOSE ** — never sell-to-open, never a short entry. +2. T1 fills → scale stop to remaining qty (+ breakevenAfterT1 moves it to BE). T2/stop-out → **cancel ALL working orders the instant flat.** Working + filled checked both after every event. No orphans, ever. +3. Walls: `timeStopSec` from fill + explicit wall-clock hard flat in rationale. Bail on tape deterioration before the stop (slow grind back = failed setup). +4. Pre-entry invalidations: structure reclaim / VWAP reclaim / no trigger by window close → no trade. **Stale plan = close clean, never chase** (2026-08-28: closed un-filled at 3:00 wall, $0 cost). + +## 15-min target math (10–25% premium band) +- 0DTE ATM-ish premium P, delta Δ: +20% ≈ 0.20·P / Δ SPY move. At P≈$0.85, Δ≈−0.50 → ~$0.34 SPY. Routine in 15 min on a break. +- Engine R:R rule forces target offset > stop offset: size stop −18 to −22% and T1 +24 to +26% to stay inside the band AND clear R:R>1 (verified combo: stop 0.18 / T1 0.20 / T2 0.40 on ~$0.84 fill). diff --git a/tos-short-squeeze/ShortSqueezeScanner.tsv b/tos-short-squeeze/ShortSqueezeScanner.tsv new file mode 100644 index 0000000..a73865f --- /dev/null +++ b/tos-short-squeeze/ShortSqueezeScanner.tsv @@ -0,0 +1,86 @@ +ShortSqueeze SqueezeScan # TOS Short Squeeze Scanner +# Run in: TOS -> Tools -> Run Scan (or save to Scan Library) +# Data period: Pre-Market (select in scan data settings) +# Columns: Symbol, Ticker, SI%(manual), Squeeze, Gap%, PremkVol, PremkVolPct, PremkRange%, PremkAboveVWAP, PremkBid, PremkAsk, PremkMid +# +# HOW TO USE: +# 1. TOS -> Tools -> Run Scan. Paste this entire file into the "ThinkScript" box. +# 2. Set data period: Pre-Market. Exclusions: None. +# 3. Universe: US stocks, price > $1, volume > 0. +# (For a tighter list, add a static pre-filter for float < 30M shares.) +# 4. Run. Sort by SqueezeScore. +# 5. Manually enter each candidate's short float % (SI%) using TOS stock +# details page, S3 Partners, or your usual source. This column feeds +# the SqueezeScore but the technical flags work without it. +# 6. Best candidates: Squeeze=1 AND PremkAboveVWAP=0 AND PremkVolPct > 1.5 +# AND SI% > 25. + +input gapUpMin = 1.0; # min % gap up from prev close +input gapUpMax = 12.0; # max % gap up (above this = news/pop already done) +input volPctMin = 1.5; # min premarket volume as % of 1-day avg +input vwAPBand = 0.75; # % band around VWAP where "hug" is valid +input squeezeScoreMin = 1; # min score to show (1-5 scale) + +# ---- Price levels (premarket context) ---- +def prevClose = Close[-1]; +def open = Open; +def high = High; +def low = Low; +def close = Close; +def vol = Volume; + +def gapPct = if prevClose > 0 then (open - prevClose) / prevClose * 100.0 else Double.NaN; + +# ---- Pre-market VWAP approximation ---- +# Built from the day's bars so far, so it includes premarket prints. +def vwapCalc = if Sum(vol, 1) > 0 then Sum(close*vol, 1) / Sum(vol, 1) else Double.NaN; + +def vwapPctFromClose = if vwapCalc > 0 then (close - vwapCalc) / vwapCalc * 100.0 else Double.NaN; + +# ---- Pre-market volume as % of 1-day average ---- +def avgVol1D = Average(vol, 22); # ~1 trading day of 1-min bars +def volPct = if avgVol1D > 0 then vol / avgVol1D * 100.0 else Double.NaN; + +# ---- Pre-market range as % of prev close (narrow range = coiling) ---- +def rangePct = if prevClose > 0 then (high - low) / prevClose * 100.0 else Double.NaN; + +# ---- Is price hugging/just under VWAP? ---- +def huggingVWAP = vwapCalc > 0 and vwapPctFromClose >= -vwAPBand and vwapPctFromClose <= vwAPBand; + +# ---- Squeeze flags ---- +def flagGap = gapPct >= gapUpMin and gapPct <= gapUpMax; +def flagVol = volPct >= volPctMin; +def flagHug = huggingVWAP; +def flagNarrow = rangePct <= 2.5; # narrow range = coiling, ready to spring + +# ---- Squeeze score: 0-5 (add SI% manually in the column) ---- +def score = (if flagGap then 1 else 0) + + (if flagVol then 1 else 0) + + (if flagHug then 1 else 0) + + (if flagNarrow then 1 else 0) + + 0; # SI% is manual - add +1 in your head if SI > 25 + +# ---- Composite squeeze signal ---- +def squeezeSignal = flagGap and flagVol and flagHug and squeezeScore >= squeezeScoreMin; + +# ---- Output columns ---- +plot Ticker = GetSymbol(); +plot Symbol = GetSymbol(); +plot SI_Pct = Double.NaN; # MANUAL: paste short float % per ticker +plot Squeeze = squeezeSignal ? 1 : 0; +plot GapPctOut = gapPct; +plot PremkVol = vol; +plot PremkVolPct = volPct; +plot PremkRangePct = rangePct; +plot PremkAboveVWAP = if vwapPctFromClose > 0 then 1 else 0; +plot PremkVWAPPct = vwapPctFromClose; +plot PremkBid = BidPrice; +plot PremkAsk = AskPrice; +plot PremkMid = (BidPrice + AskPrice) / 2; +plot SqueezeScore = score; + +# ---- Scanner conditions (filter rows) ---- +# Adjust to taste. Starting: at least a gap-up + elevated volume + VWAP hug. +condition SqueezeSetup = squeezeSignal; +condition GapUpOnly = flagGap and flagVol; +condition VolSpikeOnly = flagVol; diff --git a/workflows/backtest-strategy/CLAUDE.md b/workflows/backtest-strategy/CLAUDE.md new file mode 100644 index 0000000..fbde665 --- /dev/null +++ b/workflows/backtest-strategy/CLAUDE.md @@ -0,0 +1,32 @@ +# Backtest Strategy ICM Workspace + +This workspace orchestrates rules-based strategy backtesting for the operator's core +styles (SPY 0DTE credit spreads, single-leg equity/options day trades) using TTG data +servers as the only market-data source and a local pure-Python engine. + +The agent follows the numbered stages to freeze strategy rules, verify data depth, +build the data cache, run the backtest, and produce an honest report. + +Folder structure: +- CLAUDE.md (Layer 0): workspace identity +- CONTEXT.md (Layer 1): workspace-level routing +- stages/: numbered stage folders + - 00_clarify_strategy/: freeze rules into a spec (with operator) + - 01_verify_data/: enumerate contracts + verify historical depth + - 02_fetch_cache/: pull bars/quotes into the shared cache + - 03_run_backtest/: run the engine over cached data + - 04_report/: render findings with small-sample honesty +- _config/: Layer 3 reference material (stable across runs) +- shared/: data cache (SQLite) + engine scripts +- Each stage's output/ holds Layer 4 working artifacts for handoff to next stage. + +## Hard rules (apply to every stage) +- Market data comes ONLY from TTG data servers. Never scrape or substitute public sites. +- Python is stdlib-only. NO package installs. The operator manages packaging with uv + (pyproject.toml in the workspace root has zero dependencies by design). +- No look-ahead: a signal on bar N may only use bars <= N. +- Every stage ends at a review gate. output/ is read (and optionally edited by the + operator) before the next stage runs. +- Clear a stage's output/ before re-running it. +- Backtest results are research, not trade plans. Any live trade still goes through + the normal cockpit path. diff --git a/workflows/backtest-strategy/CONTEXT.md b/workflows/backtest-strategy/CONTEXT.md new file mode 100644 index 0000000..75bdb3d --- /dev/null +++ b/workflows/backtest-strategy/CONTEXT.md @@ -0,0 +1,51 @@ +# Workspace Context: Strategy Backtest + +## Routing +Given a strategy to backtest, the workflow proceeds through stages: + +1. **00_clarify_strategy** - Freeze the strategy rules with the operator into a + declarative spec (structure, entries, exits, session, sizing, costs). +2. **01_verify_data** - Enumerate the contract universe, then verify historical + depth (minute-bar range per contract leg, quote-history lookback) with small + test pulls. Produce a data manifest with honest date ranges and gaps. +3. **02_fetch_cache** - Single-pass bulk pull of bars/quotes into the shared + SQLite cache, keyed by (contract, timespan, window). Cache-first: existing + rows are never re-pulled. +4. **03_run_backtest** - Author/run the stdlib-only engine over cached data. + No look-ahead. Costs from risk-params. Output trades.csv + metrics.json. +5. **04_report** - Human-readable report: win rate, expectancy, drawdown, + per-hour breakdowns. Small-sample results flagged as small-sample. + +## Shared Resources +- _config/: strategy spec template + risk parameters (stable reference). +- shared/data/: SQLite cache of fetched bars/quotes (built during runs). +- shared/scripts/: engine + helper scripts (stdlib only, authored at stage exec). + +## Data rules (verified 2026-08-27) +- Options per-contract OHLC bars: minute→month timespans, history back to + 2014-06-02, max 50,000 bars per pull. Daily summaries + previous-day also exist. +- Options quote history + tick trade history exist per contract; quote-history + lookback depth is UNVERIFIED until stage 01 measures it. +- Equity bars: custom OHLC (minute-level), daily summaries, grouped-daily + (all tickers), previous-day. +- No historical greeks/IV series anywhere in the catalog: backtests are + price/levels-driven. IV-rank conditions are not testable. +- Live chain snapshots (greeks/IV) are current-tape only — never a backtest source. + +## Engine rules +- Pure Python 3.12 stdlib (json, csv, sqlite3, statistics, math, datetime, urllib). + NO pip installs — the operator manages packaging with uv; pyproject.toml + declares zero dependencies. +- A credit spread is two leg pulls stitched: credit = short-leg price − long-leg + price at entry; P&L tracked per bar on the leg diff. +- Costs: conservative slippage default (mid ± half-spread per fill), from + _config/risk-params.md. +- Exits modeled per house style: stop, T1/T2/T3 partials, hard 15:45 ET time exit, + optional breakeven-after-T1. +- RTH is the default session; extended only if the spec says so explicitly. + +## Honesty rules +- Never present a backtest result without N (trade count) and the tested window. +- Low-N results (<30 trades) must be labeled low-N in the report. +- If data has gaps (no bars for an era/strike), say so in the manifest and report — + never silently interpolate through them. diff --git a/workflows/backtest-strategy/_config/risk-params.md b/workflows/backtest-strategy/_config/risk-params.md new file mode 100644 index 0000000..0a32fcf --- /dev/null +++ b/workflows/backtest-strategy/_config/risk-params.md @@ -0,0 +1,40 @@ +# Risk Parameters (stable defaults — edit sparingly, diffs matter) +# Stage 03 reads this file; the strategy spec may override individual values. + +# Data window (v1) +window: + start: 2026-02-27 # ~6 months back; stage 01 narrows to contract reality + end: 2026-08-27 + +# Costs +costs: + slippage_model: mid_half_spread # each fill at mid ± half observed spread + min_tick: 0.01 # options tick floor + commissions_per_contract: 0.0 # 0 by default; operator may set + +# Sessions +session: + default: RTH # 09:30–16:00 ET + hard_exit_et: "15:45" # house rule: flat before the close + extended_allowed: false # only if strategy spec explicitly enables + +# Sizing +sizing: + mode: fixed_contract + contracts: 1 + +# Exits (defaults; spec overrides) +exits: + breakeven_after_t1: false + +# Cache +cache: + path: shared/data/backtest_cache.sqlite3 + key: (contract, timespan, window) # cache-first; never re-pull existing rows + max_rows_per_table: 5000000 + +# Validation +validation: + no_lookahead: true + low_n_threshold: 30 # reports must flag results below this trade count + split: none # optional in-sample/out-of-sample date split diff --git a/workflows/backtest-strategy/_config/strategy-spec-TEMPLATE.md b/workflows/backtest-strategy/_config/strategy-spec-TEMPLATE.md new file mode 100644 index 0000000..8bf8710 --- /dev/null +++ b/workflows/backtest-strategy/_config/strategy-spec-TEMPLATE.md @@ -0,0 +1,49 @@ +# Strategy Spec Template +# Fill one copy per strategy: _config/strategy-spec-.md +# The frozen spec (stage 00 output) is the single source of truth for the engine. + +# Name +name: + +# Underlying & structure +underlying: +structure: +# For spreads, list legs explicitly: +legs: + - role: short + type: put + selection: + - role: long + type: put + selection: +expiration: + +# Session +session: +entry_window: +hard_exit: + +# Entry rules (plain English, bar-level. NO look-ahead allowed.) +entry: + - + - +all_required: true # every rule must hold on the entry bar + +# Exit rules +exits: + stop: + targets: + - t1: + scale_out: + - t2: <...> + scale_out: <...> + breakeven_after_t1: false + time_exit: 15:45 ET + +# Sizing & costs +sizing: + mode: fixed_contract # v1 default: 1 contract + contracts: 1 +costs: + slippage: mid_half_spread # conservative default; see risk-params.md + commissions: 0 # set if the operator wants them modeled diff --git a/workflows/backtest-strategy/pyproject.toml b/workflows/backtest-strategy/pyproject.toml new file mode 100644 index 0000000..41d44e4 --- /dev/null +++ b/workflows/backtest-strategy/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "backtest-strategy" +version = "0.1.0" +description = "ICM backtest workspace - pure stdlib engine. Operator manages packaging with uv; zero dependencies by design." +requires-python = ">=3.12" +dependencies = [] + +# NOTE: intentionally dependency-free. If a future need for pandas/numpy arises, +# that is an explicit operator decision - not an agent action. diff --git a/workflows/backtest-strategy/shared/data/README.md b/workflows/backtest-strategy/shared/data/README.md new file mode 100644 index 0000000..2a79c8e --- /dev/null +++ b/workflows/backtest-strategy/shared/data/README.md @@ -0,0 +1,8 @@ +# shared/data/ + +Holds the backtest cache (SQLite) built by stage 02_fetch_cache. + +- `backtest_cache.sqlite3` — keyed by (contract, timespan, window). Cache-first: + existing rows are never re-pulled from TTG data servers. +- This folder may grow large. Cap discipline lives in `_config/risk-params.md`. +- Delete the .sqlite3 file to force a full re-pull (stage 02 will rebuild it). diff --git a/workflows/backtest-strategy/stages/00_clarify_strategy/CONTEXT.md b/workflows/backtest-strategy/stages/00_clarify_strategy/CONTEXT.md new file mode 100644 index 0000000..f28a9ed --- /dev/null +++ b/workflows/backtest-strategy/stages/00_clarify_strategy/CONTEXT.md @@ -0,0 +1,28 @@ +# Stage 00 Clarify Strategy: Freeze the Rules + +Purpose: turn the operator's strategy idea into a single declarative spec that the +engine can execute verbatim. Nothing downstream runs until this file exists and the +operator approves it. This is a review gate. + +## Inputs +- Layer 3 (reference): ../../_config/strategy-spec-TEMPLATE.md +- Layer 3 (reference): ../../_config/risk-params.md +- Layer 4 (working): operator's description of the strategy (from conversation) + +## Process +1. Read the template and risk params. +2. Interview the operator until every template field is answerable: structure + (single leg / credit spread / debit spread), leg selection, expiration rule, + entry window, entry rules (bar-level, no look-ahead), stop, targets with + scale-out fractions, time exit, sizing. +3. Restate the rules back in plain English and get explicit operator confirmation + before freezing. Ambiguity is resolved by the operator, never guessed. +4. Default candidate if the operator asks for a starting point: SPY 0DTE credit + put spread (short ~0.5% OTM, long $5 wider, 09:45–11:00 ET entries, 15:45 hard + exit, T1/T2 partials). This is a proposal, not a decision. +5. Write the frozen spec as a filled copy of the template. +6. Stop and hand off to the operator for review. Stage 01 does not start until + the operator approves strategy_spec.md. + +## Outputs +- strategy_spec.md -> output/ diff --git a/workflows/backtest-strategy/stages/01_verify_data/CONTEXT.md b/workflows/backtest-strategy/stages/01_verify_data/CONTEXT.md new file mode 100644 index 0000000..aa629e6 --- /dev/null +++ b/workflows/backtest-strategy/stages/01_verify_data/CONTEXT.md @@ -0,0 +1,34 @@ +# Stage 01 Verify Data: Contract Enumeration + Depth Check + +Purpose: establish, with small test pulls only, exactly what data exists for the +frozen spec's universe — before any bulk fetching. Honesty about gaps is the whole +point of this stage. Review gate. + +## Inputs +- Layer 4 (working): ../00_clarify_strategy/output/strategy_spec.md +- Layer 3 (reference): ../../CONTEXT.md (Data rules section) +- Layer 3 (reference): ../../_config/risk-params.md (window) + +## Process +1. Read the strategy spec to determine the universe: underlying, leg selection + rule, expiration cadence, session. +2. Enumerate the contract universe for the window using the options/stocks + reference endpoints (browse with retrieve_all, confirm shapes with params + before any execute — house rule for every new endpoint). +3. For 2–3 sample contracts (one recent, one mid-window, one oldest needed): + - pull a small bar window (e.g., 1 day of minute bars) per leg to confirm + minute-bar availability within the spec's entry window; + - pull a small quote-history window and record the ACTUAL earliest timestamp + returned. Quote-history lookback depth is unverified — this measures it. +4. Record per-contract-leg findings in the manifest: earliest/latest verified bar + dates, quote-history earliest date, gaps, holidays in window, and any contract + the spec's selection rule would pick that has no data. +5. If the spec's window is not fully coverable (e.g., quote history shallower than + the window), state the impact plainly and propose the largest fully-coverable + window. Do not silently shrink the test. +6. No bulk pulls in this stage. Keep total pulls small (roughly a dozen). +7. Stop for operator review of the manifest before stage 02 fetches anything. + +## Outputs +- data_manifest.md -> output/ (universe table, verified depth per leg, gaps, + proposed final window) diff --git a/workflows/backtest-strategy/stages/02_fetch_cache/CONTEXT.md b/workflows/backtest-strategy/stages/02_fetch_cache/CONTEXT.md new file mode 100644 index 0000000..43e5079 --- /dev/null +++ b/workflows/backtest-strategy/stages/02_fetch_cache/CONTEXT.md @@ -0,0 +1,29 @@ +# Stage 02 Fetch Cache: Single-Pass Bulk Pull + +Purpose: materialize every bar/quote series the manifest calls for into the shared +SQLite cache — once. Re-runs of later stages must never re-pull data. Review gate. + +## Inputs +- Layer 4 (working): ../01_verify_data/output/data_manifest.md +- Layer 3 (reference): ../../_config/risk-params.md (window, cache path, caps) +- Layer 3 (reference): ../../CONTEXT.md (Data rules) + +## Process +1. Read the manifest's final (operator-approved) universe + window. +2. Author a stdlib-only fetch script into ../../shared/scripts/ (urllib for HTTP, + sqlite3 for the cache, json/csv for any side exports). No third-party packages. +3. Cache-first: for each (contract, timespan, window), check the cache and skip + rows already present. Only missing ranges are fetched. +4. Pull legs bar-by-bar: for each contract, each leg, minute bars for the spec's + session window across the manifest's date list. Respect the 50,000-bar per-pull + cap by splitting multi-month pulls into monthly sub-windows. +5. Optional per spec: pull quote history only for the eras stage 01 verified. +6. Log every pull (contract, timespan, date range, rows returned, gaps found) to + the fetch log. A pull returning zero bars is a logged fact, not an error to hide. +7. Sanity-check the cache: row counts per contract vs expected session days; flag + any contract with <50% expected coverage. +8. Stop for operator review before the engine runs. + +## Outputs +- fetch_log.md -> output/ (pull table, coverage stats, anomalies) +- shared/data/backtest_cache.sqlite3 (the cache itself) diff --git a/workflows/backtest-strategy/stages/03_run_backtest/CONTEXT.md b/workflows/backtest-strategy/stages/03_run_backtest/CONTEXT.md new file mode 100644 index 0000000..047d580 --- /dev/null +++ b/workflows/backtest-strategy/stages/03_run_backtest/CONTEXT.md @@ -0,0 +1,38 @@ +# Stage 03 Run Backtest: Execute the Spec Over the Cache + +Purpose: author and run the stdlib-only engine against the cached data, producing +a complete trade list and metrics. The spec is law; the engine never improvises. +Review gate. + +## Inputs +- Layer 4 (working): ../00_clarify_strategy/output/strategy_spec.md +- Layer 4 (working): ../02_fetch_cache/output/fetch_log.md (coverage caveats) +- Layer 3 (reference): ../../_config/risk-params.md (costs, exits defaults) +- Layer 4 (working): ../../shared/data/backtest_cache.sqlite3 + +## Process +1. Author the engine script into ../../shared/scripts/ (pure Python 3.12 stdlib: + sqlite3, csv, json, statistics, math, datetime). No third-party packages — + the operator manages packaging with uv; pyproject.toml has zero dependencies. +2. Engine mechanics: + - Iterate bars chronologically per session; signals on bar N may only use + bars <= N (no look-ahead, hard rule). + - Entries only inside the spec's entry window and session (RTH default). + - Credit spread = short leg + long leg stitched; credit at entry = short + price − long price; per-bar P&L tracked on the leg diff. + - Exits in precedence order: hard time exit (15:45 ET) > stop > targets + (T1/T2/T3 scale-outs per spec) > optional breakeven-after-T1. + - Fills at bar close ± slippage from risk-params (mid ± half-spread default). +3. Emit one row per simulated trade: dates, entry/exit, leg prices, credit, + P&L, MAE/MFE, exit reason, session day. +4. Compute metrics: N, win rate, expectancy, avg win/loss, profit factor, max + drawdown, per-hour-of-day and per-weekday breakdowns, equity curve series. +5. Also run a costs-off variant (same trades, zero slippage) and store both — + the gap between them is the cost drag, and it must be visible. +6. If cached coverage has gaps inside the window, exclude affected days from + stats and list them — never interpolate through a gap. +7. Stop for operator review before the report stage. + +## Outputs +- trades.csv -> output/ +- metrics.json -> output/ (with-costs and costs-off variants) diff --git a/workflows/backtest-strategy/stages/04_report/CONTEXT.md b/workflows/backtest-strategy/stages/04_report/CONTEXT.md new file mode 100644 index 0000000..36191c3 --- /dev/null +++ b/workflows/backtest-strategy/stages/04_report/CONTEXT.md @@ -0,0 +1,29 @@ +# Stage 04 Report: Honest Findings, Operator-Readable + +Purpose: turn trades.csv + metrics.json into a report a trader can act on — or +consciously discard. No cherry-picking, no curve-fit praise. Final review gate. + +## Inputs +- Layer 4 (working): ../03_run_backtest/output/trades.csv +- Layer 4 (working): ../03_run_backtest/output/metrics.json +- Layer 4 (working): ../01_verify_data/output/data_manifest.md (coverage caveats) + +## Process +1. Lead with the headline numbers: N trades, win rate, expectancy per trade, + profit factor, max drawdown — for the with-costs run. Costs-off appears only + as the visible cost-drag comparison, never as the headline. +2. State the tested window, contract universe size, and data coverage honestly, + including any excluded gap days. +3. Flag low-N explicitly: below 30 trades the report must say the sample is too + small to trust and must not recommend going live on it. +4. Breakdowns: per-hour-of-day, per-weekday, exit-reason mix (stop vs target vs + time exit), and the stop-vs-target balance. These drive spec tuning. +5. MAE/MFE distribution: how deep winners usually dip before working — this is + what sets realistic stop placement and profit-target spacing. +6. End with a plain recommendation set: keep / tune (which parameter, which + direction) / discard. A losing strategy is a valid, useful result — say so. +7. Remind the reader: backtest output is research. Live execution still goes + through the normal cockpit path with its own preflight. + +## Outputs +- report.md -> output/