update to where files are stored

sources/HANDOFF_scan_workflow_scripts.md

@@ -0,0 +1,237 @@
+# HANDOFF — scan-watchlist-for-equities: mechanical data-fetch script
+
+Written: Tue 2026-08-18 9:37 PM ET. Purpose: a fresh session can resume this
+build without re-deriving anything. All API facts below were LIVE-VERIFIED
+this session (not from memory).
+
+## 1. What the member asked for
+
+Gump: "I want to create scripts to do some of the work you're doing now and
+stitch these into the workflow to avoid the inference work these stages
+undertake on each run."
+
+Follow-up approval: "yes build it and include Benzinga news pull" — then,
+after the flaky Benzinga endpoint (HTTP 500) and dropped tool payloads:
+"can you continue without benzinga angle?"
+
+=> DECISION: **v1 script = price/volume/bars/S-R only (no Benzinga).**
+News stays an optional agent MCP call at stage 01, exactly as before.
+Benzinga re-add can be v2 once the endpoint proves stable.
+
+## 2. Goal / design
+
+One stdlib-only Python 3.11 script:
+  `workflows/scan-watchlist-for-equities/shared/scripts/fetch_scan_data.py`
+
+CLI:  python3 fetch_scan_data.py <watchlist.csv> [--out DIR]
+         [--regime SPY,QQQ] [--bars-days 80] [--news-since YYYY-MM-DD]
+       (news args drop in v2; ignore for v1)
+
+It fetches, per watchlist ticker, DIRECTLY from TTG real-time data API
+(via streamable-HTTP MCP JSON-RPC — see section 5):
+  - stocks_snapshot_ticker
+  - stocks_aggs_custom_bars daily  (from = today-100d, to = today,
+    multiplier=1, timespan=day, limit=90)
+  - stocks_aggs_custom_bars 3-min  (from/to = last trading day,
+    multiplier=3, timespan=minute, limit=400)
+  - calculate_support_resistance {ticker, timeframe: "daily"}
+  - calculate_support_resistance {ticker, timeframe: "3min"}
+Plus SPY/QQQ regime snapshots.
+
+It computes (mechanical, zero interpretation):
+  - session split of 3-min bars: PM (<09:30 ET), RS (09:30-16:00),
+    AH (>16:00 ET) — per session: o/h/l/c/v and vwap=sum(c*v)/sum(v)
+  - intraday high/low timestamps; volume by RS hour; first-15-min and
+    last-30-min RS moves
+  - MA20 / MA50 from daily closes + distance % (n/a if <20 bars: IPOs)
+  - period high/low from fetched daily bars
+  - vol vs prev-day volume (snapshot day.v / prevDay.v)
+
+It writes into `stages/01_broad_filter/output/`:
+  - data_brief.md   — formatted tables, the mechanical half of shortlist.md
+  - raw_data.json   — every raw payload, audit trail
+stdout: compact fetch summary + output paths. Exit 0/1.
+Per-call: 1 retry, 2s backoff. Timestamps in bars are MILLISECONDS UTC —
+convert to America/New_York (zoneinfo; fall back to local tz on error).
+"Last trading day" for the 3-min fetch = max bar date from the daily bars
+(handles weekend/pre-market runs).
+
+Credentials: read at RUNTIME from env var MARI_MCP_CONFIG (machine-local
+JSON; see section 4). NEVER hardcode keys or write them into the workflow
+folder — the Cell syncs to Fleet.
+
+## 3. Environment facts (verified this session)
+
+- cwd = C:\Users\Eric\AppData\Roaming\MARI Desktop\localCell\accounts\
+  acct-c6fef984c60eb59b9c1c4202ababa059\workspace  (git-bash paths use /c/...)
+- Python 3.11.15 (pyenv) reachable from MARI bash; stdlib urllib HTTPS works
+  (proven by live calls below). Node is UNREACHABLE from MARI bash — use Python.
+- MCP config: env var MARI_MCP_CONFIG ->
+  C:\Users\Eric\AppData\Roaming\MARI Desktop\mcp\acct-c6fef984...json
+  Shape: {"servers": {"ttg-stocks": {"transport":"streamablehttp",
+  "url":"https://rt.data.ttg.ai/stocks/mcp",
+  "auth":{"type":"header","headers":{"Authorization":"Bearer ***",
+  "X-User-ID":"27930"}}}, "ttg-benzinga": {...benzinga/mcp...}, ...}}
+  (Do not print the real bearer key in chat/output.)
+- Workflow root: .mari/scratchpad/workflows/scan-watchlist-for-equities/
+  (Layer-3 scripts belong in shared/scripts/ per ICM Table 2; ICM.md is at
+  workflows/ICM.md — re-read on exec per member's predicate.)
+- Watchlist CSVs live in shared/ (e.g. 260818.csv = "XOS, AMLX" one per line).
+  Script should resolve the CSV relative to shared/ (script_dir.parent).
+
+## 4. MCP-over-HTTP protocol (proven working from Python)
+
+Stateless on these servers: no Mcp-Session-Id returned. Sequence:
+  1) POST url  {"jsonrpc":"2.0","id":1,"method":"initialize","params":
+     {"protocolVersion":"2025-03-26","capabilities":{},
+      "clientInfo":{"name":"fetch_scan_data","version":"1.0"}}}
+  2) POST {"jsonrpc":"2.0","method":"notifications/initialized"}  (best-effort)
+  3) POST {"jsonrpc":"2.0","id":N,"method":"tools/call","params":
+     {"name":<tool>,"arguments":{...}}}
+Headers for every POST: Content-Type: application/json,
+Accept: application/json, text/event-stream, plus the server's auth headers.
+Responses came back plain application/json (not SSE) in testing, but the
+parser MUST handle both: if CT starts with text/event-stream, scan lines
+starting "data:" and JSON-parse until one has the matching id.
+Result text = d["result"]["content"][0]["text"] -> json.loads (it's a
+JSON string). On JSON-RPC error object, raise with the message.
+
+Minimal proven client (this exact code worked):
+  cfg = json.load(open(os.environ['MARI_MCP_CONFIG']))
+  srv = cfg['servers']['ttg-stocks']
+  headers = {'Content-Type':'application/json',
+             'Accept':'application/json, text/event-stream'}
+  headers.update(srv['auth']['headers'])
+  req = urllib.request.Request(srv['url'], data=json.dumps(payload).encode(),
+             headers=headers, method='POST')
+  body = urllib.request.urlopen(req, timeout=60).read().decode()
+
+## 5. Verified response shapes (live, 8/18 evening)
+
+stocks_snapshot_ticker {ticker:"SPY"} ->
+  {"ticker":{"ticker":"SPY","todaysChangePerc":-0.675,"todaysChange":-5.22,
+   "updated":1787097600000000000,"day":{"dv":43920901.2,"o":768.7,"h":769.5,
+   "l":766.92,"c":767.45,"v":...,"vw":768.52,...}, ...}}
+  (also has prevDay with c/v — code defensively with .get; the exact AH
+  sub-fields were not captured: rely on 3-min AH bars for after-hours shape,
+  not the snapshot.)
+
+stocks_aggs_custom_bars {ticker,from,to,multiplier,timespan,limit} ->
+  {"ticker":...,"results":[{"v":22219.6,"vw":768.25,"o":769.21,"c":768.37,
+   "h":769.73,"l":767.8,"t":1787040000000,"n":446}],"next_url":...}
+  NOTE: 'multiplier' IS REQUIRED (missing it = "Missing required params").
+  Daily: multiplier=1 timespan=day. Intraday: multiplier=3 timespan=minute.
+  t = MILLISECONDS UTC.
+
+calculate_support_resistance {ticker, timeframe:"daily"|"3min"} ->
+  {"timeframe":"daily","current_price":4.44,
+   "support_levels":{"minor":[{"price":1.92,"touches":1,
+     "description":"Swing low 3 weeks ago, 1 touch"}],
+    "major":[{"price":3.04,"touches":3,"description":"..."}, ...],
+    "key_references":{"pdl":4.01,"pdc":4.44,"period_low":1.65}},
+   "resistance_levels":{"minor":[...],"major":[...],
+    "key_references":{"pdh":4.95,"period_high":...}}}
+
+benzinga_news (V2 ONLY — endpoint flaky, 500'd once this session):
+  ttg-benzinga execute {endpoint:"benzinga_news",
+  params:{"stocks":"AMLX","published.gte":"2026-08-18"}}  ->
+  {"status":"OK","results":[{"benzinga_id":...,"author":"benzinga newsdesk",
+   "published":"2026-08-18T20:02:46Z","title":"Amylyx Pharmaceuticals
+   Announces $350M Common Stock Offering","url":"...","channels":[...],...}]}
+  Use ONLY minimal params (stocks + published.gte) — extra params
+  (limit/sort) coincided with the 500; retry once on failure.
+
+## 6. data_brief.md format (mirror the mechanical sections of tonight's
+shortlist.md so stage agents can drop them straight in)
+
+  # Data Brief — scan-watchlist-for-equities
+  Generated: <ts ET> | Watchlist: <file> | Tickers: ... | Regime: SPY, QQQ
+  (Mechanical fetch + compute only — interpretation is stage work.)
+  ## 1. Market regime (SPY/QQQ)
+  | Ticker | Close | Chg % | O | H | L | VWAP | Vol | Vol vs prev | Closed vs VWAP |
+  ## 2. Watchlist snapshot
+  | Ticker | Close | Chg % | O | H | L | VWAP(all-day) | VWAP(RS) |
+  | Vol | Vol vs prev |  |  |  |  |  |  |
+  ## 3. Intraday shape — <ticker> (3-min bars, <date>)
+  Sessions PM/RS/AH: o/h/l/c/v/vwap each; high & low timestamps;
+  volume by RS hour; first-15-min and last-30-min moves.
+  ## 4. Daily structure — <ticker>
+  Close | MA20 (dist%) | MA50 (dist%) | period high/low
+  ## 5. Support/Resistance — <ticker>
+  daily + 3-min: major/minor levels w/ touches, key references
+  (pdh/pdl/pdc/period high-low).
+  ## 6. (v2) Catalyst feed — Benzinga per ticker, ET-sorted, with
+  offering/dilution auto-flag (regex on title: offering|ATM|S-3|
+  underwrit|registered direct|share sale|financing|capital raise).
+
+Format rules: prices 2dp (3dp if <2.0); volumes in M (1dp); facts only —
+"closed below VWAP" is allowed (arithmetic), "risk-off" is NOT (judgment).
+
+## 7. Stitching into CONTEXT.md (pending edits, after script works)
+
+- stages/01_broad_filter/CONTEXT.md: replace the manual fetch steps with
+  "run `python3 shared/scripts/fetch_scan_data.py <watchlist csv from
+  user_preferences.md>`; read output/data_brief.md; do NOT re-pull data the
+  brief contains (fallback to MCP calls only if the script fails)".
+  Agent keeps: regime interpretation, triage, shortlist.md.
+  (Benzinga: keep the existing agent news step for v1, unchanged.)
+- stages/02_confluence_analysis/CONTEXT.md: add input
+  ../01_broad_filter/output/data_brief.md — factors read from brief.
+- stages/03_trade_plan/CONTEXT.md: add same input — levels from brief.
+- Root CONTEXT.md "Shared Resources": update shared/ line to mention
+  scripts/fetch_scan_data.py. CLAUDE.md: one-line note under shared/.
+- Stages 00 and 04: UNTOUCHED (pure conversation/synthesis).
+
+## 8. Verification targets (tonight 8/18 run — script output must match)
+
+SPY 767.45 -0.68% VWAP 768.52 vol 43.9M | QQQ 717.51 -1.88% VWAP 719.73 vol 49.0M
+XOS 4.44 +96.65% O4.53 H4.95 L4.01 VWAP 4.4281(vol 110.1M) AH last ~4.11
+  3-min SR: resistance cluster 4.25/4.30/4.32 | daily SR: 3.04/2.70/2.11,
+  res 8.27, pdl 4.01 pdc 4.44 period_low 1.65
+AMLX 35.11 +63.42% O29.91 H35.39(ATH) L28.00 VWAP 31.4446 vol 24.0M AH ~35.02
+  3-min SR: support 35.02(48t)/34.90(28t), congestion 33.26/33.53
+  daily SR: 17.00/12.44 support, pdh 35.39
+
+## 9. Pitfalls / failure modes learned this session
+
+- file_write (and bash) DROP the payload when a single tool call carries a
+  very large content (~10KB+ observed failing). Symptom: "missing required
+  positional argument: 'content'". FIX: write in chunks of <= ~3KB
+  (file_write, or bash `cat > f <<'EOF'` then `cat >> f <<'EOF'`). Verified:
+  3 chunks landed cleanly.
+- ttg-stocks execute: 'multiplier' param is REQUIRED (500-ish error w/o it).
+- ttg-benzinga: HTTP 500 (HTML error page) occurred once; retry with
+  minimal params worked. Treat as flaky — never block the script on it.
+- 3-min bar timestamps: MILLISECONDS (not nanoseconds/seconds).
+- Node is unreachable from MARI bash (PATH isolation) — Python only.
+- Weekend/pre-market: daily bars may lack today; 3-min fetch should target
+  the last trading day found in daily bars, and brief must say so.
+- ICM standing rules (member's predicates): re-read ICM.md fresh every exec;
+  clear each stage's output/ before re-running the workflow; the workflow
+  folder is equities-only (options work was stripped 8/18 — do not re-add).
+
+## 10. Resume checklist (fresh session)
+
+1. Read this doc; read workflows/ICM.md (member predicate).
+2. Build fetch_scan_data.py per section 2 spec, using the client from
+   section 4 and shapes from section 5. Write via chunked bash heredocs.
+   Keep it stdlib-only. v1 has NO Benzinga.
+3. Run: python3 .../fetch_scan_data.py 260818.csv
+4. Diff its data_brief.md numbers against section 8 targets (must match).
+   Inspect raw_data.json to confirm snapshot field extraction (esp.
+   prevDay.v) — adjust .get() chains if fields differ.
+5. Make the CONTEXT.md stitch edits in section 7.
+6. Report to Gump with: script path, brief preview, stage-edit list,
+   and the Benzinga-v2 note (why it was deferred: flaky 500).
+7. Standing member rules: plan-first for consequential multi-step work;
+   visible progress updates; re-run = clear output/ first.
+
+## 11. Current state
+
+- Workflow folder: options work fully stripped (5 files edited 8/18,
+  verified clean). 8/18 evening run completed: all 5 stage outputs exist
+  in stages/*/output/ (shortlist, confluence, plans, summary card) with
+  AMLX primary / XOS conditional plans for Wed 8/19.
+- fetch_scan_data.py: NOT YET WRITTEN (all large file_write attempts
+  dropped). Probes above are the only code verified this session.
+- Nothing in the workflow folder is dirty beyond what 8/18 run wrote.