ite-workflows/sources/HANDOFF_scan_workflow_scripts.md

HANDOFF — scan-watchlist-for-equities: mechanical data-fetch script

Written: Tue 2026-08-18 9:37 PM ET. Purpose: a fresh session can resume this build without re-deriving anything. All API facts below were LIVE-VERIFIED this session (not from memory).

1. What the member asked for

Gump: "I want to create scripts to do some of the work you're doing now and stitch these into the workflow to avoid the inference work these stages undertake on each run."

Follow-up approval: "yes build it and include Benzinga news pull" — then, after the flaky Benzinga endpoint (HTTP 500) and dropped tool payloads: "can you continue without benzinga angle?"

=> DECISION: v1 script = price/volume/bars/S-R only (no Benzinga). News stays an optional agent MCP call at stage 01, exactly as before. Benzinga re-add can be v2 once the endpoint proves stable.

2. Goal / design

One stdlib-only Python 3.11 script: workflows/scan-watchlist-for-equities/shared/scripts/fetch_scan_data.py

CLI: python3 fetch_scan_data.py <watchlist.csv> [--out DIR] [--regime SPY,QQQ] [--bars-days 80] [--news-since YYYY-MM-DD] (news args drop in v2; ignore for v1)

It fetches, per watchlist ticker, DIRECTLY from TTG real-time data API (via streamable-HTTP MCP JSON-RPC — see section 5):

• stocks_snapshot_ticker
• stocks_aggs_custom_bars daily (from = today-100d, to = today, multiplier=1, timespan=day, limit=90)
• stocks_aggs_custom_bars 3-min (from/to = last trading day, multiplier=3, timespan=minute, limit=400)
• calculate_support_resistance {ticker, timeframe: "daily"}
• calculate_support_resistance {ticker, timeframe: "3min"} Plus SPY/QQQ regime snapshots.

It computes (mechanical, zero interpretation):

• session split of 3-min bars: PM (<09:30 ET), RS (09:30-16:00), AH (>16:00 ET) — per session: o/h/l/c/v and vwap=sum(c*v)/sum(v)
• intraday high/low timestamps; volume by RS hour; first-15-min and last-30-min RS moves
• MA20 / MA50 from daily closes + distance % (n/a if <20 bars: IPOs)
• period high/low from fetched daily bars
• vol vs prev-day volume (snapshot day.v / prevDay.v)

It writes into stages/01_broad_filter/output/:

• data_brief.md — formatted tables, the mechanical half of shortlist.md
• raw_data.json — every raw payload, audit trail stdout: compact fetch summary + output paths. Exit 0/1. Per-call: 1 retry, 2s backoff. Timestamps in bars are MILLISECONDS UTC — convert to America/New_York (zoneinfo; fall back to local tz on error). "Last trading day" for the 3-min fetch = max bar date from the daily bars (handles weekend/pre-market runs).

Credentials: read at RUNTIME from env var MARI_MCP_CONFIG (machine-local JSON; see section 4). NEVER hardcode keys or write them into the workflow folder — the Cell syncs to Fleet.

3. Environment facts (verified this session)

• cwd = C:\Users\Eric\AppData\Roaming\MARI Desktop\localCell\accounts
  acct-c6fef984c60eb59b9c1c4202ababa059\workspace (git-bash paths use /c/...)
• Python 3.11.15 (pyenv) reachable from MARI bash; stdlib urllib HTTPS works (proven by live calls below). Node is UNREACHABLE from MARI bash — use Python.
• MCP config: env var MARI_MCP_CONFIG -> C:\Users\Eric\AppData\Roaming\MARI Desktop\mcp\acct-c6fef984...json Shape: {"servers": {"ttg-stocks": {"transport":"streamablehttp", "url":"https://rt.data.ttg.ai/stocks/mcp", "auth":{"type":"header","headers":{"Authorization":"Bearer ***", "X-User-ID":"27930"}}}, "ttg-benzinga": {...benzinga/mcp...}, ...}} (Do not print the real bearer key in chat/output.)
• Workflow root: .mari/scratchpad/workflows/scan-watchlist-for-equities/ (Layer-3 scripts belong in shared/scripts/ per ICM Table 2; ICM.md is at workflows/ICM.md — re-read on exec per member's predicate.)
• Watchlist CSVs live in shared/ (e.g. 260818.csv = "XOS, AMLX" one per line). Script should resolve the CSV relative to shared/ (script_dir.parent).

4. MCP-over-HTTP protocol (proven working from Python)

Stateless on these servers: no Mcp-Session-Id returned. Sequence:

1. POST url {"jsonrpc":"2.0","id":1,"method":"initialize","params": {"protocolVersion":"2025-03-26","capabilities":{}, "clientInfo":{"name":"fetch_scan_data","version":"1.0"}}}
2. POST {"jsonrpc":"2.0","method":"notifications/initialized"} (best-effort)
3. POST {"jsonrpc":"2.0","id":N,"method":"tools/call","params": {"name":,"arguments":{...}}} Headers for every POST: Content-Type: application/json, Accept: application/json, text/event-stream, plus the server's auth headers. Responses came back plain application/json (not SSE) in testing, but the parser MUST handle both: if CT starts with text/event-stream, scan lines starting "data:" and JSON-parse until one has the matching id. Result text = d["result"]["content"][0]["text"] -> json.loads (it's a JSON string). On JSON-RPC error object, raise with the message.

Minimal proven client (this exact code worked): cfg = json.load(open(os.environ['MARI_MCP_CONFIG'])) srv = cfg['servers']['ttg-stocks'] headers = {'Content-Type':'application/json', 'Accept':'application/json, text/event-stream'} headers.update(srv['auth']['headers']) req = urllib.request.Request(srv['url'], data=json.dumps(payload).encode(), headers=headers, method='POST') body = urllib.request.urlopen(req, timeout=60).read().decode()

5. Verified response shapes (live, 8/18 evening)

stocks_snapshot_ticker {ticker:"SPY"} -> {"ticker":{"ticker":"SPY","todaysChangePerc":-0.675,"todaysChange":-5.22, "updated":1787097600000000000,"day":{"dv":43920901.2,"o":768.7,"h":769.5, "l":766.92,"c":767.45,"v":...,"vw":768.52,...}, ...}} (also has prevDay with c/v — code defensively with .get; the exact AH sub-fields were not captured: rely on 3-min AH bars for after-hours shape, not the snapshot.)

stocks_aggs_custom_bars {ticker,from,to,multiplier,timespan,limit} -> {"ticker":...,"results":[{"v":22219.6,"vw":768.25,"o":769.21,"c":768.37, "h":769.73,"l":767.8,"t":1787040000000,"n":446}],"next_url":...} NOTE: 'multiplier' IS REQUIRED (missing it = "Missing required params"). Daily: multiplier=1 timespan=day. Intraday: multiplier=3 timespan=minute. t = MILLISECONDS UTC.

calculate_support_resistance {ticker, timeframe:"daily"|"3min"} -> {"timeframe":"daily","current_price":4.44, "support_levels":{"minor":[{"price":1.92,"touches":1, "description":"Swing low 3 weeks ago, 1 touch"}], "major":[{"price":3.04,"touches":3,"description":"..."}, ...], "key_references":{"pdl":4.01,"pdc":4.44,"period_low":1.65}}, "resistance_levels":{"minor":[...],"major":[...], "key_references":{"pdh":4.95,"period_high":...}}}

benzinga_news (V2 ONLY — endpoint flaky, 500'd once this session): ttg-benzinga execute {endpoint:"benzinga_news", params:{"stocks":"AMLX","published.gte":"2026-08-18"}} -> {"status":"OK","results":[{"benzinga_id":...,"author":"benzinga newsdesk", "published":"2026-08-18T20:02:46Z","title":"Amylyx Pharmaceuticals Announces $350M Common Stock Offering","url":"...","channels":[...],...}]} Use ONLY minimal params (stocks + published.gte) — extra params (limit/sort) coincided with the 500; retry once on failure.

6. data_brief.md format (mirror the mechanical sections of tonight's

shortlist.md so stage agents can drop them straight in)

Data Brief — scan-watchlist-for-equities

Generated: | Watchlist: | Tickers: ... | Regime: SPY, QQQ (Mechanical fetch + compute only — interpretation is stage work.)

1. Market regime (SPY/QQQ)

| Ticker | Close | Chg % | O | H | L | VWAP | Vol | Vol vs prev | Closed vs VWAP |

2. Watchlist snapshot

| Ticker | Close | Chg % | O | H | L | VWAP(all-day) | VWAP(RS) | | Vol | Vol vs prev | | | | | | |

3. Intraday shape — (3-min bars, )

Sessions PM/RS/AH: o/h/l/c/v/vwap each; high & low timestamps; volume by RS hour; first-15-min and last-30-min moves.

4. Daily structure —

Close | MA20 (dist%) | MA50 (dist%) | period high/low

5. Support/Resistance —

daily + 3-min: major/minor levels w/ touches, key references (pdh/pdl/pdc/period high-low).

6. (v2) Catalyst feed — Benzinga per ticker, ET-sorted, with

offering/dilution auto-flag (regex on title: offering|ATM|S-3| underwrit|registered direct|share sale|financing|capital raise).

Format rules: prices 2dp (3dp if <2.0); volumes in M (1dp); facts only — "closed below VWAP" is allowed (arithmetic), "risk-off" is NOT (judgment).

7. Stitching into CONTEXT.md (pending edits, after script works)

• stages/01_broad_filter/CONTEXT.md: replace the manual fetch steps with "run python3 shared/scripts/fetch_scan_data.py <watchlist csv from user_preferences.md>; read output/data_brief.md; do NOT re-pull data the brief contains (fallback to MCP calls only if the script fails)". Agent keeps: regime interpretation, triage, shortlist.md. (Benzinga: keep the existing agent news step for v1, unchanged.)
• stages/02_confluence_analysis/CONTEXT.md: add input ../01_broad_filter/output/data_brief.md — factors read from brief.
• stages/03_trade_plan/CONTEXT.md: add same input — levels from brief.
• Root CONTEXT.md "Shared Resources": update shared/ line to mention scripts/fetch_scan_data.py. CLAUDE.md: one-line note under shared/.
• Stages 00 and 04: UNTOUCHED (pure conversation/synthesis).

8. Verification targets (tonight 8/18 run — script output must match)

SPY 767.45 -0.68% VWAP 768.52 vol 43.9M | QQQ 717.51 -1.88% VWAP 719.73 vol 49.0M XOS 4.44 +96.65% O4.53 H4.95 L4.01 VWAP 4.4281(vol 110.1M) AH last ~4.11 3-min SR: resistance cluster 4.25/4.30/4.32 | daily SR: 3.04/2.70/2.11, res 8.27, pdl 4.01 pdc 4.44 period_low 1.65 AMLX 35.11 +63.42% O29.91 H35.39(ATH) L28.00 VWAP 31.4446 vol 24.0M AH ~35.02 3-min SR: support 35.02(48t)/34.90(28t), congestion 33.26/33.53 daily SR: 17.00/12.44 support, pdh 35.39

9. Pitfalls / failure modes learned this session

• file_write (and bash) DROP the payload when a single tool call carries a very large content (~10KB+ observed failing). Symptom: "missing required positional argument: 'content'". FIX: write in chunks of <= ~3KB (file_write, or bash cat > f <<'EOF' then cat >> f <<'EOF'). Verified: 3 chunks landed cleanly.
• ttg-stocks execute: 'multiplier' param is REQUIRED (500-ish error w/o it).
• ttg-benzinga: HTTP 500 (HTML error page) occurred once; retry with minimal params worked. Treat as flaky — never block the script on it.
• 3-min bar timestamps: MILLISECONDS (not nanoseconds/seconds).
• Node is unreachable from MARI bash (PATH isolation) — Python only.
• Weekend/pre-market: daily bars may lack today; 3-min fetch should target the last trading day found in daily bars, and brief must say so.
• ICM standing rules (member's predicates): re-read ICM.md fresh every exec; clear each stage's output/ before re-running the workflow; the workflow folder is equities-only (options work was stripped 8/18 — do not re-add).

10. Resume checklist (fresh session)

1. Read this doc; read workflows/ICM.md (member predicate).
2. Build fetch_scan_data.py per section 2 spec, using the client from section 4 and shapes from section 5. Write via chunked bash heredocs. Keep it stdlib-only. v1 has NO Benzinga.
3. Run: python3 .../fetch_scan_data.py 260818.csv
4. Diff its data_brief.md numbers against section 8 targets (must match). Inspect raw_data.json to confirm snapshot field extraction (esp. prevDay.v) — adjust .get() chains if fields differ.
5. Make the CONTEXT.md stitch edits in section 7.
6. Report to Gump with: script path, brief preview, stage-edit list, and the Benzinga-v2 note (why it was deferred: flaky 500).
7. Standing member rules: plan-first for consequential multi-step work; visible progress updates; re-run = clear output/ first.

11. Current state

• Workflow folder: options work fully stripped (5 files edited 8/18, verified clean). 8/18 evening run completed: all 5 stage outputs exist in stages/*/output/ (shortlist, confluence, plans, summary card) with AMLX primary / XOS conditional plans for Wed 8/19.
• fetch_scan_data.py: NOT YET WRITTEN (all large file_write attempts dropped). Probes above are the only code verified this session.
• Nothing in the workflow folder is dirty beyond what 8/18 run wrote.