ite-workflows/sources/HANDOFF_scan_workflow_scripts.md

# HANDOFF — scan-watchlist-for-equities: mechanical data-fetch script

Written: Tue 2026-08-18 9:37 PM ET. Purpose: a fresh session can resume this
build without re-deriving anything. All API facts below were LIVE-VERIFIED
this session (not from memory).

## 1. What the member asked for

Gump: "I want to create scripts to do some of the work you're doing now and
stitch these into the workflow to avoid the inference work these stages
undertake on each run."

Follow-up approval: "yes build it and include Benzinga news pull" — then,
after the flaky Benzinga endpoint (HTTP 500) and dropped tool payloads:
"can you continue without benzinga angle?"

=> DECISION: **v1 script = price/volume/bars/S-R only (no Benzinga).**
News stays an optional agent MCP call at stage 01, exactly as before.
Benzinga re-add can be v2 once the endpoint proves stable.

## 2. Goal / design

One stdlib-only Python 3.11 script:
  `workflows/scan-watchlist-for-equities/shared/scripts/fetch_scan_data.py`

CLI:  python3 fetch_scan_data.py <watchlist.csv> [--out DIR]
         [--regime SPY,QQQ] [--bars-days 80] [--news-since YYYY-MM-DD]
       (news args drop in v2; ignore for v1)

It fetches, per watchlist ticker, DIRECTLY from TTG real-time data API
(via streamable-HTTP MCP JSON-RPC — see section 5):
  - stocks_snapshot_ticker
  - stocks_aggs_custom_bars daily  (from = today-100d, to = today,
    multiplier=1, timespan=day, limit=90)
  - stocks_aggs_custom_bars 3-min  (from/to = last trading day,
    multiplier=3, timespan=minute, limit=400)
  - calculate_support_resistance {ticker, timeframe: "daily"}
  - calculate_support_resistance {ticker, timeframe: "3min"}
Plus SPY/QQQ regime snapshots.

It computes (mechanical, zero interpretation):
  - session split of 3-min bars: PM (<09:30 ET), RS (09:30-16:00),
    AH (>16:00 ET) — per session: o/h/l/c/v and vwap=sum(c*v)/sum(v)
  - intraday high/low timestamps; volume by RS hour; first-15-min and
    last-30-min RS moves
  - MA20 / MA50 from daily closes + distance % (n/a if <20 bars: IPOs)
  - period high/low from fetched daily bars
  - vol vs prev-day volume (snapshot day.v / prevDay.v)

It writes into `stages/01_broad_filter/output/`:
  - data_brief.md   — formatted tables, the mechanical half of shortlist.md
  - raw_data.json   — every raw payload, audit trail
stdout: compact fetch summary + output paths. Exit 0/1.
Per-call: 1 retry, 2s backoff. Timestamps in bars are MILLISECONDS UTC —
convert to America/New_York (zoneinfo; fall back to local tz on error).
"Last trading day" for the 3-min fetch = max bar date from the daily bars
(handles weekend/pre-market runs).

Credentials: read at RUNTIME from env var MARI_MCP_CONFIG (machine-local
JSON; see section 4). NEVER hardcode keys or write them into the workflow
folder — the Cell syncs to Fleet.

## 3. Environment facts (verified this session)

- cwd = C:\Users\Eric\AppData\Roaming\MARI Desktop\localCell\accounts\
  acct-c6fef984c60eb59b9c1c4202ababa059\workspace  (git-bash paths use /c/...)
- Python 3.11.15 (pyenv) reachable from MARI bash; stdlib urllib HTTPS works
  (proven by live calls below). Node is UNREACHABLE from MARI bash — use Python.
- MCP config: env var MARI_MCP_CONFIG ->
  C:\Users\Eric\AppData\Roaming\MARI Desktop\mcp\acct-c6fef984...json
  Shape: {"servers": {"ttg-stocks": {"transport":"streamablehttp",
  "url":"https://rt.data.ttg.ai/stocks/mcp",
  "auth":{"type":"header","headers":{"Authorization":"Bearer ***",
  "X-User-ID":"27930"}}}, "ttg-benzinga": {...benzinga/mcp...}, ...}}
  (Do not print the real bearer key in chat/output.)
- Workflow root: .mari/scratchpad/workflows/scan-watchlist-for-equities/
  (Layer-3 scripts belong in shared/scripts/ per ICM Table 2; ICM.md is at
  workflows/ICM.md — re-read on exec per member's predicate.)
- Watchlist CSVs live in shared/ (e.g. 260818.csv = "XOS, AMLX" one per line).
  Script should resolve the CSV relative to shared/ (script_dir.parent).

## 4. MCP-over-HTTP protocol (proven working from Python)

Stateless on these servers: no Mcp-Session-Id returned. Sequence:
  1) POST url  {"jsonrpc":"2.0","id":1,"method":"initialize","params":
     {"protocolVersion":"2025-03-26","capabilities":{},
      "clientInfo":{"name":"fetch_scan_data","version":"1.0"}}}
  2) POST {"jsonrpc":"2.0","method":"notifications/initialized"}  (best-effort)
  3) POST {"jsonrpc":"2.0","id":N,"method":"tools/call","params":
     {"name":<tool>,"arguments":{...}}}
Headers for every POST: Content-Type: application/json,
Accept: application/json, text/event-stream, plus the server's auth headers.
Responses came back plain application/json (not SSE) in testing, but the
parser MUST handle both: if CT starts with text/event-stream, scan lines
starting "data:" and JSON-parse until one has the matching id.
Result text = d["result"]["content"][0]["text"] -> json.loads (it's a
JSON string). On JSON-RPC error object, raise with the message.

Minimal proven client (this exact code worked):
  cfg = json.load(open(os.environ['MARI_MCP_CONFIG']))
  srv = cfg['servers']['ttg-stocks']
  headers = {'Content-Type':'application/json',
             'Accept':'application/json, text/event-stream'}
  headers.update(srv['auth']['headers'])
  req = urllib.request.Request(srv['url'], data=json.dumps(payload).encode(),
             headers=headers, method='POST')
  body = urllib.request.urlopen(req, timeout=60).read().decode()

## 5. Verified response shapes (live, 8/18 evening)

stocks_snapshot_ticker {ticker:"SPY"} ->
  {"ticker":{"ticker":"SPY","todaysChangePerc":-0.675,"todaysChange":-5.22,
   "updated":1787097600000000000,"day":{"dv":43920901.2,"o":768.7,"h":769.5,
   "l":766.92,"c":767.45,"v":...,"vw":768.52,...}, ...}}
  (also has prevDay with c/v — code defensively with .get; the exact AH
  sub-fields were not captured: rely on 3-min AH bars for after-hours shape,
  not the snapshot.)

stocks_aggs_custom_bars {ticker,from,to,multiplier,timespan,limit} ->
  {"ticker":...,"results":[{"v":22219.6,"vw":768.25,"o":769.21,"c":768.37,
   "h":769.73,"l":767.8,"t":1787040000000,"n":446}],"next_url":...}
  NOTE: 'multiplier' IS REQUIRED (missing it = "Missing required params").
  Daily: multiplier=1 timespan=day. Intraday: multiplier=3 timespan=minute.
  t = MILLISECONDS UTC.

calculate_support_resistance {ticker, timeframe:"daily"|"3min"} ->
  {"timeframe":"daily","current_price":4.44,
   "support_levels":{"minor":[{"price":1.92,"touches":1,
     "description":"Swing low 3 weeks ago, 1 touch"}],
    "major":[{"price":3.04,"touches":3,"description":"..."}, ...],
    "key_references":{"pdl":4.01,"pdc":4.44,"period_low":1.65}},
   "resistance_levels":{"minor":[...],"major":[...],
    "key_references":{"pdh":4.95,"period_high":...}}}

benzinga_news (V2 ONLY — endpoint flaky, 500'd once this session):
  ttg-benzinga execute {endpoint:"benzinga_news",
  params:{"stocks":"AMLX","published.gte":"2026-08-18"}}  ->
  {"status":"OK","results":[{"benzinga_id":...,"author":"benzinga newsdesk",
   "published":"2026-08-18T20:02:46Z","title":"Amylyx Pharmaceuticals
   Announces $350M Common Stock Offering","url":"...","channels":[...],...}]}
  Use ONLY minimal params (stocks + published.gte) — extra params
  (limit/sort) coincided with the 500; retry once on failure.

## 6. data_brief.md format (mirror the mechanical sections of tonight's
shortlist.md so stage agents can drop them straight in)

  # Data Brief — scan-watchlist-for-equities
  Generated: <ts ET> | Watchlist: <file> | Tickers: ... | Regime: SPY, QQQ
  (Mechanical fetch + compute only — interpretation is stage work.)
  ## 1. Market regime (SPY/QQQ)
  | Ticker | Close | Chg % | O | H | L | VWAP | Vol | Vol vs prev | Closed vs VWAP |
  ## 2. Watchlist snapshot
  | Ticker | Close | Chg % | O | H | L | VWAP(all-day) | VWAP(RS) |
  | Vol | Vol vs prev |  |  |  |  |  |  |
  ## 3. Intraday shape — <ticker> (3-min bars, <date>)
  Sessions PM/RS/AH: o/h/l/c/v/vwap each; high & low timestamps;
  volume by RS hour; first-15-min and last-30-min moves.
  ## 4. Daily structure — <ticker>
  Close | MA20 (dist%) | MA50 (dist%) | period high/low
  ## 5. Support/Resistance — <ticker>
  daily + 3-min: major/minor levels w/ touches, key references
  (pdh/pdl/pdc/period high-low).
  ## 6. (v2) Catalyst feed — Benzinga per ticker, ET-sorted, with
  offering/dilution auto-flag (regex on title: offering|ATM|S-3|
  underwrit|registered direct|share sale|financing|capital raise).

Format rules: prices 2dp (3dp if <2.0); volumes in M (1dp); facts only —
"closed below VWAP" is allowed (arithmetic), "risk-off" is NOT (judgment).

## 7. Stitching into CONTEXT.md (pending edits, after script works)

- stages/01_broad_filter/CONTEXT.md: replace the manual fetch steps with
  "run `python3 shared/scripts/fetch_scan_data.py <watchlist csv from
  user_preferences.md>`; read output/data_brief.md; do NOT re-pull data the
  brief contains (fallback to MCP calls only if the script fails)".
  Agent keeps: regime interpretation, triage, shortlist.md.
  (Benzinga: keep the existing agent news step for v1, unchanged.)
- stages/02_confluence_analysis/CONTEXT.md: add input
  ../01_broad_filter/output/data_brief.md — factors read from brief.
- stages/03_trade_plan/CONTEXT.md: add same input — levels from brief.
- Root CONTEXT.md "Shared Resources": update shared/ line to mention
  scripts/fetch_scan_data.py. CLAUDE.md: one-line note under shared/.
- Stages 00 and 04: UNTOUCHED (pure conversation/synthesis).

## 8. Verification targets (tonight 8/18 run — script output must match)

SPY 767.45 -0.68% VWAP 768.52 vol 43.9M | QQQ 717.51 -1.88% VWAP 719.73 vol 49.0M
XOS 4.44 +96.65% O4.53 H4.95 L4.01 VWAP 4.4281(vol 110.1M) AH last ~4.11
  3-min SR: resistance cluster 4.25/4.30/4.32 | daily SR: 3.04/2.70/2.11,
  res 8.27, pdl 4.01 pdc 4.44 period_low 1.65
AMLX 35.11 +63.42% O29.91 H35.39(ATH) L28.00 VWAP 31.4446 vol 24.0M AH ~35.02
  3-min SR: support 35.02(48t)/34.90(28t), congestion 33.26/33.53
  daily SR: 17.00/12.44 support, pdh 35.39

## 9. Pitfalls / failure modes learned this session

- file_write (and bash) DROP the payload when a single tool call carries a
  very large content (~10KB+ observed failing). Symptom: "missing required
  positional argument: 'content'". FIX: write in chunks of <= ~3KB
  (file_write, or bash `cat > f <<'EOF'` then `cat >> f <<'EOF'`). Verified:
  3 chunks landed cleanly.
- ttg-stocks execute: 'multiplier' param is REQUIRED (500-ish error w/o it).
- ttg-benzinga: HTTP 500 (HTML error page) occurred once; retry with
  minimal params worked. Treat as flaky — never block the script on it.
- 3-min bar timestamps: MILLISECONDS (not nanoseconds/seconds).
- Node is unreachable from MARI bash (PATH isolation) — Python only.
- Weekend/pre-market: daily bars may lack today; 3-min fetch should target
  the last trading day found in daily bars, and brief must say so.
- ICM standing rules (member's predicates): re-read ICM.md fresh every exec;
  clear each stage's output/ before re-running the workflow; the workflow
  folder is equities-only (options work was stripped 8/18 — do not re-add).

## 10. Resume checklist (fresh session)

1. Read this doc; read workflows/ICM.md (member predicate).
2. Build fetch_scan_data.py per section 2 spec, using the client from
   section 4 and shapes from section 5. Write via chunked bash heredocs.
   Keep it stdlib-only. v1 has NO Benzinga.
3. Run: python3 .../fetch_scan_data.py 260818.csv
4. Diff its data_brief.md numbers against section 8 targets (must match).
   Inspect raw_data.json to confirm snapshot field extraction (esp.
   prevDay.v) — adjust .get() chains if fields differ.
5. Make the CONTEXT.md stitch edits in section 7.
6. Report to Gump with: script path, brief preview, stage-edit list,
   and the Benzinga-v2 note (why it was deferred: flaky 500).
7. Standing member rules: plan-first for consequential multi-step work;
   visible progress updates; re-run = clear output/ first.

## 11. Current state

- Workflow folder: options work fully stripped (5 files edited 8/18,
  verified clean). 8/18 evening run completed: all 5 stage outputs exist
  in stages/*/output/ (shortlist, confluence, plans, summary card) with
  AMLX primary / XOS conditional plans for Wed 8/19.
- fetch_scan_data.py: NOT YET WRITTEN (all large file_write attempts
  dropped). Probes above are the only code verified this session.
- Nothing in the workflow folder is dirty beyond what 8/18 run wrote.