Observability Map — Nexus Scalp Engine (NSE)

TASK-OBS-AUDIT (2026-08-31, HEAD baseline 1f60832; delivered at dcc80d7).

Agent: Hermes-ObsForensic. AUDIT-ONLY: zero runtime code changed.

Machine-readable companion: artifacts/forensics/observability-audit.json.

Method: black-box reconstruction from runtime evidence (logs / DBs / artifacts /

live API / CLI) FIRST, then targeted read-only code sweeps for every claim.

Evidence markers: [L]=live-probed today, [DB]=DB census, [LOG]=log sample, [C]=code file:line.

1. What the system actually produces today

2. Forensic identity chain (as-built)

request_id (uuid, policy) ── doubles as broker client order_id
   → TradeProposal → [EXEC_TRACE] log (id REDACTED today) → audit_signals.request_id
execution_id (EXEC-YYYYMMDD-HHMMSS-xxxxxx)
   → generated per decision → embedded into audit_orders.reason → audit_orders.execution_id
   → /api/debug/trace/{execution_id} join endpoint
ticket (broker) → position_lifecycle_events.sequence → audit_ledger → trade_autopsies
experience_id = "exp_"+request_id → audit_experiences (execution_id column 0% filled [C intelligence.py:556-573])
update correlation_id (upd-…) → update-state.json only (not logged)
deploy-gate correlation_id (fh-…) → deploy_gate_result.json only
per-boot correlation: NONE (no boot/run id; correlation contextvars never bound)

Verdict: chain is PARTIAL and broken at the log↔DB seam. DB side is strong (request_id 100% in signals/experiences; EXEC- ids on 2026-08-20 orders); the log side destroys the same ids (OBS-002). Live-probed SELL_LIMIT lifecycle EXEC-20260831-175500-a92169: signal→order→lifecycle seq 83..87→ledger CLOSED +6.24 joined cleanly through the DB alone; the log side offers only second-granularity time correlation.

3. Subsystem observability map (required columns per task §46)

4. Trace matrix (§47) — reconstructable? (evidence basis: black-box pass)

5. Black-box reconstruction results (10 scenarios, runtime evidence only)

1. Engine startup 19:26 boot — YES: config/mode/model fingerprint/scaler(70)/MT5/reconciliation trail.
2. Failed startup (MT5 auth) 16:59 — YES-PARTIAL: 3× retcode(-6) + critical; no attempt id/boot id.
3. One inference — PARTIAL: EXEC_TRACE + audit_signals rich, but ids redacted in log; no inference lifecycle events.
4. Failed inference — PARTIAL: "Async retrain failed … mat and mat shapes cannot be multiplied" — no shapes/dtype/device/op-id; scaler refit warning has good expected-vs-actual but sits at WARNING while changing runtime identity.
5. One update — NO: json terminal state only; correlation id never logged; stages invisible.
6. Failed update — PARTIAL: FAILED_SAFE + upd-id + timestamp; nothing before it.
7. One CLI op — PARTIAL: strong JSON surfaces; no per-invocation op id; CLI self-identity stale (commit 53317de vs HEAD) [L].
8. One API op — YES-PARTIAL: request-id echo + state/freshness/health rich; debug/trace endpoint DEAD in running instance (OBS-001).
9. Worker failure — YES: WORKER_KICK + heartbeat/state tables + news_health per-source.
10. Order lifecycle — YES via DB chain (execution_id join); NO via logs alone (redacted ids).

Plus: "SELL then disappeared" black-box question — answerable from DB (signal 17:55:00Z → order → POSITION_EXITED 17:56:38Z → ledger +6.24, reason BROKER_DEAL_REASON conf 0.8), NOT from logs.

6. Gap ledger (OBS-001..016; full detail in observability-audit.json)

• P0 OBS-002 (redactor eats EXEC- correlation ids — 2,631/2,631 EXEC_TRACE lines affected [C logging.py:161-163]) · OBS-003 (correlation ids never bound: bind_correlation_id 0 callers [C logging.py:691]).
• P1 OBS-001 (/api/debug/trace dead import nexus_scalp.adapters.audit_db while /health=READY [C web/server.py:4769]) · OBS-004 (audit write failures/drops lack identity; rows silently dropped) · OBS-005 (EXPERIENCE rows written with execution_id 0% [C experience/intelligence.py:556-573]) · OBS-006 (update chain: correlation id json-only, stage transitions unlogged, release_metadata empty).
• P2 OBS-007 shutdown/stop evidence absent · OBS-008 dev-run boots leave no pidfile/start id (BUG-170 covers nexus start only) · OBS-009 three time dialects (log +03:30 / DB UTC / broker epoch GMT+3, BUG-070) · OBS-010 audit_orders.latency 81% zeros; execution latency not traceable · OBS-011 async task failures unattributed (no task id/parent) · OBS-012 SSE events lack producer/op id.
• P3 OBS-013 strategy-factory / dataset-rejection floods (15,289 + 20,373 identical lines/day) · OBS-014 CLI/health identity drift (version 9.0.3/commit 53317de vs HEAD 9.0.5) · OBS-015 request-id not persisted into logs/DB · OBS-016 CLI ops expose no operation id/timestamps.

7. Reconstruction scorecard

STARTUP YELLOW · PROCESS RED · DATABASE RED · MARKET DATA YELLOW · CANDLES GREEN (DB) · FEATURES YELLOW · 50D YELLOW · 70D YELLOW · TENSOR RED · MODEL GREEN · REGIME GREEN · LIQUIDITY GREEN · STRATEGY YELLOW · RISK YELLOW · EXECUTION YELLOW (DB-side GREEN, log-side RED) · WORKERS GREEN · ASYNC RED · CLI YELLOW · WEB/API GREEN · SSE YELLOW · UPDATER RED · ROLLBACK YELLOW · RELEASE YELLOW · SHUTDOWN RED.

Overall: PARTIALLY OBSERVABLE. The DB layer is the system's real flight recorder; the log layer loses identity at the exact seam where an incident investigator needs it (log↔DB join via correlation ids).

8. Recommended future fixes (NOT implemented here — audit-only mandate)

1. Allowlist id-shaped values in _redact_value (EXEC-/req_/upd-/fh-/INC- prefixes) — OBS-002.
2. Bind correlation context per boot + per decision (wire bind_correlation_id) — OBS-003.
3. Fix adapters.audit_db import in /api/debug/trace (or fail health visibly) — OBS-001.
4. Persist stage transitions for update/rollback into logs + release_metadata — OBS-006.
5. Structured DB-write failure events (table, rows, duration, drop counters) — OBS-004.
6. Thread execution_id through to audit_experiences — OBS-005.
7. Emit SHUTDOWN/STOPPED + explicit STARTUP_COMPLETE markers with a boot id — OBS-007.
8. Deduplicate/escalate flood hotspots; demote WORKER_KICK repeats to state rows — OBS-013.