Observability

NSE treats observability as a subsystem with contracts, not a logging afterthought. If the engine cannot show why it did something, its behavior is indistinguishable from a bug — so showing why is engineered, not hoped for.

Structured logging (`observability/logging.py`)

Severity-split, date-organized storage: logs/<sev>/YYYY/MM/YYYY-MM-DD.log.
ISO-8601 timestamps (broker-local offset aware), event/component/category/

error-code fields, correlation context.

Daily + size rotation with zero-loss part files; per-severity retention.
Key + high-entropy redaction; ANSI-free stacks.

Incident response (`incidents/`)

Canonical incident model (INC-YYYY-hex8, 8 statuses, 5 severities), fingerprint correlation, causal-chain timeline reconstruction, value lineage with first-divergence location, WHY workflows, MT5/ledger divergence + clock-skew forensics. Diagnostic-only by contract: incidents never mutate trading/risk/models/DB. Recovery plans are RECOMMENDED and approval-gated — no destructive options. CLI: nexus incidents …; Telegram alerts for CRITICAL/HIGH (throttled, deduped).

Forensic health (`forensics/`)

ForensicHealthEngine — read-only continuous checks (INV-70D-001..020):

feature contract/deadness/drift, causal + parity canaries, model/scaler contract, DB integrity, duplicate economic outcomes, impossible excursions, silent fallback, "200-but-wrong" classification, worker no-progress, UI/API consistency, chart health, queue growth.

Deploy gate (nexus forensic --deploy-gate, wired into beforePush):

verdicts PASS/ALLOW · WARNING/ALLOW_WITH_WARNING · DEGRADED/REVIEW · CRITICAL/BLOCK · UNKNOWN/REVIEW (never passes). A release cannot ship green while the gate says BLOCK — the gate, not a human's optimism, decides.

Debug console

/api/debug/state canonical snapshot (18 sections) + registry-driven 70D feature matrix + contract validation + snapshot diff. Execution traces via /api/debug/trace. The UI Debug Hub renders the same data the API serves — no separate truth.

What observability caught (evidence, not theory)

BUG-105: dead shadow hook code found by flow forensics.
The 154× crash-loop of an old process (mat1 Nx50 vs 70x128) surfaced through

log forensics and became the BUG-185 record-contract fix.

"200-but-wrong" news classification (HTTP_SUCCESS_EMPTY) — a server that

answers 200 with garbage is worse than a 500; the health engine now classifies it.

Diagnostics export

nexus export-diagnostics builds a sanitized ZIP — never contains secrets.

Known gaps (tracked, not hidden)

The observability audit maintains a public gap ledger (OBS-001…016 — e.g. redaction eating correlation IDs on some paths, silent DB batch drops). See docs/architecture/observability-map.md in the repository.

Observability

Structured logging (observability/logging.py)

Incident response (incidents/)

Forensic health (forensics/)