Causal monitoring

What causal monitoring is

Traditional tracing answers: what happened, in what order, and how long did it take?

Causal monitoring answers: which agent introduced the fault, and why did the workflow fail?

blamr records a CausalEdge at every agent handoff — not just latency and tokens:

Those signals form a directed causal graph per run. When the run completes, workers walk that graph backward and assign fault-weighted blame percentages to each agent.

End-to-end pipeline

Agents ──CausalEdge/hop──► Ingest (:3001)
                              │
                              ├──► Kafka edges.raw ──► ClickHouse writer ──► causal_edges
                              │
                              └──► POST runs/complete ──► runs.completed
                                        │
                                        ▼ (settle ~2s)
                                   blame.needed ──► Blame processor
                                        │
                                        ├──► Postgres (workflow_runs, blame_reports)
                                        └──► blame.completed ──► Webhooks / UI

Dashboard (:8080) ◄── API (:3000) ◄── Postgres

1. Instrument — SDK, MCP proxy, or adapter emits one CausalEdge per hop.
2. Ingest — Validates API key, Merkle-hashes edges, publishes to edges.raw. Returns 202 immediately.
3. Edge writer — Batches edges into ClickHouse; optional semantic drift on flush.
4. Complete run — Agent calls POST /v1/runs/:run_id/complete with success or failed.
5. Run aggregator — Waits BLAMR_SEMANTIC_SETTLE_MS (default 2s), then publishes blame.needed.
6. Blame processor — Enriches signals, evaluates confidence gate, computes blame, persists results.
7. Dashboard — Renders causal graph, trace, cost, and blame/attribution tabs.

The unit of observation: CausalEdge

Each edge is one handoff in a multi-agent workflow, ordered by hop_index and linked via prev_hash → edge_hash (SHA-256 audit chain).

Agent A  ──edge──►  Agent B
  hop_index: 0
  confidence_in:   what B received with
  confidence_out:  what B emitted
  intent_delta:    goal preservation on this hop
  influence_score: downstream causal weight

Agent-side signal computation

The TypeScript SDK (@blamr/sdk) computes hop signals via computeHopSignals():

• Intent delta — from domain alignment or retrieval relevance (aligned ≈ −0.02; weak retrieval −0.15 to −0.35).
• Confidence out — composite of lexical confidence, structured JSON fields, tool scores, alignment ceiling, and upstream cap.

Agents emit honest I/O + scores; the platform derives causal primitives — no manual blame guessing required.

Run lifecycle and status

Business failure vs confidence gate

If the agent reports success but the gate fails, workers upgrade the run to failed and set error_summary from the gate reason.

Confidence gate modes

Default accept level: 0.70. Configure in SDK or Settings → Workspace → Workflow profiles.

Post-run enrichment

Semantic drift (BLAMR_SEMANTIC_DRIFT=true)

Embeds I/O previews via local Ollama. Compares tool input vs output and run goal vs downstream output. In telemetry-first mode (default):

• intent_delta = min(agent_reported, semantic_value)
• confidence_out = min(reported, semantic_similarity_ceiling)

ML drift + ranker (BLAMR_ML_ENABLED=true)

Accuracy score

Each completed run gets accuracy_score (0–1) for workflow health bands and Overview KPIs.

if no edges:
  success → 0.9 ; failed → 0.4
else:
  base = last_hop.confidence_out
  success → clamp(base, 0.65 … 0.99)
  failed  → clamp(base × 0.55, 0.25 … 0.65)

Accuracy reflects final-hop confidence tempered by outcome — failed runs score lower even when last-hop confidence was high.

Blame attribution

Blame answers: of the total fault in this run, what fraction belongs to each agent?

Fault signals per hop

Blame weights

localFault = intentHarm × 3 + confDrop × 2 + inflation × 2

if run failed:
  weight = influence_score × (localFault + 0.05)
else:
  weight = influence_score × 0.5   // influence, not fault

Normalize per agent → percentages (sum = 100%)

Design principle: blame follows where fault was introduced, not hop order alone or token count.

Failed vs successful runs

When ML is enabled on failed runs: fused_pct = (1 − α) × rule_pct + α × ml_pct (default α = 0.55). Optional LLM rewrites reasons citing trace I/O.

Dashboard business logic

Workflow health bands

Agent connection status

Run detail tabs

Event topics

Configuration map

Full markdown reference: docs/CAUSAL_MONITORING.md

Mental model for operators

1. Instrument every handoff — no edge means invisible to blame.
2. Complete every run — without completeRun(), blame never fires.
3. Failed ≠ one bad agent — blame % shows contribution, including propagation.
4. Success runs still have signal — attribution + inflation flags per hop.
5. Health bands lag reality — they aggregate historical accuracy.