“Why Did This Happen?” — Root-Cause Investigation Agent
An agent that answers questions like “Why did revenue fall last week?” by investigating like an analyst — with every number in its report backed by evidence.
Overview
A root-cause investigation agent. A user asks a business question and the system verifies the change is real, decomposes the metric, generates falsifiable hypotheses, tests each one against company data with read-only tools, actively hunts for counter-evidence, quantifies impact and writes a cited report. It is deliberately not a dashboard and not a chatbot over a database.
The Problem
When a key metric drops, analysts spend hours slicing data to find out why. Generic “chat with your data” tools are fast but unreliable: LLMs confidently invent numbers, confirm the first plausible story and can run unsafe queries. Teams need answers they can trust and audit.
Solution
A plain-Python orchestrator splits the work strictly: deterministic Python/SQL computes baselines, significance tests, decomposition and impact; the LLM only reasons over those outputs. Every tool result is written to an evidence ledger, and a citation checker verifies every number in the final report against it — a mismatch rejects the report. “Inconclusive” is a valid outcome, so the agent never forces an answer.
Architecture
Key Features
- →Verifies a change is statistically real before investigating it
- →Falsifiable hypotheses — each states its predictions and what would disprove it
- →Skeptic agent that actively searches for counter-evidence
- →Evidence ledger + citation checker: any unverified number rejects the report
- →Read-only, allow-listed data tools — no free-form SQL from the LLM
- →Strict JSON in/out validated by Pydantic, with one repair retry then a typed error
- →Seeded, traced and reproducible runs
Tech Stack
Key Decisions & Challenges
Stopping the LLM from inventing numbers
LLMs will happily state figures they never computed. The rule is enforced in code: the model may only state numbers that came back from a tool result, and the citation checker compares every number in the report against the evidence ledger before it is accepted.
Avoiding confirmation bias
A single agent tends to accept the first plausible story. A separate skeptic step tries to disprove each hypothesis, and “inconclusive” is treated as a valid result rather than a failure.
Safe data access
Letting an LLM write SQL is risky. In v1 the agent can only call typed, allow-listed tools, running under a database role that has SELECT permission only.
Testing & Quality
- ✓Hermetic pytest suite — runs without a database or network
- ✓ruff linting and mypy in strict mode on the whole backend
- ✓Evaluation harness comparing the agent against a naive baseline on seeded scenarios
- ✓Append-only decision log documenting every scoping choice
Lessons Learned
- →Reliability comes from what you don’t let the LLM do
- →Writing decisions down early (a decision log) keeps a complex project consistent
- →Designing the evaluation harness alongside the agent makes progress measurable
Results
The architecture, deterministic engine and quality tooling are in place, with the agents and UI being built phase by phase. The design guarantees that every number in a report can be traced back to the query that produced it.
What I'd Do Differently
- ↻Build the evaluation scenarios before the agents, so every agent change is measured from day one
Need something similar?
I can build a system like this for your business — tell me what you need and I'll reply with a plan and a quote.