AI reporting should explain verified numbers, not create them.
The reliable pattern is simple: source systems own facts, code calculates metrics, rules detect conditions and a language model turns the approved data into a readable summary.
Define a source of truth for every metric
Document the system, table, field, owner, timezone, currency, refresh schedule and formula behind each measure. A dashboard name is not a metric definition.
If two systems disagree, resolve the ownership rule before adding AI. The model cannot repair unclear governance by choosing the number that sounds plausible.
The same source discipline is required when connecting AI to a CRM or processing business documents.
Build a deterministic data layer
Extract data through APIs or controlled queries, normalize types and calculate totals, rates, comparisons and periods in code or an analytics engine.
Store the reporting snapshot with a version and generated time. The narrative should point to that immutable snapshot so the report can be reproduced later.
UK guidance on AI-ready datasets describes continuous data-quality monitoring with dashboards and anomaly alerts for data products. Reporting quality begins before the prompt.
Give the model a bounded reporting package
Structured output can constrain a response to a schema; Google Cloud's official documentation describes this pattern. A valid schema helps integration, but factual validation is still required.
- approved metric names and definitions;
- current and comparison values;
- known events supplied by the owner;
- threshold breaches calculated by rules;
- allowed conclusions and prohibited claims;
- links to the underlying dashboard or record;
- a schema for the required output.
Separate four report products
Dashboard
Shows current governed metrics and lets a user inspect the underlying dimensions.
Narrative summary
Explains the largest changes using the supplied snapshot without recalculating or inventing causes.
Alert
Fires from an explicit threshold, missing-data rule or anomaly detector and includes evidence and severity.
Action brief
Lists what needs review, who owns it and the link to the source. It does not execute a high-impact action automatically.
Design anomaly handling
Anomaly does not mean error or cause. It means a value differs from a defined expectation and needs review.
Include current value, expected range, comparison period, data freshness and affected segment. Suppress duplicate alerts and track acknowledgement and resolution.
Review facts and language separately
Automated checks confirm values, formulas, periods, links, schema and required sections. Human review focuses on interpretation, business context and whether the wording could mislead.
The document pipeline in AI Document Processing uses the same separation between extraction, validation and approval.
Monitor the reporting system
NIST AI 800-4 documents the challenges of monitoring deployed AI under changing contexts. The report generator itself needs production monitoring.
- data freshness and missing sources;
- calculation and reconciliation failures;
- schema validation errors;
- unsupported claims in summaries;
- alert volume and duplicate rate;
- human edits and rejected reports;
- latency and model cost;
- incidents and distribution errors.
Roll out in controlled stages
Use the gates in the AI Implementation Roadmap. NIST AI RMF and the UK AI Playbook keep ownership, measurement and risk treatment connected.
- Reproduce an existing report from governed sources.
- Generate a draft summary beside the manual version.
- Measure factual and editorial corrections.
- Add rule-based alerts with human acknowledgement.
- Automate distribution only after access, recipients and rollback are tested.
How I report an automated publication
In the multilingual publishing system, the report does not ask AI to decide whether a page works. Code checks eight binary conditions: URLs, metadata, canonical, hreflang, image, sitemap, links and HTML cleanliness.
AI turns those results into a concise explanation and updates the Trello card. The underlying status codes and URLs remain available for verification.
This division prevents a polished summary from hiding a failed check. The narrative is useful because the evidence was calculated first.
Questions and answers
Should a language model calculate KPIs?
No. Calculate KPIs in code or an analytics system and give the verified values to the model for explanation.
Can AI explain why a number changed?
Only when the relevant evidence is supplied. Otherwise it should describe the change and request investigation, not invent a cause.
What makes an alert useful?
A defined condition, severity, evidence, affected scope, owner, source link and acknowledgement path.
Is valid JSON enough for reliable reporting?
No. Structured output validates format, while data, calculations and claims need separate checks.
What should stay manual first?
Interpretation of unusual events, sensitive distribution, external claims and any action with material consequences.