Data & Analytics

refreshed source tables → metric pulls → variance analysis → narrative draft → review deck → leadership Q&A

• Automate
  Data refresh & pull — agent runs the standing queries, validates row counts and freshness against expectations, and populates the report shell · fully rule-based, high-repetition, and failures are loud rather than subtle
• Automate
  Variance detection — AI flags metrics outside expected bands and ranks what changed most · statistical thresholds are checkable and a false flag costs a minute of human attention
• Assist
  Variance explanation — AI drafts "revenue dipped because…" narratives by correlating across segments · this is causal inference dressed as summarization; the narrative reads authoritative but verifying it means redoing the analysis, so humans must construct the explanation with AI proposing leads
• Draft
  Narrative & deck drafting — AI turns the human-approved analysis into the review doc in house format · derivative writing against approved numbers, verifiable by the author, repeated every cycle
• Draft
  Leadership Q&A prep — AI generates likely follow-up questions and drafts answers from the underlying data · questions are cheap to generate; answers must be checked because a wrong number said aloud to the CFO is expensive

stakeholder question → clarified spec → SQL/notebook → sanity-checked results → findings memo

• Draft
  Question clarification — AI restates the ask, surfaces ambiguity ("active users by which definition?"), and drafts the analysis spec · cheap, checkable by the requester, and most bad analyses die here, so the human must still own the conversation
• Draft
  Query writing — AI writes SQL/Python against the documented schema · uniquely verifiable — run it, inspect results, compare to known totals — and high-volume; but only Draft, not Automate, because a query can execute cleanly and still answer the wrong question
• Assist
  Result sanity-checking — AI cross-checks outputs against known benchmarks, checks for fan-out joins, null traps, and timezone bugs · the checklist helps, but the checks that matter require knowing what the number should roughly be, which is human context
• Draft
  Interpretation & findings memo — AI drafts the "so what" from the verified numbers · fine for structure and prose, but any causal language gets human scrutiny because subtle overclaim is the default failure
• Automate
  Delivery & follow-up — agent packages the memo, logs it in the team's analysis library, and answers definitional follow-ups · clerical, low-stakes, and it builds the reuse library everyone claims to want

stakeholder requirements → metric definitions → semantic-layer/dbt models → dashboard build → documentation → adoption review

• Avoid
  Requirements & metric scoping — negotiating what "conversion" means with three teams who each mean something different · this is political alignment work, and the definition chosen silently reshapes incentives for years; AI can minute the meeting, not run it
• Draft
  Metric definition drafting — AI drafts the formal definition, edge cases (refunds? trials? internal accounts?), and dbt model code · the code is testable but the definition itself has subtle-error risk where a wrong filter is invisible until someone reconciles two reports months later
• Draft
  Model & pipeline code — AI writes and documents the dbt transformations with tests · verifiable via tests and output comparison, high leverage, but schema-context gaps mean human review of joins and grain
• Draft
  Dashboard construction — AI scaffolds the dashboard layout, charts, and filters from the spec · visual output is directly inspectable; iteration is cheap
• Automate
  Documentation & lineage — AI generates metric docs, column descriptions, and lineage summaries from the code · derived mechanically from source of truth, high-volume, and stale docs are worse than none
• Assist
  Adoption & deprecation review — deciding which dashboards to kill · usage data informs it but killing an exec's pet dashboard is a relationship problem

monitor/test suite → anomaly alerts → triaged incident → root-cause analysis → fix + backfill → incident writeup

• Automate
  Test & monitor authoring — AI generates freshness, volume, null-rate, and distribution tests from table profiles · rule-generation against observable data, verifiable by running, and over-generation just gets pruned
• Automate
  Alert triage & deduplication — agent clusters correlated alerts, mutes known-flaky monitors, and routes by table ownership · high-volume, rule-driven, reversible, and it's the toil that burns analysts out
• Draft
  Root-cause investigation — AI walks lineage upstream, diffs recent code changes, and proposes a ranked cause list · proposals are checkable against lineage and git history, but the model confuses correlation in timing with causation, so a human confirms before anyone touches prod
• Assist
  Fix & backfill — writing the correction and rerunning affected models · a wrong backfill silently corrupts history — low reversibility — so AI writes the candidate code and a human verifies against pre-incident snapshots
• Draft
  Stakeholder comms & writeup — AI drafts the "which reports were wrong, for how long" impact notice and postmortem · templated and checkable against the incident record, but trust is on the line so a human sends it

hypothesis brief → experiment design → instrumentation check → monitored run → readout → decision log

• Draft
  Hypothesis & design drafting — AI turns a product idea into a formal brief with primary metric, guardrails, and power/duration calculations · the arithmetic is checkable with a calculator, and a templated brief raises the program's floor
• Assist
  Design review — catching interference, dilution, seasonality, and underpowered designs before launch · errors here are invisible until they've wasted six weeks — checking is redoing — so AI serves as a checklist, not a reviewer of record
• Automate
  Instrumentation validation — agent verifies event firing, assignment balance, and sample-ratio checks in the first 48 hours · fully mechanical against defined expectations, and catching an SRM early saves the whole run
• Automate
  In-flight monitoring — AI watches guardrail metrics and flags degradation for human ship/kill calls · threshold-based flagging; the resulting decision stays human
• Assist
  Readout & causal interpretation — writing what the experiment means · this is the subtle-error capital of analytics: AI fluently narrates significance where there's noise, slices until something shines, and ignores novelty effects — the analyst must own every causal sentence
• Automate
  Decision logging — AI files the result, decision, and rationale into the searchable experiment repository · clerical, high-value memory, checkable against the readout

governed semantic layer → certified datasets → NL query interface → training materials → usage/accuracy audits

• Assist
  Semantic-layer curation — defining which tables, joins, and metrics the NL interface is allowed to touch · this is the load-bearing judgment: everything downstream inherits its quality, definitions are political, and errors are silent
• Automate
  Dataset documentation — AI writes and maintains column descriptions, synonyms, and sample questions for certified datasets · derived from source, high-volume, and directly improves NL-query accuracy
• Draft
  NL-to-insight answering — business users ask questions in plain language against certified data · counterintuitively not Automate: the answer engine runs itself, but treating outputs as owned-until-verified is the posture to teach, because users can't inspect the SQL underneath
• Automate
  Accuracy auditing — agent replays a benchmark question set weekly and diffs answers against certified numbers · fully mechanical regression testing; drift alerts route to the data team
• Assist
  Training & office hours — teaching business users what the tool can't do — causal questions, uncertified data, ambiguous metrics · behavior change is human work; AI drafts the materials and worked examples
• Automate
  Escalation triage — agent routes questions the interface can't answer safely into the analyst queue with context attached · classification with a human backstop built into the design