Explainable AI

Three genuinely different questions a user might have about an AI system’s output don’t share one answer: transparency covers what happened (what the system did, what process or data was involved), explainability covers how the system reached that output, and interpretability covers why that output means what it does in the context of the user’s own goal. Tailor the depth and vocabulary of an explanation to the user’s actual role, knowledge, and skill level rather than reusing one fixed explanation for every audience — a technical trace serves a developer debugging a model; a plain-language reason serves the end user asking “why was I shown this?” This distinction matters most for a subject matter expert using an AI-assisted tool (a physician, an auditor): research finds they’re typically less interested in how a system reached its output than in why — the same interpretability question an end user asks, just phrased professionally rather than personally. An explanation UI built around how the model works answers the wrong question when the actual ask is why this output.

Tailor the depth and vocabulary of an explanation to the user’s actual role, knowledge, and skill level rather than reusing one fixed explanation for every audience — a technical trace serves a developer debugging a model; a plain-language reason serves the end user asking “why was I shown this?”

Meet four explanation qualities, independently of each other

A separately useful breakdown names four things any explanation needs to get right, independently of each other:

• Explanation — give some evidence or reason for every output, satisfied even by a bare reason.
• Meaningfulness — make that reason actually understandable to the specific person asking, since different user groups — and the same person over time, as they gain experience — find different things meaningful.
• Explanation accuracy — make sure the explanation faithfully reflects what the system actually did rather than just sounding plausible, since a decision can be correct while its stated reason is fabricated; distinct from decision accuracy.
• Knowledge limits — have the system say when it’s outside its own depth, either because a question falls outside what it was built to handle or its confidence is too low to trust, rather than answering anyway (“I found a bird in the image, but the image quality is too low to identify it” is knowledge-limits-aware; a system that guesses instead isn’t).

None of this is unique to AI: tested against these same four questions, human decision-makers do poorly too — people routinely fabricate plausible-sounding reasons for their own decisions after the fact rather than accurately reporting their actual reasoning, and are unreliable judges of their own knowledge limits. Neither side of a human-AI pairing gets a free pass on any of these.

people routinely fabricate plausible-sounding reasons for their own decisions after the fact rather than accurately reporting their actual reasoning, and are unreliable judges of their own knowledge limits.

Cover the explanation type the situation actually calls for

A complementary breakdown names what an explanation can actually be about, distinct from the four qualities above:

• Rationale — the reasons behind a specific decision.
• Responsibility — who’s accountable, and who to contact for a human review.
• Data — what data was used, and how.
• Fairness — the steps taken to keep the decision unbiased and equitable.
• Safety and performance — accuracy, reliability, and robustness.
• Impact — the decision’s effects on the individual and on wider society.

Each of these six can itself be process-based (how the system is designed and governed in general) or outcome-based (why this particular decision came out the way it did) — most real explanations combine several of the six rather than picking just one.

Match timing, detail, and audience to the situation

Match an explanation’s timing and level of detail to how much time the person actually has to act on it, not to how much the system is technically capable of disclosing. An emergency alert needs a plain, immediate reason (“Tornado Warning: Take Action”) because the recipient has to act now; a loan decision can carry more detail since the applicant has time to read it; a system audit or a developer debugging a model can absorb the most detail of all, since neither is under time pressure.

Match an explanation’s timing and level of detail to how much time the person actually has to act on it, not to how much the system is technically capable of disclosing.

Beyond urgency, four further factors decide which of the six explanation types above to prioritize for a given case:

• Domain — a criminal-justice decision and a product recommendation carry very different expectations.
• Impact — how severe the consequences are for the individual.
• Data — what data was actually used to reach the decision.
• Audience — the recipient’s own expertise, and any accessibility or reasonable-adjustment needs.

The same output can also need different explanations for different audiences on purpose, not as a “dumbed down” version for some: an end user wants the reason behind their own result, a regulator or auditor wants enough evidence to verify compliance, and a developer wants enough detail to debug the system — and the explanation should also say plainly which of these purposes it’s serving, since a purely informational explanation is a different artifact from one meant to support an appeal or a dispute.

Treat delivering an explanation as a dialogue, not a one-way disclosure — layer it so the prioritized explanation arrives first, with further detail available in additional layers (expanding sections, tabs, linked pages) rather than everything at once, and give the person a way to actually discuss it with a competent human being rather than only read further static text.

Decide what can be explained before the decision is made

Some explanation types can be delivered before a decision is even made, not only after: process-based explanations generally, plus the outcome-based responsibility, impact, and data explanations, can all be published proactively. Rationale, fairness, and safety-and-performance explanations are usually specific to one decision and only become available once it’s been made.

Where the stakes are high enough to demand the fullest possible rationale explanation, match the model choice to that demand — an inherently interpretable model whose behavior is explainable by design serves a high-stakes decision better than a black-box model with an explanation layer bolted on after the fact, since a reconstructed, after-the-decision interpretation is a weaker guarantee than a model that was explainable from the start. Whatever the explanation’s format, state its own error range where one can be meaningfully calculated, weighed against how much that adds to interface complexity — a system should be honest about the uncertainty in its own explanation, not just the decision it’s explaining.

Related Concepts

Patterns

• AI Content Marking

Principles

• Human-AI Interaction
• Trust and Credibility

Further reading

Tim Miller’s “Explanation in Artificial Intelligence: Insights from the Social Sciences” (Artificial Intelligence, 267, 1–38, 2019 — free to read via arXiv, but published by Elsevier with no stated open license) argues AI explanation should borrow directly from how humans explain to each other: contrastive (why this outcome and not some specific alternative, not a bare justification), selective (a few key causes offered at a time, not a complete causal chain), and social (a back-and-forth conversation, not a one-way report) — a framing distinct from, and complementary to, this page’s six-type NIST/ICO taxonomy.

Sources

AI Risk Management Framework (AI RMF 1.0) is the source for the transparency/explainability/interpretability distinction.

Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270) is the source for the subject-matter-expert “why not how” finding.

Four Principles of Explainable Artificial Intelligence (Draft NISTIR 8312) is the source for the four explanation principles (explanation, meaningfulness, explanation accuracy, knowledge limits), the finding that humans meet these principles poorly themselves, and the time-requirement-vs-detail trade-off for matching explanation format to the situation.

Explaining decisions made with AI (ICO & The Alan Turing Institute) is the source for the six-type explanation taxonomy (rationale, responsibility, data, fairness, safety-and-performance, impact) and its process-based/outcome-based split, the domain/impact/data/audience contextual factors for prioritizing among them, and layered and dialogue-based explanation delivery.

Blueprint for an AI Bill of Rights is the source for risk-calibrated model choice (favoring inherently interpretable models over a black box with a bolted-on explanation in high-stakes settings), tailoring an explanation to its purpose, and stating an explanation’s own error range.