Human-AI Interaction
Designing how people work with an AI-powered feature: who has the final say when a person and the system disagree, and how much a person can trust what the system tells them. NIST’s AI Risk Management Framework names “human factors experts (e.g., UX/UI design)” as one of the recognized actor categories across an AI system’s lifecycle, responsible for human-centered design practice and evaluating end-user experience — the same role UX practice already plays elsewhere, applied to AI-specific interaction problems. Trust in an AI feature is earned the same way site trust generally is earned — never assumed, only earned and re-earned — with AI adding its own levers: whether a person can override the system’s decision, whether known failure modes in that pairing are watched for, and whether the system’s reasoning and AI-generated output are disclosed rather than hidden (see Explainable AI and AI Content Marking).
Designing how people work with an AI-powered feature: who has the final say when a person and the system disagree, and how much a person can trust what the system tells them.
Define human oversight roles before designing the interface
Human-AI configurations span a spectrum: a system can decide autonomously, defer entirely to a human expert, or serve as one additional opinion in a decision a human still makes. Not every system needs oversight — a model that only improves video compression usually doesn’t — but wherever it’s needed, who decides, who can override, and who’s accountable for the outcome needs to be an explicit design decision, not something a user has to infer from how the interface happens to behave.
Real-time interfaces like chatbots often can’t fit synchronous human review into the response loop at all — the answer has to go out before a person could weigh in. In that case, move human control to the stages where it can happen: thorough testing before deployment, ongoing assurance checks on the live system, and a visible way for users to flag a bad response that triggers human review after the fact, rather than treating the absence of real-time review as an excuse to drop human oversight altogether. When a chatbot or virtual assistant reaches the limit of what it can answer, hand off immediately to a named escape valve — a live agent or a self-service knowledge base — rather than leaving the person stuck with an assistant that has run out of answers. See AI Chat Interface for the concrete chat-window structure this hand-off happens inside.
A time-critical system needs its human fallback available immediately, sometimes even before anything goes wrong rather than only after — an automated building-access system backed by a manager who can simply open the door when the card reader fails is the plain version of this; waiting for a review process to run its course defeats the point once the window to act has already closed. How much staffing and availability that fallback needs should scale with the stakes: a system with more control over the outcome, or operating in a higher-stakes domain, needs proportionately more human reviewers on hand, not a fixed-size queue regardless of what’s riding on it.
Give users a visible way to appeal or override an AI decision
Residual risk — whatever a system’s output gets wrong even after mitigation — lands directly on the end user, so an AI-powered feature needs a visible, working path for a person to appeal or override its output, not only a path for a developer to fix it after the fact. This matters most wherever the system’s output has a real, direct effect on the person it’s about — a recommendation, a moderation action, an eligibility determination — rather than a low-stakes, purely internal computation. A workable appeal channel depends on a legible explanation of what’s being appealed — U.S. consumer-credit law already requires this pairing: an adverse-action notice for a declined credit application has to state a reason, precisely so the applicant has something concrete to appeal. Concrete appeal options include routing the person to a human decision-maker instead of the AI system, and letting them opt out of similar AI-generated content going forward. That opt-out option needs its own announcement — a brief, findable notice that a human alternative exists and how to reach it — rather than being a real but undiscoverable fallback nobody knows to ask for. UK data protection law draws a harder line once the stakes cross a threshold: UK GDPR Article 22 prohibits any decision based solely on automated processing that has a legal or similarly significant effect on a person, unless a human is meaningfully involved — an appeal channel is the right fallback for lower-stakes decisions, not a substitute for human involvement once a decision matters this much.
When a human reviewer overturns an automated decision, the reversal needs to propagate through every system component that acted on the original decision, undoing whatever downstream consequences it already triggered — a benefit clawback already issued, an account already flagged — not just updating the record to show the new answer. A decision that’s nominally reversed but whose consequences are still in motion elsewhere in the system isn’t actually overturned from the person’s side of it.
Watch for bias amplification in human-AI judgment tasks
Pairing a person with an AI system doesn’t reliably reduce bias. Under some conditions — particularly perceptual-judgment tasks — the AI component can amplify a human’s existing bias, producing a more biased decision than either the human or the system would reach alone. The same pairing, organized deliberately (for instance, structuring when a person should defer to the system versus push back on it), can instead produce complementary results that beat either alone. Which way a given system leans needs testing against real judgment tasks, not an assumption that adding AI assistance is automatically an improvement.
Pairing a person with an AI system doesn’t reliably reduce bias.
Four named failure patterns are common enough to watch for by name:
- Automation complacency — a person over-relies on the system’s output until their own independent judgment atrophies from disuse.
- Selective adherence — a person accepts AI advice only when it already matches what they believed going in and dismisses it otherwise, which defeats the point of a second opinion, since the system’s disagreement is exactly the case it existed to catch.
- Algorithmic aversion — the opposite failure: a person distrusts or avoids a generative AI system even where it would genuinely help, typically because they lack a detailed picture of how the system works rather than because it’s actually unreliable for the task at hand.
- Emotional entanglement — a person becomes attached to a generative AI system to a degree that risks real psychological harm, a distinct risk from simply over-trusting the system’s judgment.
Countering this is a training problem as much as a system-design one: give the people reviewing AI output recurring training on how to interpret it for its actual intended purpose, and keep reassessing whether their involvement is still catching what it’s supposed to catch rather than quietly rubber-stamping it.
Broaden “context of use” beyond the person clicking the button
Run an AI feature through the same human-centered design loop as any other interface — understand the context of use, involve real users throughout, evaluate against their actual needs, iterate — but for an AI feature, “context of use” has to span more than the person directly interacting with it: the organizational environment building the feature, the operational environment it runs in, and the larger societal environment it’s deployed into. Two concrete factors that live in that broader context can’t be fixed by adjusting the model’s math: the digital divide (people without reliable access to the underlying technology in the first place) and disability-specific interaction difficulties. A system whose outputs are statistically balanced across demographic groups can still be inaccessible to people with disabilities or shut out by the digital divide — see Accessibility. Fairness across demographic groups and accessibility for people with disabilities are separate design problems; solving one doesn’t solve the other.
Fairness across demographic groups and accessibility for people with disabilities are separate design problems; solving one doesn’t solve the other.
Related Concepts
Patterns
Principles
Standards
Further reading
Google PAIR’s People + AI Guidebook v2 (pair.withgoogle.com/guidebook-v2 — Creative Commons Attribution-NonCommercial-ShareAlike 4.0; the NonCommercial term is why this stays a Further Reading pointer rather than a fully ingested source) offers broader end-to-end guidance for designing AI products across five areas: understanding user needs before deciding whether AI is the right tool, evolving data and models over a product’s life, building and calibrating trust through explanations, designing feedback and override controls, and handling errors gracefully. It covers full product-design process rather than this page’s narrower focus on oversight and trust calibration.
Amershi et al.’s “Guidelines for Human-AI Interaction” (CHI ‘19, dl.acm.org/doi/10.1145/3290605.3300233 — ACM copyright, not openly licensed) proposes 18 validated design guidelines for human-AI interaction, developed and tested with 49 design practitioners against 20 AI-infused products — a widely-cited academic complement to this page’s NIST/GOV.UK/OSTP-sourced guidance.
Microsoft’s HAX Toolkit (microsoft.com/en-us/haxtoolkit — no stated open license, plain ”© Microsoft 2026” footer) packages the Amershi et al. guidelines above into applied design patterns, workbooks, and playbooks for teams building AI products, rather than stating the 18 guidelines as an academic list alone.
Horvitz’s “Principles of Mixed-Initiative User Interfaces” (CHI ‘99, dl.acm.org/doi/10.1145/302979.303030 — ACM copyright, not openly licensed) is a foundational academic framework for when a system should act on its own initiative versus defer to the user, complementing this page’s human-oversight-spectrum guidance with the classic mixed-initiative design vocabulary.
Shneiderman and Maes’s “Direct Manipulation vs. Interface Agents” (Interactions, 1997, dl.acm.org/doi/10.1145/267505.267514 — ACM copyright, not openly licensed) is a point-counterpoint debate between user-controlled direct manipulation and autonomous interface agents acting on a person’s behalf — an early articulation of the same control-vs-autonomy tension this page addresses for modern AI features.
Sources
AI Risk Management Framework (AI RMF 1.0) is the source for this page’s human-AI-configuration spectrum and the bias-amplification-in-human-AI-teaming finding, drawn from its Appendix C (“AI Risk Management and Human-AI Interaction”) and its §3.4–3.7 discussion of trustworthy-AI characteristics.
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270) is the source for the appeal/override recourse-channel guidance (including the adverse-action-notice precedent), the automation-complacency and selective-adherence failure patterns, and the broadened human-centered-design context-of-use framing (organizational/operational/societal environment, digital divide).
AI Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) is the source for the algorithmic-aversion and emotional-entanglement failure patterns.
AI Playbook for the UK Government is the source for the human-control-at-other-stages guidance for instant-response AI and the UK GDPR Article 22 automated-decision rule.
Blueprint for an AI Bill of Rights is the source for the opt-out-notice requirement, proportional human-fallback staffing, propagating a reversed decision through its downstream effects, immediate fallback for time-critical systems, and training/reassessing human reviewers to counter automation bias.
Federal chatbot and virtual-assistant case studies (Digital.gov) is the source for handing off to a human agent or self-service knowledge base when a chatbot reaches the limit of what it can answer.