The model works.
The handoff fails.
When an AI conversation fails, the model is the easy thing to blame. The break I keep finding is in the handoff between people and the model. Anthropic's AI Fluency research points the same way.
When a conversational AI system fails in production, the diagnostic conversation that follows tends to circle the same verdict. The model's underperforming. The prompt is too loose. We need a bigger context window, a smarter retrieval layer, a more capable model. The disappointment is real, and so are the metrics that track it. After years of building conversational AI to millions of users across 8+ markets, I've grown skeptical of the reflex that hands the failure to the model.
I ask whether the collaboration around the model had a chance to succeed. Most teams need a better theory of how people and machines cooperate at the interface. A better model does not supply that theory.
The misattribution reflex
The history of new technologies is a history of misattribution. Early automobile accidents were often blamed on the animal the driver swerved to avoid; early aviation disasters on weather that had been predictable if anyone had a standard way of recording it. The pattern repeats whenever a system grows complex enough that the failure surface spans several disciplines. When no single person can hold all the moving pieces in view, causation gets pinned to the most visible component.
In conversational AI, the most visible component is the model. It's the thing that produces the output the user reads. When that output is wrong, strange, or absent, the model earns the blame. The underlying prompt, the retrieval pipeline, the tool schema, the fallback policy, the handoff contract, and the knowledge base that feeds all of them sit behind the curtain. None of those is as narratively satisfying to indict. The model is.
The cost of this reflex is specific. Teams iterate on the wrong variable. They evaluate models when they should be evaluating their evaluation harness. They tune prompts when the retrieval layer is returning irrelevant documents. They escalate to a larger context window when the real issue is that no one defined what the model should carry between turns. Donald Norman, writing about ordinary door handles long before anyone was arguing about context windows, called this an affordance problem[1]: the system tells the user what kind of interaction is possible, and the failures live in the mismatch between what it offers and what's required. Conversational AI has the same problem, at a higher temperature.
What the research is beginning to say
Anthropic's recent work on what it calls AI Fluency[2] offers one of the cleaner articulations of the shift. The people who consistently get useful work out of AI systems, the research suggests, aren't the ones with the deepest technical knowledge. They're the ones who have learned to communicate with these systems as collaborators. They give context. They specify constraints. They treat the response as input to the next turn rather than as a final judgment. They know when to stop and when to iterate. The skill looks less like programming and more like editorial judgment applied in real time.
That framing is important because it relocates the design problem. If fluency is a collaborative practice, then the instrument of collaboration, the prompt, the tool surface, the memory structure, the handoff policy, is where design has the most impact. Anthropic's work on constitutional AI points in the same direction from the opposite side of the interaction: the behavior of a capable model is shaped by the principles the system encodes, and those principles live outside the model itself. Design is the architecture of what the system treats as given, what it treats as variable, and what it treats as out of bounds.[3]
Adjacent disciplines have been saying similar things for a long time. Conversation analysis, going back to Harvey Sacks and his contemporaries,[4]has documented how ordinary human talk depends on shared repair strategies. When one party mishears or misspeaks, a functioning conversation has well worn mechanisms to recover: paraphrase, clarification, partial repetition, confirmation. Conversational AI inherits the same problem and has to solve it without the social cues humans rely on. A system that lacks a repair strategy is underdesigned. The model can be capable and still have no way back from a miss.
Three collaboration failures that masquerade as model failures
The failure modes I see most often, across enterprise engagements and post mortems, cluster into three categories. None of them is a problem the model can solve on its own.
Context failures
The model is asked to do a task without the information it would need to do the task well. The user knows something the system does not. The system has access to a database that the model cannot query. A previous turn established a constraint, but the current turn does not carry it. In each case, the model either fabricates to fill the gap or answers thinly. Both outcomes are context engineering problems: the discipline of deciding what the system remembers, what it retrieves, what it hands across, and what it lets go. A great deal of what reads as hallucination in production logs is, on inspection, the model confidently completing a pattern in the absence of the one fact that would have corrected it.
Feedback failures
The system produces an output. The user does something with it. Some of what they do, the click, the follow up question, the abandonment, contains information about whether the output served them. In many production systems, that information is never routed back into the loop that would improve the next output. Reinforcement learning from human feedback has become a popular phrase, but the day to day version of it, the capturing of labeled moments, the promotion of useful examples into an evaluation set, the deliberate closing of the improvement loop, is often absent. The result is a system that stops learning from its own operation. Practitioners mistake this for the model plateauing. The feedback architecture is missing. The model can keep improving while the product stops learning.
Handoff failures
The collaboration is never only between the user and the model. Somewhere downstream a tool has to be called, a record has to be written, an agent has to step in, a human has to take over. Each of these transitions is a handoff, and each handoff is a point of friction where context is supposed to travel across an interface. When the handoff schema is vague, context leaks. The human agent picks up a conversation they cannot reconstruct. The downstream tool receives a payload it can't validate. The second model in the orchestration has no record of what the first one established. These are specification problems. A vague handoff drops context. The model cannot patch that.
The orchestration frame
The industry has converged, somewhat awkwardly, on the word "agent" to describe the systems that handle these multi step, multi tool tasks. The word carries more weight than it can support. Most of what gets called an agent is, in practice, an orchestrated workflow with language model backed decision points. That is a useful thing to build. It is a workflow, with a model at the decision points. Anthropic's own essay on building effective agents[5] makes the architectural distinction clear: the useful unit is an orchestration pattern, and the design work sits in the choreography, not in the personality of the component.
Conversation design, rightly understood, is orchestration design. The practitioner's job is to decide which role the model plays at which moment, what each role knows, what tools each role can reach for, and how state flows between them. The system prompt is a role specification. The tool schema is a contract. The memory strategy is a policy. None of these artifacts are visible to the user. All of them are what the user experiences. Invest in the model and neglect the orchestration, and you've hired an actor and skipped the play.
What conversation design designs
The term conversation design invites a narrow reading: the craft of composing what the bot says. The Conversation Design Institute[6] and the practitioner community around it have been expanding the definition for years, and the broader reading is the one that holds up in production. The work is the design of the collaboration, which includes but exceeds the utterance. What can the user say? What does the system understand? What does it do when it doesn't understand? What does it carry forward? When does it ask for help? When does it hand off? What does it show the user about its own confidence? What does it learn from what happens next?
A useful test: if two competent practitioners disagree about a system's behavior, I look for the place in the specification where the behavior was underdetermined, and I tighten it. Changing the prompt is the later step. That discipline looks like spec writing. The best conversational AI work sits uncomfortably between product, engineering, research, and content. None of those functions on its own can specify the collaboration, because the collaboration is what they produce together.
Building the glue
The craft question that follows from all of this is practical. If the failures we blame on the model are really failures of collaboration, how does a team start building the collaboration on purpose? The short answer, reflected in the eval harnesses and prompt specifications and handoff contracts my engagements tend to produce, is that you start taking the scaffolding seriously as a first class artifact. Prompt architecture gets versioned like code. Tool schemas get reviewed in pull requests. Memory policy becomes a written document. Handoff payloads get their own evaluation. The system prompt gets edited in the same room as the retrieval strategy, with the same care, by people who understand what each change costs the user.
The infrastructure is where the design lives. I am not asking teams to add scaffolding for its own sake. The model will get better. It already has, many times over. The collaboration around the model will get better only if someone is doing the work of designing it. That person doesn't have a single job title. They sit in conversation design, prompt engineering, LLM product strategy, AI orchestration, conversational AI architecture, and increasingly in a role that doesn't have a clean name yet. The label matters less than the work.
The failure modes are fixable. That's the point I keep returning to. The teams I've seen recover from bad launches have rarely done it by switching models. They've done it by fixing the collaboration. They specified what the system knows. They built the feedback loop. They rewrote the handoff. They stopped asking the model to carry a problem the handoff was built to hold.
The model works. The handoff between people and the model fails when the collaboration around it is thin. Thin collaboration produces thin outputs. That is a design problem, and design problems can be solved.
Common questions
Why do AI conversations fail?
The failure is usually in the collaboration around the model. Teams design the prompt and the retrieval. They skip how people repair misunderstandings, signal intent, and take turns. Thin collaboration produces thin outputs.
Is the problem the model or the design?
Usually the design. A bigger model will not fix a conversation nobody designed for real talk. Design problems can be solved.
What is conversational repair, and why does it matter?
Repair is how people fix a misunderstanding mid-conversation: clarifying, rephrasing, confirming. Human conversation is full of it. Most AI systems are built as if every turn is understood perfectly the first time, so when something goes sideways there's no path back. Designing for repair is what separates a brittle bot from a usable one.
References
- [1]Donald A. Norman. The Design of Everyday Things. Basic Books, 2013 (revised ed.)↩
- [2]Anthropic. AI Fluency: Framework & Foundations. Anthropic, 2025↩
- [3]Yuntao Bai et al.. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022↩
- [4]Harvey Sacks. Lectures on Conversation, Vols. I & II. Blackwell, 1992 · Edited by Gail Jefferson; foundational work on conversation analysis and repair.↩
- [5]Anthropic. Building effective agents. Anthropic Engineering, 2024↩
- [6]Conversation Design Institute. Conversation Design Curriculum & Practitioner Resources. CDI↩
