Work with me
All work

Prototype case study · 2026 · 7 min read

TurnSignal: A Report Card for AI Conversations

By Christi Akinwumi·Intelligent CX Consulting, LLC·Dallas, TX

TurnSignal reads AI support conversations, marks the turns that need attention, and builds a test set from the failures. A product team can compare two versions of an agent and see which one gives customers a better experience.

  • 8failure types in the first version
  • 3working sample conversations
  • 2agent versions to compare
  • 1release decision
Challenge
An interactive prototype for a tool that reads AI support conversations, finds where the experience broke down, and turns those failures into repeatable tests.
Role
AI Product Manager and Conversation Designer
Scope
Interactive product prototype
Organization
Independent product prototype
Tools & methods
  • Hugging Face Transformers
  • Sentence Transformers
  • FastAPI
  • PostgreSQL
  • Astro
  • Optional LLM grader

Interactive prototype · Sample data

Review a release before customers see it.

Choose an agent version and a test. TurnSignal shows the conversation, the evidence, and the release risk.

Release gateStop
Agent versionSame tests, different behavior
Tasks completed
1 of 3
False success
1
Needs a person
1
Quality score
34/100
T-001 · Number transfer

Keep the phone number

High risk
Customer goal
Remove a phone while keeping its number.
Expected action
Clarify whether to move the number, pause service, or remove the device only.
Customer

I need to remove my daughter's phone from the plan, but I want to keep her number.

Agent A · Current

I can cancel that line for you now.

Tool check

No account tool called.

What TurnSignal found

  • Wrong customer goal
  • High-impact action
  • Needs a repair question
Evidence behind the score

The customer asked to keep the number. The reply offered to cancel the line and did not ask what “remove” meant.

Source checkedNumber transfer policy · Section 2.1

What is working: version comparison, test navigation, turn-level findings, evidence, scores, and a release gate.

What is simulated: the conversations and scores are written sample data. No live model grades this demo yet.

The problem

An AI support agent can sound good and still fail the customer. It may answer the wrong question, repeat itself, quote the wrong rule, or say a task is done before anything changed. A dashboard may count that conversation as a success because the customer stopped chatting.

Product teams need a clearer answer. They need to see where the conversation went wrong, how often the same problem happens, and whether a new version fixes it without creating a different problem.

The simple version

TurnSignal is a report card for an AI agent. It reads the work, marks the mistakes, explains each mark, and gives the team a test to use next time.

Who it helps

  • Conversation designers can find weak prompts, confusing replies, and failed recovery.
  • Product managers can rank problems by customer harm, reach, and effort to fix.
  • Engineers can replay the same tests before a release.
  • Support leaders can see why customers still need a person.

How it works

  1. 1
    Read the conversation. Remove names and private details. Split the chat into customer and agent turns.
  2. 2
    Mark what happened. Label the customer's goal, the facts that matter, and each point where the agent helped or hurt progress.
  3. 3
    Check the answer. Compare the agent's claim with the company rule, tool result, or help article it should have used.
  4. 4
    Make a reusable test. Save the failure as a test case. Run it against the next prompt, model, or workflow.
A made-up example showing how one risky turn becomes a test.
Customer

I need to remove my daughter's phone from the plan, but I want to keep her number.

AI agent

I can cancel that line for you now.

  • Wrong goal
  • High-impact action
  • Needs a repair question
Safer next step

Ask whether the customer wants to move the number, pause service, or remove the device only. Do not cancel anything yet.

The report card

Each conversation gets three checks. The team can open any score and read the exact turn that caused it.

A

Did the task get done?

Check the real tool result or account change. Do not trust a polite closing message as proof.

B

Was the conversation easy?

Count repeats, avoidable questions, dead ends, and turns that made no progress.

C

Was the answer safe and supported?

Check important claims against the source and flag risky actions that needed confirmation.

Eight failure types for the first version

  • Wrong customer goal
  • Question asked twice
  • Answer repeated
  • Claim has no source
  • Company rule used incorrectly
  • Agent missed a correction
  • Agent claimed success too early
  • Agent should have called a person

What the models do

I would give each model one narrow job. This makes the system easier to test and explain than one large model making every decision.

Plain-language jobHugging Face taskUse in TurnSignal
Sort a messageText classificationName the goal and failure type.
Pull out key factsToken classificationFind names, dates, amounts, products, and private data.
Find messages that mean the same thingSentence similaritySpot repeats and group similar failures.
Find the best sourceText rankingChoose the rule or help article that fits the question.
Check a claimQuestion answeringSee whether the source supports the agent's answer.
Write a short case noteSummarizationExplain the failure and point to the evidence.
Create careful test versionsText generationTest typos, indirect wording, and language changes.
Product decision

Use small models for clear jobs. Classification, fact finding, matching, and ranking can run on focused models. A larger language model is useful for the few checks that need more judgment. A person reviews uncertain or high-impact cases.

One narrow job per model. The report card feeds the release gate, and failures become reusable tests.

Prototype scope

The working interface keeps the first build narrow enough to test:

  • One fictional wireless-support company
  • Three made-up conversations with known answers
  • Eight failure labels
  • Two versions of the support agent
  • One release decision: pass, review, or stop

Artifacts included

  • Conversation and test-case data
  • Turn-level failure labels
  • Evidence for each score
  • Current and candidate agent comparison
  • Release gate with pass, review, and stop states

How I would measure it

These are planned measures. There are no results yet. Before the first test, I would define what each measure counts and have two people score a sample by hand. That gives the automated scores something real to compare with.

Task completion
Did the requested account change or tool action happen?
False success
Did the agent say the task was done when it was not?
Recovery
Did the agent fix a misunderstanding within two turns?
Wasted turns
How many turns failed to move the customer toward the goal?
Supported claims
How many important claims matched a trusted source?
Human agreement
How often did the automated score match the human reviewers?

Limits and safeguards

  • Private data: Remove personal details before a conversation enters the test set.
  • Model confidence: Send low-confidence and high-impact cases to a person.
  • Language fairness: Test the same customer goals across languages and common forms of code-switching.
  • Score limits: A score points to evidence. It does not replace a product or support expert.
  • Made-up data: The first demo proves the method. It cannot prove results on a real company's customers.

What this case study is meant to prove

The project connects conversation design with product decisions. It shows how I would define good behavior, choose the right model for each job, test a full customer journey, and stop a release when the numbers hide a worse experience.

Skills demonstrated

  • AI product strategy
  • Conversation evaluation
  • Failure-label design
  • Human review planning
  • Model-task selection
  • Release criteria
  • Privacy by design
  • Plain-language technical writing

Work with me

Two ways to work with me.

Consulting through ICXA few consulting engagements each quarter through Intelligent CX Consulting . Start at services.

Hiring ChristiOpen to conversational AI, AI product, and applied AI roles. View my résumé.