A high resolution rate doesn't mean your chatbot is right

Open the reports of any AI support platform and the first number you see is some version of the resolution rate: the share of conversations the AI handled without a person. Vendors put it at the top of their pricing pages, and it’s the number store owners quote to each other. A bot at 70% sounds like a bot that works.

It is a useful number. It just measures something narrower than it sounds like.

What “resolved” counts

The definitions differ by platform, but they share a shape.

• Intercom counts a Fin outcome when the customer confirms the issue is resolved, when they don’t ask for more help after Fin responds, or when Fin completes a defined procedure.¹
• Zendesk bills automated resolutions, and under its resolution tiers a language model reads the finished conversation and judges whether the customer’s issue was resolved.²
• Tidio counts a Lyro conversation whenever Lyro replies at least once, and reports resolution separately.³

Each definition is reasonable for billing. None of them looks at your website. The question being asked is “did the customer stop?” or “does this conversation read as resolved?”, not “was the answer what our policy says?”.

Three conversations that count as resolved

Take a made-up language school that sells evening courses with a published rule: “Full refund if you cancel at least 7 days before the course starts. No refund after the first lesson.”

Conversation 1. “Can I get a refund if I cancel after the first lesson?” — “Yes, you can cancel at any time for a full refund.” — “Great, thanks.”

Conversation 2. “Do you have a beginners’ Spanish course on Tuesdays?” — “No, our Spanish courses run on Mondays and Thursdays.” (The schedule page lists a Tuesday beginners’ group.) The customer closes the chat.

Conversation 3. “How much is the intensive course?” — “The intensive course is €390.” (That’s the price of the standard course; the intensive is €520.) — “OK.”

All three are likely to be counted as resolved. The first creates a refund dispute. The second loses a sale. The third sets up an awkward conversation at payment. On a per-outcome plan, you also paid for each one.

Why the number can rise while answers get worse

A resolution rate goes up when the AI answers more questions without handing over. That happens when:

• the content covers more topics, which is good;
• the instructions push the AI to answer more confidently, which can be bad;
• the hand-over is harder to reach, which is bad;
• customers give up, which looks the same as being satisfied.

So a rising rate after you’ve told the assistant to “always give a complete answer” may mean more wrong answers, not fewer. The number can’t tell these apart.

The second number: answer accuracy

Next to the resolution rate, track one more number: of a set of questions customers really ask, how many answers match your own pages.

How to measure it:

1. Write a question set. Thirty to fifty questions in customers’ own words, covering prices, delivery, returns, schedules, opening hours and anything else people ask. Include follow-ups (“and for two people?”) and the tricky cases (exceptions, regions, product variants).
2. For each question, note the page and sentence that answers it.
3. Ask the assistant on the live site, as a customer would.
4. Mark each answer: right, wrong (contradicts the page), missing (says it doesn’t know, though the page answers it), handed over.
5. Accuracy = right answers ÷ questions the page answers. Track wrong answers separately: they’re the expensive ones.

Repeat it every month, and after every change to prices or policies. Our pre-season list is a ready-made set for stores.

Metric	Tells you	Doesn’t tell you
Resolution rate	How much work the AI takes off people	Whether its answers were right
Answer accuracy	How often answers match your pages	How many conversations the AI handles
Wrong-answer count	How often the AI says something that costs you	Why

What to do with the result

A high resolution rate with high accuracy is the goal: the AI handles most questions and gets them right. A high rate with low accuracy is the dangerous combination, and the one dashboards make look like success.

When accuracy is low, the cause is almost always in the content: an outdated page, a rule split from its exception, two pages that disagree. We list the common shapes, each with a made-up example, on what goes wrong. How an AI assistant answers explains where in the process each one happens.

If you’d like the second number measured for you every week, that’s what Assistant Care does, with each wrong answer next to the page it contradicts.

Footnotes

1. Fin by Intercom, pricing and outcome definitions, fin.ai/pricing. ↩

2. Zendesk, “About automated resolution tiers”, support.zendesk.com. ↩

3. Tidio, pricing, tidio.com/pricing. ↩