Your chatbot broke last week and nobody noticed: silent failures and how to catch them

A wrong answer is a problem. No answer at all, for every visitor, for a week, is a bigger one. And it happens more often than businesses expect, because an AI assistant depends on several things nobody on the team watches: an API key, a model version, a content sync, a script on the page. When one of them breaks, the assistant often doesn’t fail loudly. It shows an error in the chat window, or spins forever, or doesn’t appear at all, and the dashboard simply shows fewer conversations.

What breaks

Keys and billing. Assistants built on a model provider’s API stop when the key expires, the card on file fails or a usage limit is hit. The widget still loads; every answer is an error message.

Model changes. Vendors retire older models and change defaults. A bot pinned to a retired model can stop answering; one moved to a new model can start answering differently. Most platforms announce this, but the email goes to whoever set the bot up.

Content sync. A site redesign changes page addresses. The assistant’s sources still point to the old ones, now returning 404, and it starts saying it doesn’t know things it knew last week.

The page itself. A theme update, a new cookie banner, a script optimiser or a consent tool blocks or hides the chat widget. On mobile, the widget may now sit behind another button.

Flows. In assistants with scripted steps (collect order number, then look it up), one changed step can create a loop: the same question again and again.

Why nobody notices

The team rarely uses its own chat. The platform’s dashboard shows conversations, and a drop in conversations looks like a quiet week. Customers who hit an error don’t report it; they leave, or write an email that lands in the general inbox as “your chat doesn’t work” days later.

A made-up example

A coworking space rebuilt its website in August. The pricing page moved from /prices to /membership. The assistant, built on a general AI tool, had been trained on /prices.

For three weeks, every question about prices got “I’m sorry, I don’t have information about our prices. Please contact us.” Conversations didn’t drop; resolution rate barely moved, because many visitors simply left after the answer. The owner found out when a prospective member mentioned it on a tour.

The five-minute weekly check

Once a week, at a fixed time, from a phone, in a private browser window:

1. Open the site. Is the chat widget visible, and not hidden behind another element?
2. Ask one price question. Does it answer with the current price?
3. Ask one policy question. Returns, cancellation or delivery. Does it match the page?
4. Ask one question it should hand over. Does the hand-over work and go to a real inbox?
5. Glance at the conversation count for the week against the previous one. A sharp drop is worth a look.

Write the date and result in a shared note. If you change the website, prices or policies, do it again the same day.

Make the alerts reach someone

• Vendor emails: make sure the platform’s account email is a shared inbox that someone reads, not the personal address of whoever set it up two years ago.
• Billing: a payment failure on the AI provider is a site outage. Put it on the same card and alerts as your hosting.
• Error monitoring: if you built the bot yourself, log failed answers and alert on a spike, the way you would for checkout errors.

The platforms publish status pages and changelogs, which are worth subscribing to; OpenAI’s model deprecations page, for example, lists retirement dates for models that many custom bots depend on.¹ Intercom and Zendesk publish their own release notes.²

We list this as a common problem in its own right: broken after an update, along with loops and silence. If you’d rather someone else ran the weekly check, and the longer one behind it, that’s Assistant Care. For what the longer check measures, see resolution rate versus accuracy.

Footnotes

1. OpenAI, “Deprecations”, platform.openai.com/docs/deprecations. ↩

2. Intercom, “Changes”, intercom.com/changes. ↩