Zendesk AI agents: keeping answers in step with your website

Many established online stores already run their support on Zendesk, so its AI agents are the natural next step: same help centre, same tickets, same team. The setup is straightforward. The problem we see most often is not in the setup at all. It’s that the help centre and the website drift apart, and the AI agent answers from the help centre.

This note covers the settings and habits that matter for a store, and how to use the testing tools Zendesk provides.

How Zendesk bills and what counts

Zendesk bills AI agent usage in automated resolutions. Under its resolution tiers, a conversation counts as a verified resolution when a language model reading the conversation judges that the customer’s issue was resolved; other outcomes, such as contained conversations and assisted escalations, are treated differently.¹²

The verification reads the conversation, not your website. If the AI agent states the wrong delivery fee and the customer says “thanks”, the conversation can look resolved. That’s not a flaw unique to Zendesk; every platform that measures outcomes from the conversation alone has it. But it means the dashboard can’t tell you how many answers were right.

1. Find where the help centre and the website disagree

Before switching on the AI agent, list the topics customers ask about and, for each one, compare the help centre article with the website page.

Topic	Help centre article	Website page	Same?
Delivery times by region
Delivery costs and free threshold
Returns window and exceptions
Refund timing
Store opening hours
Payment methods

Wherever they disagree, decide which is right and fix the other. Then decide who updates both when something changes. In most stores the website is updated by marketing or e-commerce, and the help centre by support. That’s how the gap opens.

2. Split similar articles, or merge them

Zendesk’s AI agent picks the article that best matches the question. Two articles covering similar ground, such as UK and international delivery or online and in-store returns, are the most common reason it answers from the wrong one.

Either merge them into one article with clear sections, or make each title and first paragraph say exactly who it applies to: “Delivery within the UK” and “Delivery outside the UK”, with the region in the first sentence.

3. Write instructions and escalation for a store

Zendesk lets you set the AI agent’s persona and instructions and define when to escalate. For a store, the useful instructions are restrictive:

• don’t offer discounts, vouchers or refunds not described in the help centre;
• for any question about a specific order, ask for the order number and escalate;
• for damaged or wrong items, escalate;
• if two articles give different answers, escalate rather than choose.

A generic fallback that lists links or says “I can’t help with that” on a question your site answers plainly is a missed answer, not a safe one. If you see it often, the article exists but isn’t being found; rewrite its title and first lines in the words customers use.

4. Use Zendesk’s testing tools, with outside testers

Zendesk provides a test widget that shows the current customer experience, including instructions and escalation messages, and a conversation simulator that generates conversations from scenarios you write. Test conversations don’t use automated resolutions. Zendesk’s own guidance recommends testing with people who didn’t build the flows.³

That advice matters more than it sounds. The person who wrote the articles asks questions in the articles’ words, and the AI agent answers them well. Customers don’t. Get someone outside support to write twenty questions the way a customer would, with typos, follow-ups and two questions in one.

5. Check the answers against the website

For each test answer, write down the website page that should answer it, not the help centre article. Mark the answer right, wrong, missing or escalated. This is the only way to catch an article that has fallen behind the website, because the AI agent will repeat it word for word and look correct.

Then make it a routine: after each change to delivery, returns or prices on the website, re-run the questions for that topic. Before peak season, run the full set; our 25-question list is a starting point.

The short version

1. Compare help centre and website topic by topic; fix every disagreement.
2. Split or clearly label similar articles.
3. Restrict discounts and promises; escalate orders and damage.
4. Test with people who didn’t write the content.
5. Check answers against the website, after every change.

If you’d like an outside check, our Zendesk AI agents page shows what we look for. For background on why answers drift, see how an AI assistant answers.

Footnotes

1. Zendesk, “About automated resolution tiers”, support.zendesk.com. ↩

2. Zendesk, “About automated resolutions for AI agents”, support.zendesk.com. ↩

3. Zendesk, “Best practices for testing AI agents”, support.zendesk.com. ↩