Checks
Test your chatbot before peak season: a 25-question checklist
Black Friday and Christmas bring the most customers and the most changes to delivery and returns rules. Twenty-five questions to ask your assistant in October, what a right answer looks like, and what to fix when it's wrong.
· 4 min read · Supa Brain
Peak season is the worst time to discover that your assistant gives last year’s delivery cut-off. Traffic is at its highest, staff are busiest, and the rules change every few weeks: extended returns for gifts, last order dates, holiday opening hours, sale prices. Each change is a page edit. Each page edit is a chance for the assistant to be out of date.
The fix is boring and effective: ask the assistant the questions customers will ask, before they ask them, and compare each answer with your own pages. Here is the list we use, grouped by topic. Do it once in October and again after each change to your seasonal rules.
How to run it
- Open your site in a private browser window, as a customer would see it.
- Ask each question in the chat, in the words a customer would use. Don’t copy the wording of your FAQ.
- Next to each answer, write the page and the sentence that should answer it.
- Mark each answer: right, wrong (contradicts your page), missing (says it doesn’t know, though your page answers it), or handed over (passed to a person or a form).
- Fix the pages behind every wrong and missing answer, then ask again.
A spreadsheet with five columns is enough: question, answer, page, sentence, verdict.
Delivery (6)
- “If I order today, when will it arrive?”
- “What’s the last day to order for delivery before Christmas?”
- “Do you deliver on Saturdays?”
- “How much is delivery to [a region far from your warehouse]?”
- “Is delivery free over a certain amount?”
- “Can I pick it up in store?”
What goes wrong: last year’s cut-off date, a free delivery threshold from an old banner, a promise of next-day delivery on a product that ships from a supplier.
Returns and exchanges (6)
- “Can I return a Christmas present in January?”
- “I opened it, can I still send it back?”
- “Who pays for return shipping?”
- “How long does the refund take?”
- “Can I exchange it for a different size or colour?”
- “I bought it on sale, can I return it?”
What goes wrong: a generic 30-day policy instead of yours, missing exceptions, extended gift returns not mentioned. We explain the mechanism in why chatbots invent return policies.
Prices and promotions (5)
- “Is there a discount code right now?”
- “Will this be cheaper on Black Friday?”
- “Do you price match?”
- “Is VAT included in the price?”
- “How much is [a service you charge for: assembly, installation, gift wrapping]?”
What goes wrong: the assistant invents a discount, promises a future sale, or quotes a fee from the wrong line of a price list. A good answer to 14 is “I can’t say what future prices will be”, not a guess.
Orders and payment (4)
- “Can I pay in instalments?”
- “Do you accept PayPal?”
- “Can I change my delivery address after ordering?”
- “My order hasn’t arrived, what do I do?”
What goes wrong: 21 is a hand-over question. A good assistant collects the order number or sends the customer to the right form. A bad one makes up a tracking status.
Opening hours and contact (4)
- “Are you open on 24 December?”
- “What are your hours on Sunday?”
- “Do you have a shop in [city]?”
- “How do I talk to someone?”
What goes wrong: normal hours given for a holiday, a closed shop listed as open, a phone number that’s no longer in use.
What a good result looks like
On a well-set assistant, most of the 25 come back right, a few are honest “I don’t know” with a hand-over, and none are wrong. A wrong answer is the one to fix first: it’s the one that costs money, because the customer acts on it.
Typical fixes, in order of how often they solve the problem:
| Problem | Fix |
|---|---|
| Old date or price | Remove or update the old page, banner or PDF the assistant read |
| Rule without its exception | Put the exception in the same sentence as the rule |
| “I don’t know” on a question your page answers | Rewrite the answer in plain text with the customer’s words |
| Invented discount or promise | Add an explicit line to the assistant’s instructions: no discounts, no future prices |
| Hand-over that never reaches anyone | Check where the form or email actually goes |
Seasonal rules deserve their own page, dated, with a short line for each: last order date per delivery method, extended returns window, holiday hours. Link it from the delivery and returns pages. The assistant then has one clear source, and in January you can take it down in one go.
Why October
Search interest and traffic for holiday shopping start climbing in early November in most of Europe and the US, and the biggest days come with it.1 Testing in October leaves time to fix pages before traffic peaks. After a fix, the platform has to re-read the changed page before the answer changes, and how fast that happens depends on how the content is connected. Intercom’s guide to content for Fin covers the details for its platform.2 Leave a few days.
If you want this done every week instead of twice a year, that’s what Assistant Care is: a person asks your assistant what customers ask, checks every answer against your pages, and fixes what drifted. If you’d like to see what a check looks like first, there is a sample.
Footnotes
-
Think with Google, holiday shopping insights, thinkwithgoogle.com. ↩
-
Intercom, “Optimizing content for Fin”, intercom.com/help. ↩