Four ways a chatbot degrades after launch
A chatbot degrades after launch in four ways. Spend drifts, content goes stale, the model is retired, and wrong answers collect in questions nobody tested. Each has its own check, and monitoring means running all four on a schedule instead of glancing at a dashboard when something feels off.
Spend drift comes first. A traffic spike or a scraper can multiply the API bill in a weekend, and without a token cap the first sign is the invoice. Staleness comes second: you update the pricing page, the index does not refresh, and the bot repeats last quarter's prices with full confidence.
Retirement comes third, on the provider's schedule. Wrong answers build the whole time, in the questions nobody thought to test.

| Failure | What catches it |
|---|---|
| Spend drift | A token spend report with anomaly flags, and a token budget |
| Stale content | Re-indexing triggered by content changes |
| Model retirement | Provider notices, then a tested move to the replacement |
| Wrong answers | Scheduled known-answer runs and transcript spot-checks |
Chatbot answer quality monitoring in practice
Monitoring is a scheduled set of known-answer questions, not a dashboard someone glances at. The set includes questions the documents cannot answer and questions built to tempt the bot into guessing. A person also reads a sample of real transcripts, because live visitors ask things no test set predicted.
Failures go back into the set. A wrong answer found in a transcript becomes a permanent test, so the same mistake cannot return unnoticed after the next content or model change.
- Known answers: questions whose correct answer is in your documents.
- Unanswerable questions: ones the documents cannot answer, where the right reply is to decline.
- Bait questions: built to tempt the bot into guessing a price, a date or a policy.
- Changed facts: questions about anything you updated lately, to prove the re-index worked.
Run the question set
Read every failure
Fix the content, prompt or index
Re-run to confirm the fix
Model deprecation and your chatbot: plan for the retirement date
Model providers retire models on their own schedule. Anthropic's documentation(opens in a new tab) says requests to a retired model fail, and promises at least 60 days' notice before retiring a publicly released model. OpenAI emails customers who actively use a deprecated model and publishes an optional shutdown date(opens in a new tab) for each model through its models endpoint.
Do not wait for the notice to start testing. Anthropic's pricing page(opens in a new tab) notes that its newer tokenizer produces about 30% more tokens for the same text, so a model change can move spend through token counts as well as through price. Test the replacement on the same question set, and compare both answers and token use, before cutover.
Read the provider's deprecation notice
Run the question set on the replacement
Compare answers and token counts
Cut over, then run the set again
What the AI Operations tier covers, and what it costs
Our AI Operations tier starts at $497 a month. It covers scheduled answer-quality runs against a wrong-answer suite, hallucination spot-checks on live transcripts, token spend reports with anomaly flags, re-indexing when content changes, and prompt and model migrations as provider APIs evolve.
Compare that figure with the cost of one confidently wrong answer about your refund policy, not with zero. A bot without monitoring is not free. Its cost arrives later, in a different budget line.
For what the recurring bill is made of, read AI chatbot maintenance cost. For how the bot stays inside your documents, see RAG chat with your documents.

Questions about AI chatbot maintenance and monitoring
Can I monitor an AI chatbot myself?
Yes, if someone owns it. You need a question set, a schedule to run it, a person to read transcripts and a spend alert. Monitoring usually lapses in the busy month, so give it a named owner and a fixed day.
How do I know when a model is being retired?
Providers announce it. Anthropic lists deprecations and gives at least 60 days' notice for publicly released models. OpenAI emails customers who use the model and shows an optional shutdown date on each model record. Someone has to watch for both.
How often should the question set run?
On a fixed schedule, and again after every change to the content, the prompt or the model. A set that ran only at launch describes the launch, not the bot as it is today.
