Human-in-the-loop AI for customer service: where the line should be

Every few months a company ships a fully autonomous support bot, and every few months one of them ends up in a screenshot: inventing a refund policy, promising a delivery date that doesn't exist, or cheerfully agreeing to sell a car for one dollar. The reaction is predictable — "AI isn't ready for customer service" — and it's the wrong lesson. The bot wasn't the problem. The absence of a rule for when a human has to look was the problem.

The opposite failure is quieter and far more common: a support team that uses no AI at all, drowning in order-status questions while the genuinely hard conversations wait in the same queue. Neither extreme is a strategy. What works in practice is boring and specific: a written division of labour between people and models, and a workflow that enforces it.

What AI is genuinely good at in a support inbox

Four jobs, in order of how safe they are to hand over:

1. Classification. Intent, urgency, sentiment and customer value on every inbound message, in under a second. This is the foundation for everything else, and it's almost impossible to get catastrophically wrong.
2. Summarising. Turning a six-message thread into three lines before an agent opens it. Pure time saved.
3. Drafting. Writing the first version of a reply from your knowledge base, tone guide and the customer's history. Good models produce drafts that an experienced agent accepts unchanged most of the time — and edits in seconds the rest of the time.
4. Scoring. Grading every closed conversation against your quality rubric. This is the one most teams underrate: it replaces the 2–5% QA sample most support teams live with, and turns coaching from guesswork into a weekly routine.

In most inboxes we audit, somewhere between a third and 40% of contacts are repetitive enough to self-serve entirely — order status, account access, "where's my invoice". Another quarter can be drafted for a human to approve. That's the realistic ceiling, and it's plenty.

What it's bad at, and always will be

Exceptions to policy. Angry customers who need to feel heard before they'll accept a solution. Anything involving money leaving your account. Situations your knowledge base has never seen. A model will produce a confident answer in every one of those cases, and confidence is exactly the wrong quality to have when you're guessing.

The question isn't "can the model answer this?" It's "what happens if the answer is wrong?" — and who finds out first.

The workflow that makes it safe

The version that works looks like this, for every inbound message:

1. The model classifies it and attaches a score for urgency, sentiment and value.
2. A routing rule decides: self-serve answer, draft for an agent, or straight to a person — with a hard escalation list (negative sentiment, high order value, certain keywords) that overrides everything else.
3. For drafted replies, an agent reads, edits if needed and sends. The model never sends.
4. After the conversation closes, the model scores it, and any new edge case becomes a knowledge-base entry.

The approval step is the whole game. It costs an agent a few seconds per reply and it means nothing a customer reads was produced unsupervised. Teams that skip it to save those seconds are the ones in the screenshots.

Write the rules down

Every rule above — what triggers automation, how it routes, when it escalates, what's never automated — belongs in a playbook: a short, versioned document that an operations lead can read and change without an engineer. When a rule misfires, you fix the playbook, not the prompt. When you change vendors, the playbooks come with you. When a new agent joins, the playbooks are the onboarding.

How to know it's working

Skip "tickets handled by AI" — it rewards the wrong behaviour. Track instead:

• Deflection rate by contact type — self-serve should be winning on the boring categories and near zero on the sensitive ones.
• Draft acceptance rate by agent — if experienced agents accept drafts unchanged most of the time, the drafts are good. If they rewrite them, the knowledge base or tone guide needs work, not the model.
• Handle time and first-response time — against your pre-automation baseline, by channel.
• CSAT and QA score — these should hold or rise. If CSAT drops while deflection climbs, you've automated something customers wanted a human for.

Where to start

Pull the last few thousand conversations. Rank contact drivers by volume and by risk. Automate the top two or three high-volume, low-risk drivers first, with the approval step in place from day one. Leave everything else with people. Review the numbers monthly, promote a category to self-serve only when the draft acceptance rate has earned it, and never remove the human from the loop to hit a target. The teams that get this right aren't the ones with the cleverest models — they're the ones with the clearest rules.