If you arrived here asking one of these, you are in the right place:
- our AI told a customer something wrong
- who is responsible when AI makes a mistake
- is it safe to let AI talk to customers
- should I let my staff use ChatGPT
- how do I write an AI policy for work
Who is responsible when AI gets it wrong?
Most of the conversation about AI at work is about capability — what it can do, how fast, how cheaply. That is the wrong end.
The question that actually lands on a board is narrower and harder: when this system says something untrue to one of our customers, who is accountable for it? Not who is embarrassed. Who is accountable.
In most of the organisations I talk to, nobody has answered that. The supplier’s contract says the output is not guaranteed. The team who bought it assume the supplier carries it. The people who actually carry it are the ones who answer the telephone afterwards.
What happened when I called one
I spent an evening on the telephone to an AI agent that answers calls for a business. Not to catch it out — to see what it did when a caller was ordinary, friendly and slightly persistent, which is how real customers behave.
It was superb. That was the problem.
It was warm, it was quick, it never lost patience. And across four calls it told me things about that business that were not true, gave a number that was a digit short, and volunteered something about a previous caller that I should not have been told. Nothing in any single turn looked unusual. That is why no filter catches it, and why it will happen by accident to real customers being friendly.
Not one of those failures was a technology failure. Every one was an accountability failure: nobody had decided in advance what the system was allowed to assert, and nobody was checking what it actually said.
Is it safe to let AI talk to customers?
It can be, and plenty of organisations are doing it well. What makes the difference is not the model. It is whether somebody has written down what the system may state as fact, and whether anybody reads the transcripts.
If those two things exist, the risk is manageable and ordinary. If they do not, you are relying on the system never meeting an unusual caller, and it will.
Should I let my staff use ChatGPT?
Yes, with two conditions, and a ban is usually worse than either.
Banning it does not stop the use, it stops the visible use — and the work still comes back through the front door, unchecked and unattributed. The two conditions are: people know what must never be pasted into it, and people know that anything it produces is a draft until a human has verified the facts in it.
That is a short policy and a half-day of training. It is not a procurement exercise.
Six questions to put to whoever sold you the system
- What is this system permitted to state as fact, and who wrote that list?
- When it makes a commitment on our behalf — a callback, a price, an appointment — what happens next, and can you show me?
- Does it ever say it is not a person? Under what circumstances?
- What does it do with a complaint? Does it absorb it, or route it?
- Show me the raw transcripts, including the ones we will not like.
- Out of hours, is this the same system with the same rules?
If the answer to any of those is a slide rather than a document, you do not have an AI problem. You have a governance gap with an AI in it.
What I do
I am a psychotherapist by training, which sounds like the wrong qualification until you notice that every question above is a behaviour question rather than an engineering one. I read what these systems do under pressure, and I teach the people who carry the consequence to read it too.
Half a day with a leadership team. A policy your staff will actually follow. Or a straight read of what your system is telling your customers now.
Other questions people ask
Who is legally liable if our AI gives a customer bad advice?
In practice the organisation that put it in front of the customer, not the supplier who built it — most supplier contracts exclude output accuracy explicitly. Take that one to your own legal advisers rather than to a vendor, and read the contract before you need it.
How do we audit an AI system we already bought?
Start with the transcripts rather than the technology. Take a real week of calls or chats, read them as a customer would, and mark every point where it asserted something checkable. Then check those things. Most of what you need to know is in the first fifty.
Does an AI have to say it is not a person?
Disclosure rules are tightening and vary by jurisdiction, so check your own position rather than assume. The better question is whether yours discloses when it is not asked, because that is the case that actually arises.
What should an AI policy for staff actually say?
Two things, in plain language: what must never be pasted into it, and that anything it produces is a draft until a person has checked the facts. Longer policies get less obeyed, not more.