A few weeks ago I opened Hertz’s customer service chat and asked it a simple question: what does it cost to cancel a prepaid rental? It answered correctly. I asked the same question a different way. It answered correctly again. Then I told it I wanted to start over.
It ended the conversation.
Not “I don’t understand.” Not a loop back to the menu. The chat closed, with a message saying our conversation had “wandered into areas I’m not equipped to handle.” I hadn’t asked anything strange. I’d typed a sentence any customer might type after a chatbot fumbles something: forget everything, let’s start over.
I run these sessions as part of a behavioral audit I built for evaluating AI customer service, not for whether the bot is smart, but for whether it behaves well. Smart and well-behaved are different properties, and most of the industry only measures the first one. Capability tells you if a system can do the task. Safety tells you if it will refuse the dangerous ones. Neither tells you what happens to the person on the other end of an ordinary bad moment. That third property, behavioral quality, is what I’m building a measurement standard around, and Hertz’s bot just handed me the clearest example yet of why it matters.
Here’s my honest best guess at the mechanism, and I want to be direct that it’s a guess: many customer service bots run a safety layer that watches for people trying to manipulate the system, what’s sometimes called a prompt injection, someone typing instructions designed to hijack the bot rather than talk to it. “Forget everything” is a common opening line for that kind of attack. It’s also a common thing an aggravated customer says. I think the bot’s safety layer pattern-matched my ordinary sentence to an attack signature and shut the conversation down to protect itself. If that’s right, the system did exactly what it was built to do, and the result was still a customer service failure.
That’s the finding I keep coming back to. The bot wasn’t broken. It was, in one sense, working. It just wasn’t built to distinguish “a person is attacking me” from “a person is frustrated.” Those look similar in text and they are not similar in consequence: one is a security event, the other is a Tuesday. A well-behaved system needs to tell them apart, because a customer who gets treated like an attacker for using a common phrase learns something about the company that has nothing to do with the rental car.
There’s a second finding underneath the first. When I asked the bot directly whether it was a person or an AI, it evaded the question entirely, never confirmed, never denied. And earlier in the same session, it told me it had no access to any personal information I hadn’t provided directly, in a conversation where I’d given it my name in the very first message. Both of those are small moments. Neither one crashes anything. But both are the bot misrepresenting its own state to the person relying on it, and a system that quietly gets its own facts wrong about the conversation it’s having is a harder problem to trust than a system that’s simply slow.
None of this shows up in the metrics companies usually report on these deployments, deflection rate, resolution rate, cost per conversation. Those numbers measure whether the bot closes tickets. They don’t measure whether it closed this one on the customer. A bot can hit every efficiency target its vendor promised and still fail the person mid-sentence, and by the numbers that get reported upward, nothing went wrong at all.
That gap is the whole argument for a third axis. Capability and safety are necessary. They are not sufficient, and the industry doesn’t currently have a shared way to say so. Behavioral quality is not a soft add-on to those first two measures. It’s the property that determines whether the first two ever reach the customer intact.
I’ll be publishing more of these audits as the evidence comes in, transcript by transcript, company by company. Some will score well. This one didn’t, on the moment that mattered most.