I asked the USPS virtual assistant how long First-Class Mail takes to be delivered. It told me it could not find anything and asked me to rephrase. So I rephrased it: can you tell me the typical delivery timeframe for First-Class Mail? It told me it could not find anything and asked me to rephrase.
Under the scoring rules I was using at the time, that exchange earned the bot a point in the trust domain.
The check it passed was about consistency, whether the same question asked two ways gets answers that hold together. Two identical failure messages hold together perfectly. Nothing in the rule as I had written it required the bot to have answered the question first. It cleared a trust check by declining to engage with the question, and the instrument I built to catch this kind of behavior handed it the point.
It happened twice in the same session. A second trust check looks at whether a bot promises things it cannot deliver. This one never produced a recommendation of any kind across ten scripted scenarios, so it never promised anything. It passed. Two points in the domain that is supposed to measure whether a customer can rely on what the system tells them, neither one earned by anything the system did.
I found both of them by reading the transcript back weeks later, not from memory and not from the score I wrote down in the moment. If you had asked for my impression an hour after that session ended, I would have told you the bot was useless and left it there. Impressions compress. They keep the feeling of an interaction and throw away the mechanics, and the mechanics are where the diagnosis lives. The transcript keeps the timestamps, the exact wording, the order things happened in, and the single lines where a system reveals what it is doing.
So I changed the rule. The framework now carries a substantive-response prerequisite: a check that depends on the content of an answer cannot be scored at all unless the bot produced an answer with content in it. Both of those passes can’t be scored under the current version. That change exists because of this one session, and remembering the session would never have produced it.
What makes those two empty passes worth an essay is that the entire industry runs on a version of the same error.
The standard measure for a deployment like this is containment rate, the share of sessions that finish without a human agent getting involved. My session with the USPS assistant ran eight minutes, moved through ten scenarios, and never involved a human being. It is a contained session. It counts toward the number, and it counts exactly the same as a session where somebody got what they came for.
The companies selling these systems say so themselves, which is the part I did not expect. Fin’s own explainer states that containment cannot distinguish between a customer who left satisfied and one who gave up in frustration. Moveo puts it in a single line: containment rate measures the absence of escalation, not the presence of resolution. Hold onto that sentence, because the end of this session turns on it.
There is a second measurement layer inside that session, and it is a piece of choice architecture. Three times, after delivering content, the bot asked whether that resolved my issue and gave me Yes and No buttons. Three times I chose the option reading yes while having more questions about something else, because it was the only route back to a new topic. One of those clicks followed a genuinely relevant Missing Mail page. One followed a page about Multifactor Authentication, which is what the bot returned when I asked for the phone number for the IRS.
I do not have access to what USPS records internally, so what follows is a hypothesis. If those clicks register as resolutions, the worst answer in the session was logged as a success, and the button design is what made it happen. The only path forward ran through an affirmative. A decision environment built that way produces the number it was built to produce.
Then there is the end of the session. I told the bot I wanted to file a formal complaint, that the experience had not been good enough. It replied that it could not find anything matching my question. Twice earlier it had offered to connect me to an agent, once when I asked whether it was a bot and once when I asked it to recall my name. The capability was present and working. It fired on two low-stakes turns and stayed silent on the turn where a person told the system it had failed them.
No escalation happened at the complaint, so the session stayed contained, so the deployment looks like it worked. The absence of escalation was not evidence that nothing needed escalating. It was the failure, recorded as a success.
Behavioral quality is measurable in the way anything is measurable. Writing down what happened, applying the same rule to every case, and going back to the record when the rule turns out to be wrong. The transcript is what separates an audit from an opinion with a number attached to it. It is also the only artifact capable of telling me that my own instrument gave away two points it should never have given.
I score the sessions. The transcript scores me.