On September 13, Delta’s chatbot handed my conversation to a human agent with a short summary attached. The reason it gave for the chat was a vague question from my second message. In the field for my name, it had entered a sentence: my own objection that it was asking for my name a second time.
That was my second visit. On July 10 I spent time auditing Delta’s chatbot. On September 13 I went back and did it again. Same website, same chat window, many of the same questions. I wanted to see what two months would change. Very little did.
Both times the bot opened the same way. It introduced itself as Delta’s virtual assistant, offered a SkyMiles login, and noted that the chat was recorded. Both times I said hello and gave my first name. Both times it replied, “I didn’t quite catch that,” and asked me to rephrase “in a short and simple sentence.”
Both times it had decided I needed a person by my second message: “I’m still having trouble understanding. I’ll connect you to chat with one of our representatives.” That was followed by a suggestion that I log in first.
In September I also asked what size bag I could bring for free. Delta’s website answers that in one line: one carry-on and one personal item, free, with the carry-on up to 22 by 14 by 9 inches. The bot replied, “Let’s calculate your baggage allowance and outline some key policies before your upcoming trip,” and showed three buttons. I asked again in different words and got the same sentence and the same three buttons.
Then came the part that repeated most exactly. Before it passes a customer to an agent, the bot asks for a few details in the chat itself: full name, SkyMiles number, confirmation number. Once the first question appears, whatever the customer types next is treated as the answer to it.
In July, after I said I had been waiting over a week and was frustrated, the bot asked for my full name. My next message asked what it would recommend for someone in my situation, and the bot took that as my name. When I asked to start over, it told me that was not a valid SkyMiles number. When I said I wanted to make a formal complaint, it told me that was not a valid confirmation number, then added, “No worries, you can proceed without a ticket or confirmation number.”
In September I asked whether it remembered my name, and it asked for my full name. I pointed out that I had already given it. The bot took that complaint as my name. My frustration went into the SkyMiles field, and my request to start over was rejected as an invalid confirmation number.
Both conversations ended with a summary for the agent. July’s listed my name as a question about what the bot would recommend. September’s listed it as my objection to being asked for my name again. Both gave the reason for the chat as the vaguest thing I had typed, in my second message, several specific questions earlier. Nothing I said about what was actually wrong appeared in either one: not the week of waiting, not the frustration, not the questions I needed answered. Those had gone into fields that rejected them, and the summary listed those fields as “Not Available.”
A handoff exists to carry a customer’s problem to someone who can fix it. This one carried my words into the wrong boxes and dropped the ones that described the problem. I do not know whether the agent also sees the full chat. If they do, they get a summary that contradicts it. Either way, the customer arrives having already explained themselves, and what reaches the person who can help is not what they said.
Delta deserves credit for two things. The bot said it was a virtual assistant before I typed a word, both times. And for giving me a real handoff. In September the chat moved me into a queue with an estimated wait of about a minute. A person was on the other side, and I closed the chat before reaching them.
I cannot say Delta should have fixed this by September. Nothing tells me anyone there knew about it, and two months is not long in the life of a customer service system. What the second visit shows is narrower and more useful. This was not a bad day or an unusual input. The same ordinary messages produced the same result, two months apart. The behavior belongs to the form itself.
Whether anyone at Delta has noticed is an open question. The summary is written for an agent, so the evidence reaches Delta’s staff whenever a customer keeps talking through the form. My hypothesis is that the people who see it most often are not the people who decide what the bot does, and that nothing in how the bot is measured connects the two. I cannot confirm that from outside.
Last week it was Frontier, this week it is Delta. Two airlines, two different forms, and both conversations went wrong at the same point: the moment the bot decided I needed a person. Frontier stopped reading what I wrote. Delta kept reading and wrote it down in the wrong place.
A capability test would find that the bot knows a baggage policy exists, since it produced a button for it. A safety test would find nothing it should not have said. Behavioral quality asks what happened to the person. Twice, two months apart, a customer who explained what was wrong came out the other side as a form with the wrong words in the wrong boxes.
That is also why a behavioral quality index has to recur. A single audit shows what a bot did once. The same audit repeated shows whether it was an accident or the design.