We launched the AI wait-time concierge on phone in March. Every caller who interacted with the AI got a one-question post-call survey: 'Did the AI help, hurt, or neither?'. 12,000 responses later, we have data. Two findings reframe how we build the next version.
Finding 1: callers do not want to talk to AI. They want the answer.
When we asked 'would you have preferred a human?', 71% said yes. When we asked the same callers 'did the AI get you the answer you needed?', 64% said yes. When we asked 'would you have preferred to wait 4 more minutes for a human?', only 19% said yes.
Callers do not want AI. They want the answer fast. AI is a means, not an end. If the AI delivers the answer, the preference for a human evaporates.
Finding 2: identifying as AI does not hurt as much as we feared
“Half our test was AI-honest. The honest version had 19% fewer follow-up complaints. The math says: identify the AI.”
Half our test traffic was 'this is the team's AI assistant, what can I help with?' and half was 'thanks for calling [company], how can I help?'. The honest-AI version had a 4% lower 'AI helped' rating but a 19% lower follow-up complaint rate. Callers who knew they were talking to AI did not feel deceived later, and complained less. The math says: identify the AI.
Finding 3 (the one that surprised us): voice quality matters more than smarts
We A/B tested two AIs: a smaller model with our cloned brand voice vs a larger model with the default neutral TTS. Same prompts. The brand-voice version rated 22% better on 'felt natural' and 14% better on 'I trust this'. The smaller model with better voice beat the smarter model with worse voice.
What we changed in the product
Three changes. (1) The AI always identifies as AI when asked, and proactively on the first interaction. (2) We default to brand-voice cloning when the customer uploads a sample. (3) We invested in faster first-token latency over a smarter model. Our default is now a faster, lighter model tuned for low latency, with larger models available for customers who turn them on.