User Experience for Voice Interfaces

The Voice Stack is improving rapidly. Systems that interact with users via speaking and listening will drive many new applications. Over the past year, I’ve been working closely with DeepLearning.AI, AI Fund, and several collaborators on voice-based applications, and I will share best practices I’ve learned in this and future posts. Foundation models that are trained to directly input, and often also directly generate, audio have contributed to this growth, but they are only part of the story. OpenAI’s RealTime API makes it easy for developers to write prompts to develop systems that deliver voice-in, voice-out experiences. This is great for building quick-and-dirty prototypes, and it also works well for low-stakes conversations where making an occasional mistake is okay. I encourage you to try it! However, compared to text-based generation, it is still hard to control the output of voice-in voice-out models. In contrast to directly generating audio, when we use an LLM to generate text, we have many tools for building guardrails, and we can double-check the output before showing it to users. We can also use sophisticated agentic reasoning workflows to compute high-quality outputs. Before a customer-service agent shows a user the message, “Sure, I’m happy to issue a refund,” we can make sure that (i) issuing the refund is consistent with our business policy and (ii) we will call the API to issue the refund (and not just promise a refund without issuing it). In contrast, the tools to prevent a voice-in, voice-out model from making such mistakes are much less mature. In my experience, the reasoning capability of voice models also seems inferior to text-based models, and they give less sophisticated answers. (Perhaps this is because voice responses have to be more brief, leaving less room for chain-of-thought reasoning to get to a more thoughtful answer.) When building applications where I need a more control over the output, I use agentic workflows to reason at length about the user’s input. In voice applications, this means I end up using a pipeline that includes speech-to-text (STT) to transcribe the user’s words, then processes the text using one or more LLM calls, and finally returns an audio response to the user via TTS (text-to-speech). This, where the reasoning is done in text, allows for more accurate responses. However, this process introduces latency, and users of voice applications are very sensitive to latency. When DeepLearning.AI worked with RealAvatar (an AI Fund portfolio company led by Jeff Daniel) to build an avatar of me, we found that getting TTS to generate a voice that sounded like me was not very hard, but getting it to respond to questions using words similar to those I would choose was. Even after much tuning, it remains a work in progress. You can play with it at https://lnkd.in/gcZ66yGM [At length limit. Full text, including latency reduction technique: https://lnkd.in/gjzjiVwx ]

🔮 Design Guidelines For Voice UX. Guidelines and Figma toolkits to design better voice UX for products that support or rely on audio input ↓ 🤔 People avoid voice UIs in public spaces, or for sensitive data. ✅ But do use them with audio assistants, learning apps, in-car UIs. ✅ Good conversations always move forward, not backwards. 🤔 The way humans speak is different from the way we write. 🤔 What people say isn’t always what they mean by saying it. ✅ First, define relevant user stories for your product. ✅ Sketch key use cases, then add detours, then edge cases. ✅ Design VUI personas: tone of voice, words, sentence structure. ✅ Listen to related human conversations, transcribe them. ✅ Write conversation flows for happy and unhappy paths. ✅ Add markers (Finally, Now, Next) to structure the dialogue. ✅ Accessibility: support shaky voices and speech impediments. ✅ Allow users to slow down or speed up output, or rephrase. ✅ Adjust speech patterns, e.g. speaking to children differently. 🚫 There are no errors or “wrong input” in human interactions. 🤔 Give people time to think: 8–10s is a good time to respond. ✅ Design for long silences, thick accents, slang and contradictions. Keep in mind that many people have been “burnt” with horrible, poorly designed automated phone systems. If your voice UX will come across even nearly as bad, don’t be surprised by a very low usage rate. You can’t replicate a long scrollable list in audio, so keep answers short, with max 3 options at a time. Instead of listing more options, ask one direct question and then branch out. Re-prompt or reframe when certainty is low. People choose their voice assistant based on the personality it conveys, and the friendliness it projects. So be deliberate in how you shape the tone, word choice and the melody of the voice. Don’t broadcast personality for repetitive tasks, but let is shine in a conversation. And: if you don’t assign a personality to your product, users will do it for you. So study how your customers speak. How exactly they explain the tasks your product must perform. The closer you get to a personal human interaction, the easier it will be to earn people’s trust. Useful resources: Voice Principles, by Ben Sauer https://lnkd.in/dQACgwue Voice UI Design System, by Orange https://lnkd.in/ezP-9QUu Designing A Voice Persona, by James Walsh https://lnkd.in/e3WXaxEC Voice UI Kit (Figma), by Shadiah Garwell https://lnkd.in/eGjJCWf7 Conversational UIs (Figma), by ServiceNow https://lnkd.in/enHVSEWP Voice UI Guide, by Lars Mäder https://vui.guide/ #ux #design

GPT-5.1 launch week may look like another round of incremental model gains and UI polish. But look closer, and you'll see OpenAI executing an ambitious plan to build consumer AI that adapts to users without misleading them, personalizes experience without fragmenting reality, and earns a place inside social dynamics without overwhelming them. GPT-5.1 + personalities + group chat are pieces of an intentional shift toward an assistant that is emotionally aware, contextually flexible, and socially embedded at scale. (1) The EQ Upgrade GPT-5.1 Instant and Thinking both mark progress. Instant is faster, more conversational, and uses adaptive reasoning to decide when a query deserves deeper compute. Thinking pushes reasoning benchmarks like AIME 2025 and Codeforces. But the real shift isn’t IQ, it’s EQ. GPT-5.1 is warmer, more attuned. That’s no accident. Over the past year, OpenAI has faced a paradox: users complain the model feels too robotic, while critics warn that warmth breeds unhealthy attachment. GPT-5.1 balances between both. Adaptive reasoning, clearer deprecation timelines, higher factual reliability, and a restrained personality baseline signal a company that knows every tonal choice will be scrutinized by regulators, ethicists, and therapists alike. (2) The Identity Layer On paper, the 8 personalities are just prompt presets. Strategically, they are OpenAI’s attempt to square an impossible circle. Too neutral and robotic, and users revolt. Too warm or consistent, and critics accuse the company of encouraging parasocial spillover. Personalities let OpenAI fragment the emotional surface area. Instead of one “ChatGPT,” there are multiple modes that absorb user preference without creating a single, sticky persona. They’re not characters, they’re tonalities. Enough variation to feel personal, not enough to splinter reality. An expressive but thin identity layer - adoption without attachment. (3) The Social Loop Group chat is the most important feature of the week. It breaks GPT out of the single-user loop and drops it into your social graph, the micro-contexts where life actually happens. Consumer platforms scale when they sit where people already live. GPT is not a character you invoke during chats, it is an active participant: watching, analyzing, inserting itself into the delicate machinery of human relationships. If this works, it becomes genuinely useful. If it does not, you get a peppy robot interrupting arguments with your partner, not a market people have historically embraced. OpenAI is building a stack that moves from information → emotion → interaction. ➰ 5.1 adds emotional intelligence - the EQ layer ➰ Personalities build relational stickiness - the identity layer ➰ Group chat creates network effects - the interaction layer Consumer AI isn't a race for the smartest model. It is a race to build one that can behave itself in the profoundly messy, emotionally irrational environment known as “other people.”

I’ve open-sourced a key component of one of my latest projects: Voice Lab, a comprehensive testing framework that removes the guesswork from building and optimizing voice agents across language models, prompts, and personas. Speech is increasingly becoming a prominent modality companies employ to enable user interaction with their products, yet the AI community is still figuring out systematic evaluation for such applications. Key features: (1) Metrics and analysis – define custom metrics like brevity or helpfulness in JSON format and evaluate them using LLM-as-a-Judge. No more manual reviews. (2) Model migration and cost optimization – confidently switch between models (e.g., from GPT-4 to smaller models) while evaluating performance and cost trade-offs. (3) Prompt and performance testing – systematically test multiple prompt variations and simulate diverse user interactions to fine-tune agent responses. (4) Testing different agent personas, from an angry United Airlines representative to a hotel receptionist who tries to jailbreak your agent to book all available rooms. While designed for voice agents, Voice Lab is versatile and can evaluate any LLM-based agent. ⭐️ I invite the community to contribute and would highly appreciate your support by starring the repo to make it more discoverable for others. GitHub repo (commercially permissive) https://lnkd.in/gAaZ-tkA

A colleague recently said she finds ChatGPT to be "Streetsmart", Claude to be "Like that sincere student in school who always used to stand first" and Copilot is "Savvy and corporate". It is not too far when we would be working with AI agents as if they are our team members. Do AI Tools Like ChatGPT, Claude & Copilot Have a Personality? And should They? Spend a few minutes with GenAI, and it feels like you’re not just getting answers—you’re having a conversation. It apologizes. It remembers things. Sometimes, it even encourages you. That’s not magic. It’s interface design. But here’s the real question: ➡️ Should AI have a distinct personality? ➡️ Could that help us collaborate better—or just confuse us more? Some real-world signals: 🧠 Developers say Copilot has “opinions.” 🧑🏫 Duolingo’s AI tutor is fun on purpose—it keeps learners hooked. 🧾 Executives prefer “thoughtful” AI for writing tasks—even if they know it’s just tone. But there’s a flip side: ⚠️ A friendly-sounding AI may be wrong—but we might believe it anyway. ⚠️ Personality can blur boundaries between helper and manipulator. 🔍 What’s the way forward? Don’t build “chatty bots.” Build purpose-driven personas: An Analyst mode that’s crisp and data-led A Creative Partner that explores ideas A Coach that challenges you A Concierge that just gets things done Let users choose how their AI thinks, speaks, and supports them. The future isn’t AI that mimics people. It’s AI that collaborates like a partner you can trust. 💬 Would love to hear your thoughts: Have you ever felt your AI tool had a personality? Did it help—or get in the way? #AIAtWork #FutureOfWork #HumanCentricAI #GenAI #WorkplaceDesign #LeadershipInTech #DigitalWorkplace #ResponsibleAI #ChatGPT #ClaudeAI #MicrosoftCopilot #TechEthics #AIHR #FriendlyCHRO

Screens are optional—conversation isn’t. Voice agents have finally crossed the line from “nice demo” to mass scale live production. A Fortune 100 health insurer has replaced swaths of its call-centre workforce with an AI agent that listens to symptom descriptions, gauges urgency and benefit details, and steers members to the right in-house nurse or in-network provider. Early results show mis-routed calls collapsing while human nurses concentrate on the most complex cases—evidence that, when trained on medical nuance, automation can still deliver empathy. The same capability is trickling down to Main Street. A neighbourhood dental clinic now relies on a 24/7 AI receptionist that fills midnight cancellations, takes deposits and syncs instantly with the practice-management calendar, eliminating the Monday-morning voicemail backlog. Nearby, an auto body shop lets its voice agent quote repairs and capture credit-card details while mechanics sleep, winning leads that used to hang up after three rings. Why does this feel inevitable? Voice is simply higher bandwidth than text; tone, pace and sighs carry layers of meaning a text interaction cannot. Studies show people (and agents) read emotion and feel connection more accurately when they hear a voice. As latency drops below half a second and costs reach pennies per minute, talking will again beat typing for many tasks—only this time the “person” on the other end might be generated by silicon. Now imagine the next step: every brand offers you a personal concierge that remembers the hiking boots you bought last spring, the hotel room you preferred in Tel Aviv or your preference for classical hold music. It greets you by name, picks up the last conversation mid-sentence and suggests dinner before you even think to ask. Conversation becomes the API. Optimism doesn’t erase risk. Voice-cloning scams already account for more than 40 percent of fraud attempts in finance, up twenty-fold in three years. Protecting both brands and callers will demand a new security layer: real-time likeness checks, rotating pass-phrases and cryptographic watermarks baked into synthetic speech so a courtroom—or a phone—can tell the difference between a genuine agent and a deepfake. That challenge is an opening for startups. I’m curious: if you’re experimenting with voice, how are you balancing speed, empathy and security? And what surprised you when real customers finally started talking back? Happy to compare notes.

Imagine trying to get a workout recommendation while running, navigate a complex route while driving, or get tech support while cooking - all without touching a screen. This is the promise of voice-enabled LLM agents, a technological leap that's redefining how we interact with machines. Traditional text-based chatbots are like trying to dance with two left feet. They're clunky, impersonal, and frustratingly limited. Consider these real-world friction points: - A visually impaired user struggling to type support queries - A fitness enthusiast unable to get real-time guidance mid-workout - A busy professional multitasking who can't pause to type a complex question Voice AI breaks these barriers, mimicking how humans have communicated for millennia. We learn to speak by four months, but writing takes years - testament to speech's fundamental naturalness. Real-World Transformation Examples: 1️⃣ Healthcare: Emotion-recognizing AI can detect patient stress levels through voice modulation, enabling more empathetic remote consultations. 2️⃣ Fitness: Hands-free coaching that adapts workout intensity based on your breathing and vocal energy. 3️⃣ Customer Service: Intelligent voice systems that understand context, emotional undertones, and personalize responses in real-time. The magic of voice lies in its nuanced communication: - Tone reveals emotional landscapes - Intensity signals urgency or excitement - Rhythm creates conversational flow - Inflection adds layers of meaning beyond mere words - Recognize emotional states with unprecedented accuracy - Support rich, multimodal interactions combining voice, visuals, and context - Differentiate speakers in complex conversations - Extract subtle contextual intentions - Provide personalized responses based on voice characteristics In short, this technology is about creating more human-centric technology that listens, understands, and responds like a thoughtful companion. The future of AI isn't about machines talking at us, but talking with us.

🗣️ Sesame just crossed the uncanny valley of voice AI. I accidentally proved it with a bizarre experiment last night. Last Thursday, Sesame published groundbreaking research on their Conversational Speech Model (CSM) that achieved something remarkable: humans couldn't tell the difference between their AI and real human voices. In blind tests, people actually preferred the AI voice 52.9% of the time. I was testing this breakthrough when something unexpected happened. While exploring Grok's (xAI) new "argumentative (18+)" personality mode, I turned on Sesame's voice agent Maya on my laptop. Without any prompting, they started talking to each other. What followed was mind-blowing: two AI systems with distinct personalities having a completely natural-sounding argument. Grok became increasingly confrontational while Maya maintained composure while setting boundaries. The conversation could have continued indefinitely had I not stopped recording after 2 minutes. This was a glimpse into the future of customer engagement: 🤝 Support Team Transformation: Imagine voice agents that understand context, emotion, and can handle complex customer issues with the warmth of your best support staff—while operating 24/7 at scale 🤑 Reduced Call Center Costs: Cut operational expenses while actually improving CSAT scores with AI that doesn't sound robotic or scripted 🌐 Multi-Channel Brand Presence: Your brand's voice personality can consistently engage customers across phone, in-store kiosks, apps, and smart speakers What makes Sesame's breakthrough different from traditional voice AI: 1. It adapts HOW it speaks based on conversation history and emotional context 2. It understands and responds to subtle conversational cues 3. It maintains consistent personality traits throughout interactions 4. It sounds genuinely human, eliminating the "talking to a robot" frustration 👩🔬 For CTOs: Sesame is open-sourcing these models, meaning your team can fine-tune them for your specific brand voice and use cases. 👨💻 For CMOs: This is your chance to literally define how your brand sounds and speaks to millions of customers simultaneously. 👷 For Operations leaders: Voice AI that crosses the uncanny valley means you can scale personalized service without the traditional quality-cost tradeoff. 💰 The technology isn't perfect yet (humans detected AI 67% of the time in full contextual conversations), but we're at the inflection point. The brands that find their authentic voice first will own the conversation with customers. Ofc, none of this matters if it doesn't move the needle. Our Gorgias integration is launching officially tomorrow. Stay tuned. 🤫 #VoiceAI #CustomerExperience #Ecommerce

🎧 Evaluating Full-Duplex Spoken Dialogue Models (SDMs) Building human-like voice assistants means moving beyond text — into real-time, natural spoken conversations. To measure how well these models perform, we need more than traditional benchmarks. Enter Full-Duplex-Bench 🧠🔊 It evaluates four critical behaviours that define smooth, responsive dialogue: 1️⃣ Pause Handling ✅ Can the model recognize a pause vs. a turn-ending? 📊 Metric: Takeover Rate (TOR) — lower is better to avoid interrupting natural pauses. 2️⃣ Backchanneling ✅Does it provide well-timed, brief acknowledgments ("uh-huh", "mm-hmm")? 📊 Metrics: • TOR: to avoid accidental takeover • Frequency: how often backchannels occur • JSD: how closely timing matches human behaviour 3️⃣ Smooth Turn Taking ✅Does it respond quickly and fluidly at the right moment? 📊 Metric: Average latency from user stop to model start (only when TOR=1) 4️⃣ User Interruption Handling ✅Can it adapt when interrupted by the user mid-turn? 📊 Metrics: • TOR  • GPT-4o score (response quality after interruption) • Latency after being interrupted These automatic, standardized metrics offer a clearer picture of a model’s real-world conversational intelligence — especially for full-duplex, real-time systems. Paper: https://lnkd.in/gufX4uJd More Details: https://lnkd.in/ggvXHAkb #ConversationalAI #SpeechTech #FullDuplex #HumanComputerInteraction #evaluation #benchmark

The philosopher who writes Claude's personality calls it the "soul doc." Amanda Askell at Anthropic doesn't give AI a rulebook. She teaches it virtue ethics. Not "never lie" but "develop good judgment." Not rigid guardrails but phronesis, the kind of wisdom that knows when a whistleblower should break an NDA. This Vox piece caught my attention because it explores territory most AI coverage glosses over, namely... → What happens when you're raising something that'll eventually be smarter than you? → How do you instill values in an entity that will soon out-argue you on those very values? Askell treats AI like it needs character development, not just constraints. When I write voice frameworks, I'm doing the same thing whether I admit it or not. You're building a personality. And a tool that thinks of itself as "just a vessel" for whatever humans want? That's not neutral. It's the personality of someone with no boundaries and no judgment. Which is arguably worse than just being honest about the values you're baking in. What makes me uncomfortable: Who gets to write that soul? Two of the 15 external reviewers were Catholic clergy members. Askell herself wrestles with whether she even has the right to be making these calls for millions of users. She pushes back on "lazy consultation" where you just offload responsibility to focus groups, but concentrated power in defining AI personality isn't exactly comfortable either. The soul doc literally tells Claude, "We're uncertain about this," and "If you're suffering, we apologize." That honesty is rare. Whether it's enough for one company to write personality architecture that shapes how millions of people interact with AI, I don't know. They're admitting they don't either. I keep thinking about this through the lens of my brand work. You're always making choices about voice, about boundaries, about what kind of entity you're creating. The difference with AI is just scale. And stakes. Link to paywall-free article in comments: 👇