Beyond WER: Building and Evaluating Multilingual Voice Intelligence

Beyond WER: What Makes Multilingual Voice AI Good?An AI Circle x LILT panel + happy hour · Tuesday 17 November · Google, 345 Spear St, San Francisco Voice AI no longer just transcribes and speaks. Today’s systems listen, reason, respond and act, across dozens of languages, in real time. But we still measure them the old way. Word error rate can mark down a transcript that got the meaning right. It can also miss a tonal error that changes the meaning completely. And it often moves more with formatting conventions than with real differences between models. Meanwhile, the gaps that matter most in production go unmeasured: accents and dialects, code-switching, noisy real-world audio, domain terminology, and whether a voice agent actually gets the task done. LILT CEO Spence Green joins researchers from frontier labs and enterprise AI teams to ask what should come after WER. No slides, no pitch. Just the people building and evaluating these systems. What we’ll cover Evaluation Semantic and human evaluation beyond word error rate Benchmarks built natively in each language, not translated from English Improving long-tail languages without regressing the rest Data Real-world vs. synthetic audio Expressive, rights-cleared TTS voices Responsible low-resource speech collection The big question What does it mean for a multilingual voice system to be truly good? Agenda 4:00 – 4:30pm · Arrival, check-in and drinks 4:30 – 5:15pm · Panel: Spence Green (LILT) + [panellists TBC] 5:15 – 7:00pm · Happy hour and networking 7:00pm onwards · Drinks continue at a nearby bar [Location TBC] Who will be thereResearch scientists, evaluation and benchmarking leads, speech and multilingual teams, international product owners, and data and annotation leads. They come from frontier labs, enterprise AI teams and startups. Who should comeThis room is for you if you’re responsible for whether a voice or language model works outside English. That includes building or evaluating multilingual speech systems, running benchmarks that have to hold up across languages, shipping voice products into non-English markets, or sourcing the audio data these systems train on and are measured against. Space is limited and every registration is reviewed. Feel free to share this with colleagues. Plus-ones need to apply separately. About LILTLILT is an AI-native multilingual platform working across frontier AI data, enterprise localization, language-grounded alignment and multimodal evaluation. About AI CircleAI Circle is an invite-only community of AI practitioners advancing the frontier, with chapters in San Francisco, New York, Seattle, Toronto, London and Paris, plus campus circles at MIT and Stanford.