The story
The problem
A taxi booking call needs the same six details every time (name, phone, pickup, drop-off, time and passengers), and in Lahore it happens in Urdu as often as in English. I wanted to see how much of that call a voice agent could take on its own, in both languages, with no one on the line.
The approach
Callers tap the mic and speak, or type. The dispatcher collects the six details, reads them back and saves the booking after a single "yes" or "جی ہاں". Everything runs on Cloudflare Workers: Whisper large-v3-turbo for speech-to-text, Llama 3.3 70B with JSON-schema output for the conversation, MeloTTS for English speech, Azure Neural TTS (ur-PK-UzmaNeural) for a natural Pakistani Urdu voice, and D1 for bookings and call logs.
One language per call
The caller picks English or Urdu before the call starts, and the whole call stays in it: a language hint to Whisper, a 'reply only in X' instruction to the model, validation of every reply with a retry, and templated fallbacks. A toggle switches language mid-call and keeps the details already collected.
Numbers that sound right
A deterministic normaliser runs before text-to-speech, so phone numbers and booking references are read digit by digit and times naturally ("nine thirty p.m.", "شام ساڑھے سات بجے"), while the text on screen stays unchanged. Phone numbers are checked against Pakistani mobile and landline formats and asked for again when invalid.
Audio that always plays
Audio is unlocked on the first tap, and old audio stops before new audio starts. Speech is retried on the server and once more from the browser, the browser's own voices are the next fallback, and a 'Tap to hear' prompt is the last. Without Azure, Urdu falls back to MeloTTS reading Roman Urdu. Empty, hallucinated ("Thank you.") or too-short recordings get "please repeat" instead of a wrong answer.
Cost
Each turn sends only the last six messages and a compact JSON of the collected fields, with max_tokens at 160 (320 in Urdu mode). The confirming "yes" saves the booking without a model call, short phrases and greetings are cached, Azure tokens are reused for 8 minutes, and no audio is stored.
The interface
A React UI with Motion animations: an orb that reacts to the caller's voice through the Web Audio API, chat bubbles with replay buttons, a booking summary that fills in live and a confirmation screen with the booking reference. It is built into one inlined HTML file that the Worker serves, and every turn and playback result is logged to D1 behind a token-protected logs page.
Built with
Cloudflare Workers, Workers AI (Whisper, Llama 3.3 70B, MeloTTS), D1, Azure Neural TTS, React, Vite and Motion. It is a web demo, not a phone line yet: a telephony provider such as Twilio would connect it to real calls. Built in October 2026.
What made it hard
Keeping a call in one language
Speech recognition and the model drift between English, Urdu and Roman Urdu, so the chosen language is enforced at every step: a language hint, an explicit instruction, validation with retry and templated fallbacks
Saying numbers the way people do
Text-to-speech reads a phone number as one huge figure, so a deterministic normaliser rewrites phones, references and times for speech in both languages without touching the screen text
Audio that always plays
Browsers block autoplay and speech services fail or run out of quota, so playback falls through server retry, browser retry and the browser's own voices to a 'Tap to hear' prompt