The agent answered my test call in a cloned version of my own voice, paused, and asked me to spell the company name I'd just said. Not the email — the company. Two minutes earlier I had told Claude Code to stop letting the agent guess at proper nouns, and it had quietly added one instruction to the system prompt. That pause is when I stopped thinking of the thing as a script and started treating it like an employee — and it's also the thesis of this post: the build is the easy part now. The iteration loop is the skill. Anyone can wire ElevenLabs to a calendar in an afternoon. The difference between a demo and a production agent is how fast you can spot wrong behavior and direct the next change.
The stack: Claude Code as the orchestration brain, ElevenLabs Agents as the voice and conversation engine, Cal.com as the booking backend. No Zapier, no Make, no glue services — three APIs, one .env file, and a widget snippet. The result answers site visitors in my voice, qualifies them across five fields, checks my real calendar in Central Time, and books a 30-minute discovery call. The first version was broken in five distinct ways. The fifth version works.
The Four Pieces of Any Voice Agent
Every voice agent is four components, and debugging is guesswork until you know which component you're inside.
Persona is the system prompt: tone, boundaries, objection handling, what to do when it doesn't know. Most builds paste "you are a helpful assistant" and wonder why the agent sounds like every AI on the internet. Mine bans the words "absolutely," "amazing," and "I'd be happy to" outright.
Voice is the synthesis layer. ElevenLabs gives you stock voices, Instant Voice Cloning from a few minutes of audio, or Professional Voice Cloning from thirty-plus minutes. I fed mine roughly four hours of my own recordings — past about two hours the clone picks up cadence and breathing patterns that shorter clones miss; past four, returns flatten.
Knowledge base is what it knows: my services doc and content transcripts, ingested as RAG documents the agent can quote from and — importantly — refuse to answer outside of.
Tools are what it can do, and this is where demo agents die. They can talk; they can't act. Mine has exactly two tools: check Cal.com slot availability, create a Cal.com booking. Narrow on purpose. The principles are the same ones I laid out in advanced tool calling for AI agents — an agent with two reliable tools beats an agent with ten flaky ones.
How the Build Actually Went
I briefed Claude Code in one paragraph — what the agent should do, the five qualification fields (name, email, company, problem, team size), the tone, the calendar target — and ended with "ask me whatever you need to know." It didn't start coding. It asked five questions first: event type slug, timezone, working-hours window, out-of-hours behavior, whether I had API keys ready. That question round matters; a builder who starts typing immediately rewrites the qualification flow an hour later when the requirements surface anyway.
Then it built in this order: the .env.local with stubbed keys (added to .gitignore unprompted — noticed and appreciated); the ElevenLabs agent created programmatically via API with the system prompt stored as a markdown file in version control so I could iterate on it like code; the two tools as serverless routes in the Next.js app; and finally the widget — two tags, an <elevenlabs-convai> element plus a script include, mounting a floating call button on the page.
One architectural decision deserves emphasis because it's a security decision wearing integration clothes: the agent never holds credentials. The ElevenLabs tools call my endpoints via webhook; my endpoints hold the Cal.com key server-side and make the real calls. If you paste an API key directly into an agent's tool config, that key lives in the agent definition. Route through your own proxy and the agent has nothing worth stealing. This pattern costs twenty minutes and eliminates a whole category of exposure — the same least-privilege thinking I covered in secure agent onboarding.
Five Bugs, Five Lessons
Silence on call open. The connection started and the agent just... waited, like an awkward Zoom call. ElevenLabs agents have a first_message field I hadn't specified, so it was empty. One config update, no redeploy.
The smile in the voice. The clone was accurate but every sentence ended in a slight upward lilt — customer-support energy, wrong for B2B sales. The fix lived in voice settings, not the prompt: stability up to 0.75, similarity up to 0.85, style exaggeration down to 0.3, plus one prompt line — "You are not selling. You are listening." That combination was the single biggest quality lever in the entire build.
The timezone bug. The agent claimed my earliest slot was 6 PM on a wide-open morning. Cal.com's v2 slots endpoint takes UTC inputs but needs an explicit timeZone parameter for output; without it, the agent read UTC times aloud as if they were Central. 11 AM Central is 4 PM UTC; add the minimum-notice window and you get 6 PM. Four lines of code. Write this on your wall: when an LLM agent does something weird with time, it's almost always a timezone bug and almost never the model.
The misspelled email. I said "mejba dot one three at gmail dot com"; the agent confirmed back "M-E-G-B-A." Voice transcription reliably fumbles proper nouns without dictionary context. The fix was a prompt block I now put in every voice agent: never trust transcription on proper nouns — read names, email local parts, and company names back letter by letter, and re-confirm after any correction. That block produced the pause that opens this post, and it's nearly free to add.
The booking refusal. The agent wouldn't book anything inside two and a half hours. Not a bug: Cal.com event types default to a 120-minute minimum-notice policy. I lowered it to 30 minutes in the event settings. The lesson generalizes — your agent inherits every default policy of every platform it touches, and no model can know which defaults you intend to override until you say so.
Total debugging time: about ninety minutes, with Claude Code reading the relevant docs page and proposing each fix before I'd finished reading the error.
Security, Because Voice Minutes Are Money
I nearly shipped without thinking about abuse, and voice agents punish that oversight specifically: unlike a text chatbot, every minute of conversation costs real money. ElevenLabs bills agent conversations per minute — roughly $0.08 to $0.10 depending on plan, with LLM costs on top — so an unprotected public widget is an open tap. Four layers before deploy:
- Hostname allowlist in the agent's security settings — connections from domains not on the list are rejected at the handshake, so nobody can paste your widget snippet into their own site and burn your credits.
- Duration cap — eight minutes per conversation. That's enough to qualify and book; a prospect who needs more belongs on a real call. It also caps the worst-case cost of any single abusive session under a dollar.
- Per-IP rate limiting on an edge function in front of the widget — a handful of sessions per hour, enough to block obvious automation without punishing a visitor who calls back.
- A signed-URL feature flag — the widget stays public because a sales agent behind a login is a sales agent nobody talks to, but I can flip to token-gated connections in thirty seconds if abuse shows up. Internal-facing agents should start in signed-URL mode and never leave it.
None of these is clever individually. Together they turn "potentially unbounded" into "annoying but capped," which is the bar a public voice agent has to clear.
The Honest Bill
Professional Voice Cloning comes with the ElevenLabs Creator plan at $22/month, and the clone itself is a one-time processing cost — only synthesized conversation costs money after that. Creator includes a block of agent minutes (around 250), then per-minute billing applies. A typical qualification call runs four to six minutes, so figure well under fifty cents per qualified conversation. Cal.com is free at solo volume; the hosting free tier carries the landing page and two proxy routes.
At ten calls a day you're roughly in the $100-150/month zone all-in. A part-time human answering inbound runs twenty to thirty times that and doesn't work Sunday nights. The agent doesn't replace a good salesperson — it catches the leads that would otherwise hit voicemail and evaporate. That's the narrow, boring, correct business case, and the same cost discipline applies here as anywhere else in agent work — I keep my general framework in the agent cost optimization guide.
One more thing the architecture gets right: the same agent config can answer a Twilio phone number. Same persona, same voice, same tools, same calendar — the agent doesn't care whether audio arrives from a browser microphone or a phone line. One brain, multiple surfaces. And when you eventually want the agent querying databases or internal systems, the webhook-proxy pattern evolves naturally toward MCP-style tool servers — the direction I've been building all my agent integrations, per my must-have MCPs breakdown.
The Transferable Skill
The build took an afternoon. The five bug-fix rounds took ninety minutes. Which of those two numbers matters? The second one. Each fix followed the same loop: notice the wrong behavior, describe it precisely, let Claude Code propose the change, sanity-check, redeploy, retest. The builders shipping voice agents that survive production in 2026 aren't the ones with the best voice clone — they're the ones who ran that loop fifteen times before going live. Most people quit at three.
Start narrower than feels ambitious: one use case, one calendar, two tools, bulletproof. Widen only after the narrow version has booked real meetings for a month. And if you'd rather skip the ninety minutes of timezone archaeology and have an agent like this designed, hardened, and shipped for your business, this exact kind of build is on my services page — bugs pre-fixed.
