We built a production AI calling agent that speaks Gujarati. It makes outbound calls for a retail business, talks about the product range, books appointments and sends an SMS confirmation afterwards. It runs on Twilio for telephony, Deepgram for speech, and GPT-4o-mini for the conversation.

This article is what we learned building it, including the parts that didn't work.

Most writing on this subject is by companies selling a platform, so it tends to describe a version of the technology that always works. The reality is more useful: AI calling agents are genuinely excellent at a narrow set of jobs and genuinely bad at others, and knowing which is which is the difference between a system that pays for itself and an expensive novelty that annoys your customers.

What an AI calling agent actually is

Four components, chained together, running fast enough that the person on the other end doesn't notice the joins.

   Phone network  (Twilio / Exotel / Plivo)
        ↓  audio in
   Speech-to-text  (Deepgram / Whisper / Google STT)
        ↓  text
   Language model  (GPT-4o-mini / Claude / Gemini)
        ↓  reply text  +  actions
   Text-to-speech  (ElevenLabs / Deepgram / Azure)
        ↓  audio out
   Phone network
        
   Alongside all of it:
   → your CRM     (log the call, update the lead)
   → your calendar (book the appointment)
   → WhatsApp/SMS  (send the confirmation)
   → a human       (transfer when it should)

The engineering difficulty is not in any one box. It's in latency. Every component adds delay, and human conversation is intolerant of it. Above roughly 1.5 seconds of silence, people start talking over the agent, repeating themselves, or assuming the line has dropped. Most of the work in building a good calling agent is shaving milliseconds and handling interruptions gracefully.

The second difficulty is interruption handling — what happens when the caller starts talking before the agent has finished. A naive implementation talks over them. A good one stops, listens, and picks up the thread.

What they're genuinely good at

Be specific about this, because the honest list is shorter than the marketing list — and everything on it is valuable.

Calling back every single lead, immediately. This is the strongest use case in India by a distance. A lead comes in at 9:40pm from a listing portal. A human calls at 11am the next day, by which time the person has spoken to three competitors. An agent calls at 9:41pm, confirms interest, captures requirements and books a callback. The agent isn't better than your salesperson. It's better than the call that never happened.

Answering the same eight questions, in the caller's language. Timings, location, price range, availability, documents required, what to bring. An agent handles these identically at 2am and at 2pm.

Appointment booking and confirmation. Check the calendar, offer slots, book, send the WhatsApp or SMS confirmation. This is a bounded task with a clear success condition, which is exactly what these systems handle well.

Reminder and follow-up calls at volume. Appointment reminders, payment reminders, renewal reminders, feedback calls. Repetitive, scripted, low-stakes, and nobody enjoys making them.

Qualifying before a human gets involved. Ten minutes of a salesperson's time spent discovering someone's budget is 10% below the minimum is time wasted. An agent can establish that in ninety seconds and route accordingly.

Working in Indian languages. This matters more than people outside India assume. A customer who is uncomfortable in English will simply not engage with an English-only agent. Hindi and regional-language capability isn't a nice-to-have here — for many businesses it's the entire difference between a system that works and one that doesn't.

Where they still fail

Angry customers. An agent handling a genuine complaint makes it worse, every time. Detect frustration and transfer immediately — and build that path before you launch, not after the first bad call.

Anything requiring judgement or negotiation. Discounts, exceptions, "can you do anything about the price." Don't let the agent near these.

Bad phone lines. Speech recognition degrades sharply on poor connections. This is a real constraint on Indian mobile networks and it's the failure mode you'll hear most.

Heavy accents and code-switching. Indian callers routinely switch between English, Hindi and a regional language within a single sentence. Recognition quality varies significantly depending on the mix, the speaker and the model. This is improving quickly but it is not solved.

Long, meandering conversations. Agents hold a thread well for a few turns. Over a long, digressive call they lose context and start repeating themselves.

Anything where being wrong is expensive. Medical advice, legal or financial specifics, binding commitments. Not because the model can't produce an answer — because it will produce a confident one that might be wrong.

The rule we work to: an AI calling agent should handle the calls your team doesn't have time to make, and hand over the calls your team needs to handle. It's a capacity tool, not a replacement.

What it costs

Three cost layers, and quotes usually conflate them.

Build. Roughly ₹75,000 – ₹3,00,000 depending on how many conversation paths it handles, how many languages, and what it has to connect to. A single-purpose outbound agent that books appointments sits at the low end. An inbound agent handling twelve intents across three languages with CRM writes and calendar integration sits at the high end.

Per-minute running cost. This is the one to interrogate. Every minute of conversation costs you telephony + speech-to-text + language model + text-to-speech. One Indian provider advertises pricing around ₹0.40 per minute in their published material, but that figure varies substantially with the voice quality, the model, and how long calls actually run.

Work out your own number before you commit:

Cost per month  =  calls/month × avg call minutes × cost per minute

500 calls a month averaging 3 minutes at ₹2/minute is ₹3,000/month. The same volume at ₹8/minute is ₹12,000/month. Ask any vendor for the all-in per-minute rate including every component, and ask what happens if a call runs to ten minutes.

Maintenance. Conversation flows need tuning based on what real callers actually say — which is never what you expected. Budget for a few hours a month, especially in the first quarter.

What we'd tell you before you build one

Start with outbound, not inbound. Outbound is a bounded problem: you control who's called, when, and why, and a failed call costs you almost nothing. Inbound means handling whatever anyone says at any hour, and a failed inbound call is a lost customer. Prove the technology on outbound first.

Pick one job. "An agent that handles all our calls" fails. "An agent that calls every new website enquiry within two minutes, confirms interest and books a callback" succeeds. Narrow scope, clear success condition.

Design the transfer path first. Before you design a single conversation, decide exactly when the agent gives up and hands to a human, and make sure the human receives the full context. This is the difference between a good experience and a terrible one.

Say it's an AI. Not just for the ethics, which matter — for the practical reason that callers who know they're talking to a system speak more clearly, keep to the point, and are far more forgiving when it doesn't understand. Pretending otherwise gets you worse recognition and a worse reputation.

Listen to the first hundred calls. All of them. You will be surprised by what people actually say, and every one of those surprises is a fix.

What didn't work for us

Two things worth passing on.

We initially let the agent handle too many intents. The first version tried to cover the full product range, pricing questions, availability, store timings and appointment booking. Conversations wandered, the agent lost the thread, and the booking rate was poor. Cutting it to a narrow script with one clear goal — and transferring anything outside it — improved outcomes substantially. Narrower is better. Almost always.

Our first latency budget was too generous. We tuned for correctness and ended up with pauses long enough that callers assumed the line had dropped and started talking over the agent, which broke recognition, which made the pauses worse. Optimising the speech-to-text and streaming the model's response — rather than waiting for a complete reply before speaking — mattered more than any prompt change we made.

Key takeaways

  • An AI calling agent is four systems chained together. The engineering difficulty is latency and interruption handling, not intelligence.
  • The strongest use case in India is calling every lead back immediately. The agent isn't better than your salesperson — it's better than the call that never happened.
  • Indian-language capability is not optional. Hindi and regional languages are the difference between engagement and silence for a large share of callers.
  • They fail on angry customers, negotiation, poor lines, and anything where a confident wrong answer is expensive. Build the human-transfer path before you launch.
  • Costs sit in three layers: build (₹75k–₹3L), per-minute running cost (the one to interrogate), and ongoing tuning. Calculate your own monthly figure before signing.
  • Start outbound, pick one job, tell callers it's an AI, and listen to the first hundred calls.

Want to hear what an AI calling agent sounds like in your language?

We've built and deployed one in production — outbound calls in Gujarati, product conversation, appointment booking, SMS confirmation. We'll show you the real thing, in your language, handling a call from your business, and tell you honestly whether it's a fit for what you're trying to do.

Book a live demo call → · WhatsApp us

Related: AI calling & voice agent services · What automation costs in India · AI receptionist for clinics