We built a production AI calling agent that speaks Gujarati. It places outbound calls for a retail bag store, presents the product catalogue in conversation, books appointments and sends an SMS confirmation afterwards — with every call and customer logged. The stack is Twilio for telephony, Deepgram for speech-to-text and GPT-4o-mini for the conversation. This is how it fits together and what was hard.
The problem
The store is run by a small retail team. They had a list of customers worth calling — people who had bought before, people who had asked about new stock — and no time to call them. Showcasing products over the phone and booking a visit takes a few minutes per customer, and there were far more customers than minutes. Outreach happened in bursts when someone was free, and sales depended on whether that burst happened.
The brief was specific: call customers, talk about the catalogue the way a staff member would, book an appointment if the customer wants one, and confirm it by SMS. In Gujarati, because that is what the customers speak.
The architecture
The agent is a pipeline of four stages running inside a single phone call, plus a confirmation and logging step afterwards.
- Twilio places the call. The outbound call is initiated through Twilio's voice API and the audio is streamed to our backend in real time. Twilio also handles the telephony edge cases — no answer, busy, voicemail — so the backend only deals with live conversations.
- Deepgram turns speech into text. The caller's audio is streamed to Deepgram for speech-to-text as they speak, rather than waiting for them to finish, so the agent can respond without long silences.
- GPT-4o-mini holds the conversation. The transcript, the catalogue information and the conversation so far go to GPT-4o-mini, which decides what to say next: present a product, answer a question, offer appointment slots, or wrap up politely. The prompt carries the store's catalogue and the rules of what the agent may and may not promise.
- Text-to-speech plays the reply. The model's reply is converted to speech and played back into the call. Then the loop repeats until the conversation ends.
- SMS confirmation and logging. If an appointment is booked, Twilio sends an SMS confirmation to the customer. Every call — outcome, transcript, booked slot — is written to persistent storage against the customer's record, so the team can see who was called, what was said and what happened next.
The choice of GPT-4o-mini over a larger model was deliberate: on a phone call, the time between the customer finishing a sentence and the agent starting to reply matters more than the extra reasoning a larger model brings, and the conversations here are bounded — a catalogue, a set of questions, a booking.
What was hard about Gujarati
Most voice-agent demos you see online are in English. Gujarati brought problems that English does not.
Mixed-language speech. Gujarati speakers routinely mix in English words — product names, numbers, days of the week, "appointment", "confirm". The speech-to-text layer has to handle sentences that switch language mid-way without dropping the English fragments or forcing them into Gujarati phonetics. The prompt also has to expect this and not treat an English word as a change of language.
Names, numbers and times. Dates, prices and phone numbers are where transcription errors are expensive: a mis-heard time becomes a mis-booked appointment. The agent confirms these details back to the customer before booking, which costs a few seconds per call and removes a whole category of mistakes.
Register and politeness. A store calling a customer is a specific kind of conversation — warm, respectful, not pushy. The forms of address and the way you decline or close a call in Gujarati are part of the prompt, not an afterthought. Early versions that were translated from an English script sounded wrong to native speakers, and the fix was writing the Gujarati conversation design from scratch rather than translating.
Knowing when to stop. A customer who says they are busy, or not interested, gets a polite close and no second pitch. That rule is hard-coded, not left to the model's judgement.
What it does today
In production, the agent:
- Places outbound calls to the store's customer list and converses naturally in Gujarati.
- Presents the product catalogue in conversation and answers standard questions about it.
- Offers and books appointments, confirming the details back before booking.
- Sends a Twilio SMS confirmation after a booking.
- Logs every call and customer interaction persistently so the team can follow up.
Calls that need a person — a complaint, a question outside the catalogue — are handed over rather than improvised. The team's time goes to those conversations and to the customers who show up for appointments, instead of to the calls that only needed a catalogue and a calendar.
FAQ
Can the same agent handle Hindi and English?
Yes. The pipeline is the same; the language is a matter of speech-recognition settings and conversation design. For a business whose customers switch between Gujarati, Hindi and English, the agent follows the customer.
What happens if the customer asks something the agent doesn't know?
It says so and offers to have a person call back, and that request is logged for the team. It does not guess about stock, prices or policies — the prompt restricts it to the catalogue and rules it has been given.
How is the customer's data handled?
Call logs and customer records are stored in the client's own system. The client owns the code, the server and the data, as with everything we build.
Could this run inbound as well as outbound?
Yes. The same pipeline answers inbound calls — for example as an AI receptionist that books appointments when staff are busy or the shop is closed. The outbound version was what this store needed first.
Free 30-minute call. Written plan within 7 days. No obligation.