A large part of your operations probably still runs over the phone. Customers call your hotline. Your team calls vendors, partners and drivers to confirm times, check availability and chase status updates. None of that ends up in your systems unless someone types it in.
That is changing. AI voice agents for business can now hold a natural phone conversation, pull the relevant information out of it as structured data and trigger the same automations you already use for email and text. This works for both inbound and outbound calls.
You also do not have to rent this from a big provider. With open-source models, you can run a voice agent on your own infrastructure, at low cost, without depending on anyone else’s platform. Here is how it works, where it pays off and how to decide whether it is the right first step for you.
In this video, Hassan Ghiassi and David Grimm, our co-founder and CTO, discuss how open-source voice agents can take over phone-heavy workflows. The article below covers the key points.
Can AI voice agents for business really handle phone calls?
Yes. Large language models no longer work only with text. They can process audio and video, and they can answer in audio. This is called speech to speech: the caller speaks, the model understands and replies with a spoken answer, without anyone typing in between.
The big providers already offer this, for example Google Voice AI and GPT Realtime. What caught our attention is a recent demo from Hugging Face that showed a complete speech-to-speech pipeline built only from open-source components, including open-source language models. David set it up and ran it locally himself.
The quality is good enough that callers may not notice the difference.
“You can even run with your own models in your own environment and AI where it might be fairly hard for a customer even to recognize that they are talking with an artificial intelligence.” David Grimm, CTO, Vylos
That is exactly why David recommends telling callers up front that they are talking to an AI. It is a matter of trust. The EU AI Act will also make it mandatory to mark AI content as such, so plan for a clear disclosure from the start.
Why does the phone still matter for your operations?
Many companies still depend on the phone in two directions.
- Outbound: your team calls third parties such as vendors, partners and drivers to coordinate work.
- Inbound: customers call your hotline with problems, requests and questions.
On the inbound side, the standard answer so far has been the IVR menu. “If you have an issue with your device, press one. To change your address, press two.” Pick the wrong option and you start over. David describes the experience plainly:
“And so, it took half an hour before you were through the menu.” David Grimm, CTO, Vylos
What can replace the IVR phone menu?
A voice agent that simply talks with the caller. A customer calls about a broken product and wants a refund. The agent asks for the order number and handles the rest as a normal conversation, not a menu tree. No buttons, no restarts after a wrong choice.
How does a phone call turn into structured data and backend actions?
This is the part that matters most for operations leaders. A voice agent is not a separate technology that needs its own automation strategy. The same techniques you use for text apply.
“It doesn’t matter the technology if it outputs audio or if it output text the underlying technology is the very same.” David Grimm, CTO, Vylos
In practice, the conversation is turned into structured data: who is calling, what they want, which product or order it is about. That data can then be used to look up records in your back end and trigger API calls.
Take a simple example. A customer calls and asks to activate a product, and gives their name. The language model, or the automations behind it, looks up the customer in the back end and may resolve the request directly through an API call.
Can the agent fix the issue during the call instead of opening a ticket?
That is the real shift. In the classic setup, first-level support takes the call, opens a ticket and hands it to IT, which fixes it later. The customer waits, and your team touches the same issue several times.
If you give the language model the capabilities to act, for example access to the right systems through APIs, it can fix the issue while the customer is still on the line. David describes the result from the customer’s side: “my issue is understood and solved and fixed within a minute or two of conversation without even having to write an email.” That is a very positive customer experience.
The value here comes from the connection to your systems, not from the voice alone. If you want to see how we build these connections, our AI agent and automation implementation services cover data extraction, system sync and rule-based actions.
What about outbound calls?
Voice agents do not only answer calls. They can make them.
Much outbound calling is coordination: confirming times, checking availability, asking whether something is still possible. These are short, structured conversations that follow a clear goal. An AI voice agent can make these calls, capture the answers as structured data, update your ERP system and trigger follow-up automations, such as informing your client about the result.
How do you get data from partners who are not digitally connected?
Not every business you work with has an API. Many small vendors, subcontractors and field workers have one interface: a phone. Until now, that meant a person had to call them and type the answers into a system.
David puts it in a memorable way:
“Your voice or the humans become the API kind of.” David Grimm, CTO, Vylos
The voice conversation becomes the integration layer. The AI calls, asks the right questions and produces structured output your automations can use. According to David, this opens automation and efficiency possibilities that simply were not possible before this technology.
Hassan Ghiassi frames it for non-technical decision makers:
“It doesn’t have to come from a database. It actually comes from the person talking that gets pulled into this AI.” Hassan Ghiassi, Vylos
What does this look like in logistics?
Logistics is a good example of a low-tech setting. In theory, every truck would report its position and status automatically. In reality, as David points out, “a lot of logistics companies have trucks which are 20 plus years old” and are not connected at all.
What those drivers do have is a mobile phone. An AI voice agent can call them about delays, capture status updates and feed them into your systems automatically.
The same applies to vendors without modern connectivity. Wherever there is only a mobile phone, a landline or a voice messaging system, this technology can collect information in real time and process it in real time.
Can you run an open-source AI voice agent on your own infrastructure?
Yes, and it is less demanding than most people expect. The core task is holding a conversation and extracting the relevant information. For a language model, that is a basic job.
“It’s having a conversation and extract the topics and the information which is relevant and that’s kind of the 101 for large language models.” David Grimm, CTO, Vylos
Because the task is simple, you do not need a complete data center. The open-source demo runs on local models, so you pay for infrastructure only. There are some technical challenges, mainly around audio decoding, but existing software solutions can be stitched together to run on a modest server. In David’s words, you won’t pay thousands of dollars or euros a month for a server.
Open-source models are models whose weights are publicly available, so you can download them and run them on hardware you control instead of calling a provider’s service. For voice agents, David sees three advantages:
- Low cost: you pay for your infrastructure, not per call to a provider.
- High quality: the open-source pipelines are good enough for natural conversations.
- Independence: you are not dependent on the big providers, and the models sit inside your own system.
Is voice automation the right first step for your business?
Not for everyone. It depends on your industry and your audience.
If you serve a young B2C audience, they are more likely to reach you on WhatsApp or Instagram than by phone. Messaging automation may be the better place to start.
If your customers include elderly people, or if your operations depend on calls with drivers, vendors and partners, the phone is still central. Those phone-heavy industries should look into voice agents now. If you are unsure where your biggest lever is, our AI strategy consulting can help you compare options before you build anything.
What should you do next?
- Map your phone traffic. Where do inbound calls end up as tickets? Where does your team spend time on outbound coordination calls?
- Check your audience. Do your customers and partners actually use the phone, or do they prefer messaging?
- Connect the agent to your systems. Plan the back-end automations and API calls so requests are resolved during the call, not ticketed.
- Start with information gathering. Calls to drivers, vendors or partners who only have a phone are a clear, contained use case.
- Tell callers they are talking to an AI. Build disclosure into the conversation from day one.
- Consider open source. Running the agent on your own infrastructure keeps costs low and keeps you independent.
Hassan sums it up: phone-related work that used to be slow and manual is now possible much faster with AI and automations, at reasonable cost, and the models can sit inside your own system. “So you should definitely look into it.”
FAQ
Can AI answer and make business phone calls naturally?
Yes. Large language models can now listen and respond in audio, which is known as speech to speech. Big providers offer this, and a recent Hugging Face demo showed a fully open-source pipeline. The conversations can be natural enough that callers may find it hard to tell they are talking to an AI, which is why you should tell them up front. Agents can handle both inbound hotline calls and outbound coordination calls.
How does a voice conversation turn into structured data and backend actions?
The same techniques used for text automation apply to voice. The agent extracts the relevant details from the conversation, such as the caller’s name and request, as structured data. That data is used to look up records in your back end and trigger API calls. For example, a customer asking for product activation can have the request resolved during the call instead of through a ticket.
Can I run an AI voice agent on my own infrastructure with open-source models?
Yes. A recent Hugging Face demo showed a speech-to-speech pipeline built entirely from open-source components, which our CTO David Grimm ran locally. Holding a conversation and extracting information is a basic task for language models. There are some challenges with audio decoding, but existing software can be combined to run on a modest server, keeping the models inside your own system.
What does it cost to run an AI voice agent?
With open-source models on your own infrastructure, you pay for the infrastructure only. You do not need a complete data center, and according to David Grimm you won’t pay thousands of dollars or euros a month for a server. The advantage over big providers is low cost with high quality, and no dependence on those providers.
How can low-tech businesses like logistics use AI voice agents?
Many logistics companies run trucks that are 20 or more years old and not digitally connected, but drivers have mobile phones. An AI voice agent can call drivers about delays and capture status updates automatically. The same works for vendors without modern connectivity. The person on the phone becomes the data source, and the conversation produces structured output for your automations.
Find out where voice agents fit in your operations
If your teams spend hours on the phone collecting information or passing tickets around, there is likely a clear automation opportunity. Book a free AI automation audit and we will look at your phone-heavy workflows together and show you where an AI voice agent would make the biggest difference.