Blog · · 4 min read
Connecting Twilio Calls to a Voice AI with Media Streams
Stream Twilio call audio to a voice AI with <Connect><Stream>, pass context with <Parameter>, and route the call afterwards with the Connect action. TwiML and failover.
Key points
- To connect a Twilio call to a voice AI, return TwiML with
, which opens a bidirectional WebSocket. is one-way and can't play the agent's voice back into the call. - The Stream url doesn't accept query strings, so pass tenant IDs and keys as
elements. They arrive in the WebSocket start message. - Add an action URL to
. When the WebSocket closes, Twilio requests it, and you decide what happens next (transfer, hang up) based on the agent's result.
To put a voice AI on a Twilio call, respond to the incoming-call webhook with TwiML containing <Connect><Stream>. Twilio opens a bidirectional WebSocket to the agent and streams call audio both ways. When the conversation ends and the WebSocket closes, Twilio requests the action URL on <Connect>, where you look up the agent's result and decide whether to transfer the caller, hang up or fall back to your normal flow. Your numbers stay on Twilio; only the conversation goes to the AI.
Connect vs. Start
For a conversational agent, use <Connect><Stream>. Twilio offers two ways to stream (TwiML Stream):
<Start><Stream> |
<Connect><Stream> |
|
|---|---|---|
| Direction | Call → WebSocket only | Bidirectional (you can send audio into the call) |
| Next TwiML verb | Runs immediately | Waits until the WebSocket closes |
| Use for | Recording, transcription, analytics | Conversational voice AI |
<Start><Stream> lets you listen, but not talk back. <Connect><Stream> lets the agent's audio flow back over the same socket.
The TwiML
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect action="https://saas.example.com/twilio/after">
<Stream url="wss://voice.example.com/ws">
<Parameter name="tenant" value="ten_123" />
<Parameter name="from" value="+15555550123" />
</Stream>
</Connect>
</Response>Three things to know:
urlmust bewss, and it doesn't support query strings (TwiML Stream). Anything the agent needs to know (which customer this is, which configuration to use) goes in<Parameter>elements.- Parameter values arrive in the
startmessage on the WebSocket. - With an
actionURL, Twilio makes a request to it when<Connect>ends. Without one, Twilio moves on to the next verb, and if there isn't one, the conversation ends (TwiML Connect).
What flows over the WebSocket
Your media server exchanges these messages with Twilio (WebSocket messages):
| Direction | Message | Contents |
|---|---|---|
| Twilio → you | connected |
Socket established |
| Twilio → you | start |
Stream metadata and your <Parameter> values |
| Twilio → you | media |
Caller audio (base64) |
| Twilio → you | dtmf |
Keypad presses |
| Twilio → you | mark |
Audio you sent has finished playing |
| Twilio → you | stop |
Stream ended |
| You → Twilio | media |
Audio to play into the call |
| You → Twilio | mark |
A label to be echoed back when playback reaches it |
| You → Twilio | clear |
Discard buffered audio (for barge-in) |
Audio is always 8,000 Hz μ-law, so you'll likely need to convert it for your STT and TTS providers. To stop the agent mid-sentence when the caller starts talking, send clear.
Routing the call with action
When the conversation is over, the agent closes the WebSocket. Twilio then requests your action URL with the standard voice parameters, including CallSid. The TwiML you return decides what happens next:
- Agent collected the details and said goodbye →
<Hangup/> - Agent decided to hand off →
<Dial>a staff number - Agent unreachable or errored → your usual flow, such as ringing staff
The pattern: in the action handler, fetch the agent's result for this CallSid and branch on it.
Failing safe
Design so that calls still reach a human when the AI doesn't:
- Have the agent's WebSocket server close immediately on a bad key or an internal problem. Twilio ends
<Connect>and goes toaction. - In the
actionhandler, treat "no result" as a failure and route to staff. - Set a Fallback URL on the Twilio number so that if your own webhook fails, Twilio still has somewhere to go.
With that in place, an outage costs you automation, not calls. More in Fallbacks: keeping calls flowing when the voice AI fails.
Connecting to voicast
voicast's voice endpoint is wss://voice.voicast.jp/ws. Pass the tenant ID (the business taking the call), its connection key (vts_…) and the caller's number as parameters. After the conversation, call GET /v1/calls/by-provider/{CallSid} from your action handler and branch on outcome. An Express example:
import express from 'express';
const app = express();
app.use(express.urlencoded({ extended: false }));
const esc = (s: string) =>
s.replace(/[&<>"']/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[c]!);
// Incoming call: find the tenant for this number and stream to voicast
app.post('/twilio/voice', async (req, res) => {
const t = await findTenantByNumber(req.body.To); // your DB: { voicastTenantId, voicastSecret, staffNumber }
res.type('text/xml').send(`<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect action="https://saas.example.com/twilio/after">
<Stream url="wss://voice.voicast.jp/ws">
<Parameter name="tenant" value="${esc(t.voicastTenantId)}" />
<Parameter name="secret" value="${esc(t.voicastSecret)}" />
<Parameter name="from" value="${esc(req.body.From ?? '')}" />
</Stream>
</Connect>
</Response>`);
});
// After the conversation: route based on the result
app.post('/twilio/after', async (req, res) => {
const t = await findTenantByNumber(req.body.To);
const r = await fetch(
`https://api.voicast.jp/v1/calls/by-provider/${encodeURIComponent(req.body.CallSid)}`,
{ headers: { Authorization: `Bearer ${process.env.VOICAST_API_KEY}` } },
);
const call = r.ok ? await r.json() : null;
let twiml: string;
if (call?.outcome === 'completed' || call?.outcome === 'answered' || call?.outcome === 'hangup') {
twiml = '<Hangup/>';
} else {
// handoff, error, or no result: ring staff
twiml = `<Dial>${esc(t.staffNumber)}</Dial>`;
}
res.type('text/xml').send(`<?xml version="1.0" encoding="UTF-8"?><Response>${twiml}</Response>`);
});
app.listen(3000);Keep connection keys on your server and inject them only when rendering TwiML. Validate X-Twilio-Signature on incoming Twilio webhooks (Webhooks security). voicast doesn't own numbers or lines, so number configuration and the transfer flow stay in your app. See how it works.
FAQ
Can I put the tenant ID in the Stream URL?
No. Twilio's Stream url doesn't support query strings. Pass values like tenant IDs and keys as <Parameter> elements and read them from the WebSocket start message.
What happens if I leave out action?
When <Connect> ends, Twilio continues with the next verb in the document, or ends the call if there isn't one. If you want to branch on the agent's result, use action and decide in the handler.
What happens to calls if the voice AI is down?
If the agent's WebSocket closes right away, Twilio proceeds to your action URL. Route "no result" to staff there, and callers still reach a person.