Blog · · 5 min read
Designing AI-to-Human Handoff for Phone Calls
AI-to-human handoff needs observable triggers, context for staff before they pick up, and a plan for when nobody answers. Conditions, whisper, Twilio Dial and fallbacks.
Key points
- Write handoff conditions as things the agent can observe in the conversation, like "the caller asks to speak to a person". Vague conditions such as "when it gets complicated" produce inconsistent decisions.
- Once the agent decides to hand off, it should transfer right away rather than keep collecting fields. Deliver the reason, whatever fields were captured and the caller's number to staff before they answer.
- Transfer with Twilio's
, and decide in advance where the call goes when staff don't answer (no-answer, busy), such as a promised callback or voicemail.
A good AI-to-human handoff has three parts: specific trigger conditions, context delivered to staff before they pick up, and a plan for when nobody answers. Once the agent decides to hand off, it should stop collecting details and transfer immediately. The goal underneath all of it: the caller should never have to repeat themselves.
Choosing handoff conditions
Write conditions as things the agent can observe in the conversation. "When the issue is complex" or "when needed" will be interpreted differently across calls, even by the same model.
Common conditions by type:
| Type | Example wording |
|---|---|
| Caller's request | The caller asks to speak to a person or a specific staff member |
| Complaint | The caller makes a complaint or expresses strong dissatisfaction |
| Urgency | The caller describes severe pain, an injury, an accident or another urgent situation |
| Out of scope | The caller wants to negotiate prices or cancel a contract |
| Repeated failure | A field still can't be captured after several attempts |
| Identity-sensitive | The caller asks about contract details or payment status |
We strongly recommend always including "the caller asks for a person." It's the escape hatch for anyone who's uncomfortable talking to an AI or in a hurry.
What counts as urgent depends on the business, so write it in that business's terms: "severe pain or swelling" for a dental clinic, "a breakdown while driving" for a repair shop, "a water leak" for property management. Specific language keeps decisions stable. The broader prompt structure is in How to write prompts for an AI phone receptionist.
Hand off immediately
Once the agent decides to hand off, it should transfer right away instead of finishing the questionnaire. Telling someone who just asked for a human, "Sure, but first, can I get your name and number?" makes things worse, and with an urgent call it's actively harmful.
Instead, pass along whatever has been captured so far. Even without a name, knowing the reason and the calling number lets staff open with "Hi, I understand you're calling about your appointment change."
Keep what the agent says brief: "I'll connect you to a member of staff. One moment, please." Telling the caller what's about to happen is enough.
What to tell staff
Before staff pick up, get this in front of them:
- Why the call was handed off (asked for a person, reported severe pain)
- Fields captured so far (reason, name, number: whatever exists)
- The calling number
- A call summary (for the record afterwards)
The reason matters most. Staff open a complaint call very differently from an urgent one.
Two ways to deliver it:
| Method | What it carries | Best when |
|---|---|---|
| Screen or chat notification | Reason, fields and number as text | Staff are at a computer or have a phone app |
| Whisper | A short spoken message after staff answer, before the caller is connected | The front desk works from a desk phone |
In Twilio, a url on <Number> inside <Dial> returns TwiML that runs on the called party's end after they answer but before the parties are connected (TwiML Number). Use it to say "Handoff from the AI: the caller wants to reschedule," and staff start the conversation already knowing why.
The transfer flow
- The agent decides to hand off, tells the caller, and ends the conversation.
- The voice AI's WebSocket closes and Twilio requests the
<Connect>actionURL. - Your
actionhandler fetches the agent's result and, for a handoff, returns<Dial>. - When staff answer, the whisper plays, then the caller is connected.
- If nobody answers, the
<Dial>actiondecides what happens next.
When <Dial> ends, Twilio sends DialCallStatus to its action URL: completed, busy, no-answer, failed or canceled (TwiML Dial). The timeout attribute controls how long to ring. Twilio's default is 30 seconds, and it adds a five-second buffer to whatever you set.
When nobody answers
This is the part of handoff design most often forgotten. If the agent says "Let me connect you" and the caller then hears ringing until the line drops, the experience is worse than if there had been no AI at all.
Options:
- Ring a second person or team in turn.
- Say "Nobody's available right now; we'll call you back," and add the call to a callback queue.
- Take a voicemail.
Outside business hours, it's often better not to transfer at all, and let the agent take the details for a callback.
Doing this with voicast
In voicast, handoff conditions go in the agent settings's handoff_when. When one is met, the agent decides to hand off and ends the conversation; the result has outcome: handoff and the reason in reason. A call.handoff_requested webhook (with reason and fields) also fires, which you can use to notify staff. The transfer itself happens in your app, by returning <Dial>.
const esc = (s: string) =>
s.replace(/[&<>"']/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[c]!);
// <Connect> action: branch on the agent's result
app.post('/twilio/after', async (req, res) => {
const r = await fetch(
`https://api.voicast.jp/v1/calls/by-provider/${encodeURIComponent(req.body.CallSid)}`,
{ headers: { Authorization: `Bearer ${process.env.VOICAST_API_KEY}` } },
);
const call = r.ok ? await r.json() : null;
if (call && call.outcome !== 'handoff' && call.outcome !== 'error') {
return res.type('text/xml').send('<Response><Hangup/></Response>');
}
const whisper = `https://saas.example.com/twilio/whisper?reason=${encodeURIComponent(call?.reason ?? '')}`;
res.type('text/xml').send(`<Response>
<Dial timeout="20" action="https://saas.example.com/twilio/dial-done">
<Number url="${esc(whisper)}">${esc(process.env.STAFF_NUMBER!)}</Number>
</Dial>
</Response>`);
});
// Played only to staff, before the caller is connected
app.post('/twilio/whisper', (req, res) => {
const reason = String(req.query.reason ?? '');
res.type('text/xml').send(`<Response><Say>Handoff from the AI. ${esc(reason)}</Say></Response>`);
});
// Staff didn't answer
app.post('/twilio/dial-done', async (req, res) => {
if (req.body.DialCallStatus === 'completed') {
return res.type('text/xml').send('<Response><Hangup/></Response>');
}
await addToCallbackList(req.body.CallSid, req.body.From); // your code
res.type('text/xml').send(
"<Response><Say>Sorry, nobody is available right now. We'll call you back shortly.</Say><Hangup/></Response>",
);
});The full Twilio setup is in Connecting Twilio calls to a voice AI with Media Streams.
FAQ
Should the agent collect a name and number before handing off?
Usually not. Once the decision is made, transfer right away; making someone who asked for a human (or has an urgent problem) wait makes things worse. Pass along whatever was captured plus the calling number, and staff can pick up the thread.
What if too many calls get handed off?
Look at the breakdown of handoff reasons. Lots of out-of-scope questions means your FAQ needs more answers. Lots of capture failures means revisiting vocabulary hints and how questions are phrased. Lots of "I want a person" can sometimes be reduced by saying briefly in the greeting what the agent can help with.
What happens to calls staff can't answer?
<Dial> posts DialCallStatus to its action URL, so define a path for anything other than completed: ring someone else, promise a callback and queue it, or take a voicemail. Just never leave the call at a dead end.