Blog · · 5 min read

Japanese Speech Recognition on Phone Calls: Names, Numbers and Jargon

Why Japanese speech recognition on phone calls stumbles on names, digits and jargon, and how vocabulary hints, read-back and number normalization fix it.

Key points

  • Japanese speech recognition on phone calls fails most often on three things, names (one reading, many kanji spellings), numbers (similar-sounding digits and multiple readings, like ichi and shichi) and industry terms (heard as everyday words).
  • Accuracy alone won't fix it. Combine vocabulary hints, reading details back for confirmation, and normalizing and length-checking numbers.
  • Capture names by their reading (katakana) rather than guessing kanji, and let a person confirm the kanji later if you need it.

If you're building a voice agent for Japanese callers, three things will cause most of your recognition errors: names, numbers and industry jargon. Names have one pronunciation but many possible kanji spellings. Digits sound alike (ichi for 1, shichi for 7) and most have more than one reading. Domain terms get heard as common words. Better models help only at the margins; the reliable fix is designing the conversation so errors get caught and corrected on the call, using vocabulary hints, read-back confirmation and number normalization.

Why phone audio is hard

Phone audio carries much less information than a close-up smartphone recording. Twilio Media Streams, for example, always delivers 8,000 Hz μ-law audio (WebSocket messages). That narrow band blurs consonant distinctions, which is exactly what separates many Japanese syllables, such as shi and chi, or hi and shi.

Add the usual phone conditions:

  • Street or car noise in the background
  • Weak mobile signal and dropouts
  • Callers holding the phone away from their mouth
  • Lots of fillers (eeto, anoo, the Japanese "um")

So beyond choosing a good model, assume errors will happen and design for recovery.

Names: one sound, many spellings

Japanese names are written in kanji, and the same spoken name can be written many ways. A recognizer hears sounds, so it has to guess which spelling you meant, and the sound alone can't decide.

Reading Possible spellings
Saitō 斉藤・斎藤・齋藤・西東
Watanabe 渡辺・渡部・渡邊・渡邉
Katō 加藤・加東
Yūko 優子・裕子・祐子・夕子

The fix: stop trying to get the kanji right, and capture the reading. For bookings and callbacks, the katakana reading plus a phone number is enough to identify someone. If you truly need kanji (for paperwork, say), have staff confirm it or collect it through a web form.

When confirming a name, read back the reading, not a kanji guess: "Saitō-sama, is that correct?" lets the caller say yes, or correct it ("No, Saidō").

Numbers: similar sounds, multiple readings

Japanese digits trip up recognition in two ways: some sound alike, and most have alternate readings.

Digit Readings Commonly confused with
0 zero, rei, maru maru recognized as a word ("circle")
1 ichi 7 (shichi)
2 ni, nii the particle ni
4 yon, shi the start of shichi (7)
7 nana, shichi 1 (ichi)
9 kyū, ku ku absorbed into a neighboring word

Phone numbers add separators: callers often say no (roughly "of") between groups, as in "zero-san no ichi-ni-san-yon". Dates have irregular readings: the 1st is tsuitachi, the 2nd futsuka, the 20th hatsuka. And April (shigatsu) versus July (shichigatsu) is a classic mix-up.

Three defenses:

  1. Normalize to digits. Convert full-width digits, kanji numerals and separators to plain ASCII digits before doing anything else.
  2. Check the length. Japanese phone numbers start with 0 and have 10 digits (landlines) or 11 (mobiles and 050 IP numbers). If it doesn't fit, ask again.
  3. Read it back in groups. Read the number back in three chunks, such as "090, 1234, 5678", and have the caller confirm.

Here's a normalizer for recognized phone numbers:

python
import re
import unicodedata

KANJI_DIGITS = str.maketrans("〇零一二三四五六七八九", "00123456789")

def normalize_jp_phone(text: str) -> str | None:
    """Extract a Japanese phone number from an ASR result. None means ask again."""
    s = unicodedata.normalize("NFKC", text)   # full-width digits/symbols to ASCII
    s = s.translate(KANJI_DIGITS)             # kanji numerals to digits
    digits = re.sub(r"\D", "", s)             # drop "の", hyphens, spaces
    if digits.startswith("0") and len(digits) in (10, 11):
        return digits
    return None

assert normalize_jp_phone("〇九〇の一二三四の五六七八") == "09012345678"
assert normalize_jp_phone("03-1234-567") is None  # 9 digits: ask again

For dates, callers often say "tomorrow" or "next Tuesday". Put today's date and weekday in the agent's instructions so it can resolve these to a concrete date, then read the date back. More in Read-back confirmation.

Jargon: heard as everyday words

Industry terms show up rarely in the text recognizers learn from, so they get swapped for common words that sound similar. Dental terms like 根管治療 (konkan chiryō, root canal treatment), real estate's 内見 (naiken, property viewing) or auto repair's 車検 (shaken, the mandatory vehicle inspection) can all arrive as something else.

The fix is vocabulary hints. Most speech recognition APIs let you boost specific words or phrases. More isn't better: limit hints to terms callers actually say in that business, plus proper nouns like the business name, product names and local place names. Choosing hints is covered in Vocabulary hints for better speech recognition.

Designing for correction

You won't get recognition errors to zero. What matters is catching them and fixing them during the conversation:

  • Always read back names, phone numbers and dates.
  • When something doesn't make sense, ask again rather than letting the model fill in the gaps.
  • Validate the shape of numbers (length, range) and re-ask when they don't fit.
  • If repeated attempts fail, hand off to a person or offer a callback.

Cap the number of retries. Being asked the same question four times is worse for the caller than being passed to a human.

Doing this with voicast

In a voicast agent settings, domain (the industry) and vocabulary (terms) are passed to speech recognition as hints. Set a field's type to phone and voicast reduces the captured value to digits, checks that it's a Japanese number of 10 or 11 digits starting with 0, and asks again if it isn't. The agent reads back what it heard and corrects it when the caller says it's wrong.

json
{
  "domain": "歯科医院",
  "vocabulary": ["根管治療", "インプラント", "ホワイトニング", "定期検診", "さくら歯科"],
  "fields": [
    { "key": "name", "label": "お名前(カタカナの読み)", "required": true },
    { "key": "phone", "label": "折り返しの電話番号", "required": true, "type": "phone" }
  ]
}

FAQ

Will a better speech model get the kanji in names right?

No. Multiple spellings for one reading isn't an acoustic problem, so no model can resolve it from sound alone. Capture the reading, and confirm kanji separately if you need it.

How do I catch a misheard phone number?

Normalize to digits and check the length; that alone catches many dropped or extra digits. Japanese numbers start with 0 and have 10 or 11 digits. Then read the number back in three groups and have the caller confirm.

What belongs in vocabulary hints?

Terms that callers actually use in that business, plus proper nouns: the business name, products, nearby place names. Overloading the list can pull unrelated words toward your terms, so keep it focused.

Related posts