Switching language mid-conversation

VoiceThere voice sessions stay on one WebRTC connection while STT and TTS follow the caller's language. The runner swaps the listening and speaking models without tearing down the peer connection, and your agent keeps receiving transcripts as before.

Requirements

Automatic switching needs VoiceThere for both speech-to-text and text-to-speech (API id voicethere). Only the VoiceThere catalog maps a language to a speech model and a voice, so the runner can start the right pair on its own. With a cloud vendor on either side the dashboard keeps languageId.autoSwitch.enabled disabled, the API answers 400 with the code auto_switch_requires_voicethere_speech, and a value that was saved earlier is ignored at runtime (the runner logs language_switch.auto_unavailable).

Cloud vendors still work with switches your agent starts, through per-language profiles.

How a switch works

The caller usually keeps talking when the language is detected, so the runner stays quiet by default. It holds the caller's speech, starts the target speech services, and speaks only after the caller has finished.

Automatic and manual switches

There are two ways a switch starts. In both, the runner swaps the STT and TTS models without dropping the WebRTC connection. Only the automatic switch sends voice_language_* events to the client.

Agent API

import { defineAgent, setVoiceLanguage } from "@voicethere/agent";

export default defineAgent({
  onVoiceLanguageChanged({ sessionId, language }) {
    // Fired after an automatic switch (the runner's voice_language_changed event).
  },
  async onUserSpeechFinal({ sessionId }) {
    // Imperative switch (WebRTC stays up):
    await setVoiceLanguage(sessionId, { language: "de" });
  },
});

The voice-starter template skips calling setVoiceLanguage on every user_language event when deploy env has SHERPA_LID_AUTO_SWITCH enabled, because the runner owns STT and TTS in that mode.

Clients see the progress of an automatic switch as events on speech_event, listed under Client events. A manual switch with setVoiceLanguage sends none of the voice_language_* events; your agent knows when it switched and can tell the client itself.

Request shape

setVoiceLanguage and prepareVoiceLanguage in @voicethere/agent 0.10.0 take the same object, and it looks the same for every provider:

{
  language?: string,
  scope?: "stt" | "tts" | "both",
  sttVendor?: { provider, model?, language? },
  ttsVendor?: { provider, model?, voice?, language? },
}

STT providers: voicethere, openai, deepgram, assemblyai, google, elevenlabs, fishaudio, groq, azure, aws. TTS providers: voicethere, openai, elevenlabs, cartesia, fishaudio, google, deepgram, groq, azure, aws. The built-in speech is voicethere. The older ids local-sherpa and cluster-sherpa are deprecated aliases of it, accepted wherever a provider is accepted: the agent SDK, the API, the CLI and the dashboard.

What you can leave out

An omitted model, voice or language means that provider's default for vendor.language ?? language ?? the session's current language. For voicethere, model on the STT side is a catalog id (en, en-small), and voice or model on the TTS side is a catalog id (en-lessac). ttsVendor.language is accepted only for voicethere. A both switch to a language that has no VoiceThere STT model (Italian, Portuguese, Dutch, Polish, Hindi) changes TTS only.

Examples

VoiceThere on both sides, German:

import { setVoiceLanguage } from "@voicethere/agent";

await setVoiceLanguage(sessionId, {
  language: "de",
  sttVendor: { provider: "voicethere" },
  ttsVendor: { provider: "voicethere" },
});

Deepgram nova-3 for German speech-to-text, leaving TTS as it is:

await setVoiceLanguage(sessionId, {
  scope: "stt",
  sttVendor: { provider: "deepgram", model: "nova-3", language: "de" },
});

OpenAI tts-1 with the alloy voice, leaving STT as it is:

await setVoiceLanguage(sessionId, {
  scope: "tts",
  ttsVendor: { provider: "openai", model: "tts-1", voice: "alloy" },
});

VoiceThere speech-to-text with ElevenLabs text-to-speech:

await setVoiceLanguage(sessionId, {
  language: "de",
  sttVendor: { provider: "voicethere" },
  ttsVendor: { provider: "elevenlabs" },
});

prepareVoiceLanguage takes the same object and only starts the models, so the later setVoiceLanguage call is fast.

Failures and messages

A switch that cannot happen resolves { ok: false, reason, message }. The message says what to change. Where a message has two forms, the one for the side you passed is returned.

The agent log records a failure as Voice settings change failed: <reason>. <message> on the voice.settings_change_failed line, with message as a field. See System log events.

Deprecated fields

The top-level voice and stt strings still work when you pass them alone. Pass one together with the matching vendor object and the switch fails with conflicting_fields. New code should use ttsVendor.voice and sttVendor.model.

Per-language models

Only the STT and TTS models you picked for the project start warm. When a language switch needs another language, its models start on demand (a cold start). Each language can use its own models, and the table decides which ones.

In the dashboard, open Advanced voice pipeline and look under Language switch & presets. The Per-language models table has one row per language with an STT model and a TTS model, picked from the Sherpa catalog. Unknown model ids are rejected when you save. The table is stored in voice.profilesByLanguage, so the CLI and API take the same JSON.

A language without a row uses the catalog default for that language. voice.allowedLanguages limits which languages a switch may target. Leave it empty to allow the boot language plus every language in the table, or use * to allow any code for agent-driven switches.

A cloud vendor works as well: sttVendor, ttsVendor and voice can be set in the JSON (see STT & TTS vendors). The dashboard table shows those rows read-only.

voice.expectedLanguages is gone. It used to keep extra language pools warm for the whole life of a runner. Stored values are deleted and ignored.

What happens during a switch

The runner checks whether the target language is already running. If it is, the swap is instant and the caller notices nothing. If it is not, the runner starts the models, holds the caller's utterance, and speaks a holding message in the voice the call is using. The message names the target, for example “One moment, switching to German.” When it plays follows waitMessage.mode: with end_of_utterance the runner waits until the caller stops, immediate speaks at once, and off keeps it silent. Texts you set for a project can use the {language} placeholder. A healthy cold start takes roughly 4 to 6 seconds, and an occasional slow start can take longer. When the models are ready, the runner swaps, replays the held speech into the new STT, and the conversation goes on in the new language.

Each attempt waits 20 seconds. If the models are not running by then, the holding message plays again once the first one has finished, and the runner asks for the models once more and waits up to another 20 seconds. The second request is logged as language_switch.retry and sends no extra event to your agent. After about 40 seconds in total with no models, the runner sends voice_language_switch_failed with reason: "language_unavailable"and the call stays in the previous language. That event goes out before the held speech is given back to the previous language's STT, so your agent already knows that the next transcript comes from the old language model. When the wait message is on, the caller also hears a short sentence in the old voice, for example “Sorry, that language isn't available right now. Let's continue in English.” Built-in sentences exist for English, German, Spanish, French, Italian, Portuguese, Dutch and Polish. For any other language nothing is spoken, and you cannot change the text yet.

Held speech when a switch fails

Two limits to know about:

• Speech during the wait is only kept when the decision comes before the final. If language ID decides only after the final has arrived (for example with end_of_utterance timing), the runner replays that utterance but cannot buffer what the caller says during the wait. The language_switch.stt_hold log line then shows holdOutcome: "not_held" with reason: "final_already_seen".
• immediate plays over the caller. An immediate wait message starts while the caller may still be talking, and their speech does not cut it off. If a session ever cannot play it that way, the message moves to the end of the utterance and the language_switch.wait_message log line carries reason: "immediate_unavailable".

Language check hold

With auto-switch on, the runner holds a user_speech_final until language ID has decided whether the utterance is in the session language, so your agent never answers a German sentence that the English STT transcribed. The hold is capped by languageId.autoSwitch.finalHoldMs (default 5000 ms). If language ID has not answered by then, the final is released as it is.

In practice most finals are not held at all. With languageId.timing at its default, early, the language is identified once while the caller is still speaking, as soon as languageId.minSpeechMs of speech is buffered (1500 ms by default). By the time the caller stops and the final arrives, the decision is already there.

Utterances shorter than minSpeechMs(a quick “yes” or “ok”) skip language ID entirely. They go to your agent immediately in the current language with no hold, the runner sends language_id_skipped instead of user_language, and the language of that utterance stays undetermined.

A lower minSpeechMs identifies sooner and lets fewer utterances skip the check, but Whisper tiny is less accurate on very short clips. A higher value is more accurate and lets more short replies through unchecked.

To guard against a false switch from one misheard clip, set languageId.autoSwitch.confirmUtterances to 2: the runner then switches only after two utterances in a row are identified as the new language.

Wait settings

All four settings live under languageId.autoSwitch and apply on the next deploy. They only matter when auto-switch is on.

The text for a language says that the agent is switching away from it, so it is written in that language. If the caller leaves English for German, the English text plays with the English voice. {language}becomes the name of the target in the caller's current language, so the English text above says “German” and the German text says “Englisch” when the caller goes the other way. A language without a text falls back to the built-in text for it, and then to English. The message plays once per attempt, so at most twice when the first attempt times out.

Example timelines

The agent starts in English. The caller says, in German, “Guten Tag, ich möchte meine Bestellung ändern” (about 3 seconds) and then, still talking, “die Nummer ist 4711”. Times are illustrative and measured from the moment the caller starts talking.

Target language already running

Cold models, end_of_utterance (default)

Cold models, immediate

Use immediate when silence feels worse than being talked over. If your runner does not yet support protected playback, the message plays when the caller stops talking, as in the end_of_utterance timeline.

Cold models, off

off fits agents that already tell the caller to hold on (a manual switch is one example) or setups where the client shows its own loading state from voice_language_switching.

Short utterance

early and end_of_utterance timing

Same German sentence (about 3 seconds), German models already running, minSpeechMs at its default of 1500 ms.

With continuous the first answer arrives at the same time as with early, and further passes keep running until the caller stops.

buffer_replay and first_utterance

The caller keeps talking while the German models start, so the two modes differ in what the new STT hears and what your agent receives. Cold models, end_of_utterance, swap at 8.0 s.

Client events

The runner sends these on the voice control channel as { type: "speech_event", event, ts, ...fields }. Fields with ? may be missing.

Manual switches with setVoiceLanguage send no voice_language_* events.

A UI can turn these into a small status line. A language check usually takes well under a second, so show it only if the hold lasts a little while. Call this handler with each speech_event message your client receives:

const names = new Intl.DisplayNames(["en"], { type: "language" });
let checkTimer: ReturnType<typeof setTimeout> | undefined;
let switching = false;

function handleSpeechEvent(msg) {
  switch (msg.event) {
    case "user_speech_final_held":
      if (msg.reason === "language_check") {
        checkTimer = setTimeout(() => setStatus("Checking language…"), 400);
      }
      break;
    case "voice_language_switching":
      clearTimeout(checkTimer);
      switching = true;
      setStatus(`Switching to ${names.of(msg.to)}…`); // "Switching to German…"
      break;
    case "user_speech_final_released":
      clearTimeout(checkTimer);
      if (!switching) setStatus(null); // a running switch keeps its status
      break;
    case "voice_language_changed":
      clearTimeout(checkTimer);
      switching = false;
      setStatus(null);
      break;
    case "voice_language_switch_failed":
      clearTimeout(checkTimer);
      switching = false;
      if (msg.reason === "language_unavailable") {
        setStatus(
          `${names.of(msg.to)} isn't available right now, continuing in ${names.of(msg.fallbackLanguage)}`,
        );
        setTimeout(() => setStatus(null), 5000);
      } else {
        setStatus(null);
      }
      break;
  }
}

Dashboard and CLI

In the dashboard, open the project Voice panel, then Advanced voice pipeline and Spoken language identification. The wait settings, the wait texts table and the per-language models table are there. Deploy to apply changes.

voicethere projects voice-advanced list
voicethere projects voice-advanced set languageId.autoSwitch.enabled true
voicethere projects voice-advanced set languageId.timing end_of_utterance
voicethere projects voice-advanced set languageId.autoSwitch.waitAudio first_utterance
voicethere projects voice-advanced set languageId.autoSwitch.waitMessage.mode immediate
voicethere projects voice-advanced set languageId.autoSwitch.waitMessage.skipWhenReady false
voicethere projects voice-advanced set languageId.autoSwitch.readyMessage.minSwitchMs 3000
voicethere projects voice-advanced set languageId.autoSwitch.waitMessage.texts '{"en":"One moment, switching to {language}.","de":"Einen Moment bitte, ich wechsle auf {language}."}'
voicethere projects voice-advanced set voice.profilesByLanguage '{"de":{"stt":"de","tts":"de-thorsten-high"},"es":{"stt":"es","tts":"es"}}'
voicethere projects voice-advanced set voice.allowedLanguages en,de,es
voicethere projects voice-advanced set languageId.autoSwitch.confirmUtterances 2

Other auto-switch settings

• languageId.autoSwitch.enabled, default false. Requires languageId.enabled.
• replayLastUtterance, finalHoldMs, minDwellMs, confirmUtterances and switchVoice tune stability and whether TTS follows STT. finalHoldMs is a safety cap (default 5000, max 10000): the runner holds a final until language ID decides and releases it after that many ms if LID has not answered.

Conversation history

When conversation history is enabled, each stored utterance includes an optional language field (ISO code from turn metadata). The dashboard Conversation timeline shows a language badge when present. Standalone user_language events are not stored as their own turns. Only finals and agent TTS text are persisted, with the language on the utterance row when known.

See Conversation history.

Observability

Every hold, switch and voice settings change also shows up in the session's agent logs as a line marked VoiceThere. See System log events for the full list with fields and example excerpts.

Session diagnostics and export bundles also carry runner logs with kind voice_language_auto_switch and voice_switch_wait_message (mode, played, duration and text length, not the text).

Wait and ready messages

Two short messages can mark a switch. The wait message covers a switch that is not finished when the caller stops talking. The ready message covers a switch that finished earlier but took long enough that a silent change would feel odd. Both are optional and use the project settings below. They apply to automatic switches. For switches your agent starts, the runner passes the same settings to the agent (getVoiceLanguageSwitchSettings).

Built-in ready messages:

voicethere projects voice-advanced set languageId.autoSwitch.readyMessage.minSwitchMs 3000
voicethere projects voice-advanced set languageId.autoSwitch.readyMessage.texts '{"de":"Gut, dann auf Deutsch."}'

Switching from your agent

The runner does not switch for you when your agent calls the API. The language-switch template shows the same steps in agent code, built from small calls that work with any speech vendor:

1. prepareVoiceLanguage(lang) starts the target speech services as soon as you know the language and says whether they are ready.
2. When the caller has finished, say your own wait message if the target is not ready.
3. setVoiceLanguage(lang) swaps STT and TTS. If the switch was slow, say a ready message in the new voice.
4. replayLastUtterance()feeds the caller's last utterance to the new STT, then answer it.
5. getVoiceLanguageSwitchSettings() returns the project settings above, so your agent follows the same wait and ready-message choices as the automatic flow.

Cold models and manual switches

The wait for cold models is bounded for switches your agent starts, too. setVoiceLanguagewaits up to 2 × 20 seconds, with one retry after the first 20, and then resolves { ok: false, reason: "language_unavailable" } instead of waiting on. prepareVoiceLanguage makes a single 20-second attempt and resolves ready: false with reason: "timeout" when the models are not up by then.

The language-switchtemplate handles both outcomes. It speaks the same holding message that names the target language, prepares once more after a timeout (and repeats the message), and tells the caller that the language isn't available when the second attempt fails or setVoiceLanguage reports language_unavailable. Customers get this template with the next @voicethere/agent release. Manual switches still send no voice_language_* events. The retry shows up in the system log as language_switch.retry with trigger: agent.

With a cloud vendor, map each language to a vendor and voice in voice.profilesByLanguage (sttVendor, ttsVendor, voice). Your agent logs each decision itself.

System log lines

Every decision of the automatic flow writes a language_switch.* line to the agent logs. Message texts you configured appear in the line; caller transcripts never do. Field details are in System log events.

Related

• Advanced voice settings for the full key reference
• STT & TTS vendors for cloud provider ids used in per-language models