Load testing VoiceThere: 45 voice conversations, measured
October 19, 2026 · platform
We load test the voice pipeline with synthetic callers. Each caller is a real WebRTC session that speaks to an agent, so every turn can be checked against a phrase we know. This post has the results, with the date of each run: the 45-conversation series, how many conversations one pair of speech servers carries, and how long the agent waits after a caller stops talking. All runs ran in our test environment between October 6 to 18, 2026. None of the numbers come from production.
- 20
- runs behind the tables (3 ramp, 7 of 45 conversations, 2 restart, 5 pause-probe settings, 2 end-of-turn runs, 1 live check)
- 1,725
- turns in the headline run, 1,725 of 1,725 understood
- 45
- conversations at once in the headline run
- Oct 6 to 18
- 2026-10-06 to 2026-10-18, 2026
A run that appears in two tables is counted once. Ramp runs are the 2 to 20 conversation steps in the capacity table.
Method
The callers run on a host outside the test environment. They speak with a synthetic voice over WebRTC (Opus audio). Our in-house STT and TTS speech services inside the environment do the speech recognition and the agent's voice. Each turn is a spoken counting phrase of 4 or 5 numbers, and the agent has to repeat it.
A turn counts as understood only if two checks pass. The platform's speech recognition has to have heard the phrase correctly (server side), and the caller's own transcription of the agent's reply has to match it too. A turn that fails either check is a failed turn.
Before every run we pause scheduled probes and test jobs and drain pending ones, then restore and check them afterwards. The capacity runs also refuse to start until the environment runs the expected server build.
Four latency numbers appear below:
- Speech end to transcript runs from the caller's last word to the final transcript. It includes the end-of-turn wait (see end-of-turn wait), which is configured waiting and not compute. After the wait, finalizing adds roughly 0.1 to 0.25 s.
- Transcript (server side) runs from the moment the caller starts speaking to the final transcript. It contains the phrase itself (about 3 s) and the end-of-turn wait, so it says little about recognition speed.
- Transcript to reply start runs from the final transcript to the agent's first audio.
- Speech end to reply start runs from the caller's last word to the agent's first audio. It includes the end-of-turn wait. In the load tests the caller's speech-end mark also includes trailing playback, so it reads longer than in a single conversation.
If you want to run something similar against your own project, the steps are:
- Open real WebRTC voice sessions with a headless client, the same way the browser and Node SDKs do.
- Have each caller speak a random string of 4 or 5 numbers (for example "seven. one. three. four.") in a synthetic voice. Random phrases keep the TTS phrase cache from hiding synthesis cost, so it is off in the sizing runs.
- Wait for the agent to repeat the numbers. A turn passes only if the platform's speech recognition heard the phrase correctly and the caller's own transcription of the agent's audio matches it.
- Measure from the caller's side unless a metric says server side, and take percentiles over all turns of the run. Run 45 callers for 10 minutes, with nothing pinned.
- Pause scheduled probes and test jobs before the run and restore them afterwards, so nothing else shares the environment.
Headline: 45 conversations at once
The final run of the series, October 8, 2026, put 45 conversations on the Advanced plan for 10 minutes. All of them stayed clean, and 1,725 of 1,725 turns were understood. The median time from the final transcript to the agent's first audio was 231 ms (667 ms at p95). The server-side transcript p95 was 4.45 s, which includes the phrase and the end-of-turn wait. Speech end to reply start was 2.54 s at p95, with the end-of-turn wait that was in force at the time.
The run used 3 agent workers at about one CPU core each, plus speech servers that scaled on their own. The default end-of-turn wait has changed since, so the two speech-end figures here are not the current wait (see below).
- First run (2026-10-06)
- Final run (2026-10-08)
Reply start (transcript to first audio)
Transcript (server side)
Speech end to reply start
Milliseconds, p95
Show data
| First run (2026-10-06) (ms) | Final run (2026-10-08) (ms) | |
|---|---|---|
| Reply start (transcript to first audio) | 982 | 667 |
| Transcript (server side) | 7,510 | 4,450 |
| Speech end to reply start | 4,620 | 2,540 |
Show data
| Run (UTC) | Change before the run | Clean sessions | Turns understood | Reply start p95 | Transcript p95 | Speech end to reply start p95 |
|---|---|---|---|---|---|---|
| 2026-10-06 16:15 | First 45-conversation run | 29/45 | 98.8 % | 982 ms | 7.51 s | 4.62 s |
| 2026-10-07 03:51 | Earlier scale-out of speech recognition | 31/45 | 98.85 % | 1069 ms | 6.74 s | 4.74 s |
| 2026-10-07 05:52 | One speech stream per utterance | 44/45 | 99.94 % | 832 ms | 6.66 s | 4.27 s |
| 2026-10-07 08:52 | Scaling that reacts in seconds | 43/45 | 99.88 % | 696 ms | 4.94 s | 3.14 s |
| 2026-10-07 18:22 | Late transcript fix, lower stream cap | 45/45 | 100 % (1,740 of 1,740) | 705 ms | 5.89 s | 3.89 s |
| 2026-10-08 01:41 | No change | 45/45 | 100 % | 602 ms | 4.55 s | 2.69 s |
| 2026-10-08 15:55 (final) | No change | 45/45 | 100 % (1,725 of 1,725) | 667 ms | 4.45 s | 2.54 s |
Reply start is measured from the final transcript to the agent's first audio. Transcript is measured from the moment the caller starts speaking.
Capacity and sizing
Per pair of one STT server and one TTS server, the number of conversations that fit in the latency budget went from 6 to 12 to 15. On the agent worker we measured about 60 to 70 m of CPU (thousandths of a core) per conversation, 69 % of it in native audio threads, linear and without a leak. One speech recognition stream costs about 33 m.
- Conversations
- Highlighted: the new default or latest value
Sized by estimate
Sized from a 1-CPU sweep
Agent workers on a full core
Conversations that fit in the latency budget
Show data
| Conversations (conversations) | |
|---|---|
| Sized by estimate | 6 |
| Sized from a 1-CPU sweep | 12 |
| Agent workers on a full core | 15 |
Show data
| Run (UTC) | Change | Result |
|---|---|---|
| 2026-10-06 02:27 | Starting point: speech servers sized by estimate, agent workers at 0.85 CPU core | 6 conversations pass (reply start p95 991 ms). 8 fail on latency (1,977 ms). |
| 2026-10-06 11:50 | Speech servers sized from a 1-CPU sweep: one TTS engine on one thread, STT on one thread with two decode slots | 12 pass (reply start p95 593 ms). 15 fail on latency. |
| 2026-10-06 14:49 | Agent workers get one full CPU core (0.85 to 1.0) | 15 pass (reply start p95 896 ms). At 20, one turn failed. |
Capacity is per pair of one STT server and one TTS server. The agent worker was the bottleneck at the last step, which is why that step changed the agent worker and not the speech servers.
The changes that improved latency or reliability in the 45-conversation runs, with the runs they were measured in:
| Change | Measure | Before | After | Runs (UTC) |
|---|---|---|---|---|
| One speech stream per utterance instead of per conversation, so each turn lands on the least busy server | Clean sessions | 31/45 | 44/45 | 2026-10-07 03:51 to 05:52 |
| Same change | WebRTC connect p95 | 13.2 s | 3.6 s | 2026-10-07 03:51 to 05:52 |
| Metrics every 5 s, scaling on peaks and refused requests, round-robin routing with retry | Transcript p95 | 6.66 s | 4.94 s | 2026-10-07 05:52 to 08:52 |
| Same change | Refused speech stream opens | 66 | 0 | 2026-10-07 05:52 to 08:52 |
| A late transcript is dropped instead of answering the next turn, the agent waits up to 8 s, stream cap lowered from 30 to 25 | Clean sessions | 43/45 | 45/45 | 2026-10-07 08:52 to 18:22 |
| Same change | Finalize timeouts | 7 | 0 | 2026-10-07 08:52 to 18:22 |
| Speech servers take traffic only after their models are loaded | Sessions that survive a restart with a simulated 20 s model load (12 conversations) | 0/12 | 11/12 | 2026-10-07 (rollout runs 6 and 5) |
End-of-turn wait
These are the newest results, from 2026-10-18. The wait between the caller's last word and the agent's answer is mostly configured, not computed.
Two settings set it. vad.minSilenceDurationMs is how much silence ends the caller's speech. vad.sttGateHoldMs keeps the speech recognizer open after that, so speech that resumes inside the hold is merged into the same turn. Lower values answer sooner and split a sentence when the caller pauses mid-sentence.
To find the trade-off we spoke each sentence in two halves with an exact pause between them, and checked whether the platform heard one turn or two.
- Wait, last speech to final (p50)
- Shortest pause that splits a turn
- Highlighted: the new default or latest value
1300/1000old default
1300/400
1100/400new default
1000/400
800/400
Milliseconds
Show data
| Wait, last speech to final (p50) (ms) | Shortest pause that splits a turn (ms) | |
|---|---|---|
| 1300/1000 | 1,194 | 700 |
| 1300/400 | 911 | 500 |
| 1100/400 | 801 | 500 |
| 1000/400 | 755 | 300 |
| 800/400 | 692 | 300 |
- Cyan: stays one turn
- Amber: split in 2 of 4 conversations
- Grey: splits into two turns
Show data
| Min silence / gate hold (ms) | 300 ms pause | 500 ms pause | 700 ms and longer |
|---|---|---|---|
| 1300/1000 | one turn | one turn | split |
| 1300/400 | one turn | split | split |
| 1100/400 | one turn | split | split |
| 1000/400 | 2 of 4 split | split | split |
| 800/400 | split | split | split |
Show data
| Min silence / gate hold (ms) | Pauses that stay one turn (ms) | Pauses that split | Last speech to final transcript, p50 / p95 |
|---|---|---|---|
| 1300 / 1000 (old default) | 300, 500 | 700 and longer | 1194 / 1371 ms |
| 1300 / 400 | 300 | 500 and longer | 911 / 1023 ms |
| 1100 / 400 (new default) | 300 | 500 and longer | 801 / 927 ms |
| 1000 / 400 | 300 in 2 of 4 | 300 in 2 of 4, and 500 and longer | 755 / 876 ms |
| 800 / 400 | none | 300 and longer | 692 / 793 ms |
The energy-based voice activity detection counts the fading end of a word as silence, so the real tolerance is shorter than the two values added together. We measured the split point instead of calculating it.
We then ran 45 conversations on the 1000 / 400 setting and compared them with a 45-conversation run on the old defaults from 2026-10-16.
- 1300 / 1000 (2026-10-16)
- 1000 / 400 (2026-10-18)
p50
p95
Milliseconds
Show data
| 1300 / 1000 (2026-10-16) (ms) | 1000 / 400 (2026-10-18) (ms) | |
|---|---|---|
| p50 | 2,540 | 1,648 |
| p95 | 2,857 | 1,941 |
Show data
| 1300 / 1000 (2026-10-16) | 1000 / 400 (2026-10-18) | |
|---|---|---|
| Speech end to final transcript, p50 / p95 | 2366 / 2568 ms | 1487 / 1672 ms |
| Speech end to reply start, p50 / p95 | 2540 / 2857 ms | 1648 / 1941 ms |
| Turns ok / failed / tolerated STT slips | 1769 / 1 / 2 | 1881 / 1 / 2 |
| Step result | pass | pass |
The 2026-10-18 run used 1000 / 400. The shipped defaults are 1100 / 400, checked in a single conversation (see below).
Under 45 conversations the wait is longer than in the single-conversation probe (1.49 s against 0.76 s median for 1000 / 400). The load client's speech-end mark includes trailing playback and the servers are busy. The like-for-like comparison is the same harness against 2026-10-16, which is 0.88 s lower at the median. A single conversation on the old defaults measured 1.2 s.
New defaults: 1100 / 400
On 2026-10-18 the defaults moved from 1300 / 1000 to 1100 / 400. We checked them live with no overrides. Pauses of 300 ms stay one turn (0 of 2 split), pauses of 500 ms split (2 of 2 split), and the wait from last speech to the final transcript is 778 ms at the median. Projects that set their own values keep them. When to tune them, and the trade-off table, are in End-of-turn wait in the docs.
Limits
What these numbers do not show:
- Call setup did not get faster. Session start p95 went from 53 s to 68 s between the first and the final 45-conversation run, and WebRTC connect p95 from 3.6 s to 5.9 s. Both vary run to run (47 to 71 s and 3.6 to 13.2 s) and were not the target of this work.
- The transcript p95 of the 2026-10-07 18:22 run (5.89 s) is worse than the run before it (4.94 s). The speech recognition servers scaled down between waves of speech and 270 stream opens were refused for a moment. Later runs came back to 4.55 s and 4.45 s.
- Each pause-probe setting was run with 4 conversations, so the split points are indicative and not a statistical result.
- The callers are synthetic voices speaking short counting phrases. We did not measure real callers here.
- Everything ran in our test environment, as one 45-conversation burst per run. Latency in these runs also depends on the network path between the load client and the environment: on 2026-10-16 we traced slow minutes to that path, not to the speech servers.
- No number in this post comes from production.
Related
- Docs: End-of-turn wait and Advanced voice settings