Automatic speech recognition is usually evaluated with word error rate (WER), although voice workflows often require exact written values. VoiceCodeBench measures whether transcripts preserve identifiers, paths, commands, and other structured tokens needed by downstream software. It contains 300 human-recorded English workplace segments (5.59 hours, 85 speakers) and 1,482 audited entities across 26 types and eight domains. Under a raw-audio-only protocol, we evaluate 19 batch and streaming systems using WER, Canonical Token/Entity Match (CTEM), and strict segment-level Task Success Rate (TSR). Across systems, WER has little rank agreement with CTEM (Spearman ρ=−0.28) or TSR (ρ=−0.22). The best CTEM and TSR are 91.8% and 68.7%. Even the strongest system therefore leaves nearly one-third of recordings with an unrecovered critical value. Symbol-, separator-, and boundary-sensitive entities account for most errors.
Figures & tables
Domain
Recordings
Contact/routing
45
Technical/IT/developer
55
Retail/logistics/order
45
Finance/billing
40
Healthcare/administration
35
Legal/insurance/government
35
Table 1: Recording distribution by workflow domain.
Family
Entity types
Contact/routing
Email address, phone number, phone extension, person/team name, postal address
Network/web
URL, IP address, port number
Code/system
Command, CLI flag, file path, environment variable, code symbol, version
Identifiers
Reference ID, product code, account/record number
Numeric/measurement
Currency amount, percentage, measurement, plain number, date, time
Language form
Acronym/initialism, spelled sequence, domain term
Table 2: Entity taxonomy used for structured-token scoring.
Band
Recordings
Entities
Words
Light
30
3
95–130
Standard
114
4
103–169
Dense
93
5–6
122–177
Stress
63
7–8
156–206
Table 3: Difficulty-band composition.
Target
Transcript evidence
Decision
212-555-0104
“212 555 0104”
Pass: format
--dry-run
“dry run”
Fail: symbols
.env.provision.local
“.env.provision”
Fail: suffix
500 mg
“500 micrograms”
Fail: unit
Table 4: Illustrative recoverability decisions.
Batch
Streaming
System
WER ↓
CTEM ↑
TSR ↑
System
WER ↓
CTEM ↑
TSR ↑
ElevenLabs Scribe v2
6.9
91.6
67.7
OpenAI gpt-live-transcribe
5.9
91.8
68.7
Deepgram Nova-3
8.2
90.9
68.7
Cartesia Ink 2
9.3
90.1
63.7
OpenAI gpt-transcribe
6.1
89.3
61.0
Deepgram Nova-3
9.3
88.9
61.7
Google Cloud Chirp 3
5.8
88.8
60.3
Meta Muse Voice Transcribe 1.0
7.4
86.8
55.7
Whisper large-v3
5.8
87.6
54.3
Google Cloud Chirp 3
6.7
86.1
50.3
Table 5: Raw-audio-only results on 300 recordings and 1,482 entities. All values are percentages. Best values are bold.
Figure 1: WER against CTEM and TSR for 18 configurations. Inworld STT 1 streaming is omitted for scale. Plot correlations use the displayed points. Lower WER and higher CTEM/TSR are better.
Type
Targets
Min.
Median
Max.
URL
62
9.7
46.8
66.1
Command
50
8.0
52.0
78.0
File path
51
7.8
54.9
70.6
Postal address
40
27.5
55.0
77.5
Email address
65
9.2
63.1
80.0
Table 6: CTEM distribution for the five hardest entity types.
Family
Targets
CTEM (%)
Numeric/measurement
400
93.7
Language form
202
92.1
Identifiers
305
87.3
Contact/routing
213
75.3
Code/system
245
69.1
Network/web
117
65.9
Table 7: Aggregate CTEM by entity family across 19 systems.