Streaming automatic speech recognition transcribes at low latency but produces an unattributed word stream, while real-time applications such as customer-service analytics need each word tied to a speaker role. Role attribution is usually a separate stage, either an offline or streaming speaker-diarization model aligned to the words or a text-based large language model applied to the transcript, and role labels for training are scarce. We present RoleHear, a semi-supervised pipeline that makes a single cache-aware streaming transducer emit roles inline through lightweight in-vocabulary role tokens: a fast turn-boundary marker followed by a per-role token a few words later. An offline transducer teacher, fine-tuned into a role-attributed recognizer, relabels unlabeled audio, and a multimodal large language model supplies silver role labels where gold annotations are unavailable. The streaming student is fine-tuned on the mixture in a single pass. Across customer-service calls with gold roles and earnings calls with silver roles, RoleHear attributes roles accurately while preserving transcription quality and narrowing the gap to the offline teacher.
Figures & tables
Split
Earnings-21
Earnings-22
ServiceCall
Labeled Train
10.72
0 24.76
0 31.08
Labeled Val
0 3.09
00 9.16
00 7.22
Labeled Test
0 4.88
0 12.38
00 6.86
Unlabeled
18.85
0 63.01
0 62.56
Total
37.54
109.31
107.72
Chunk (s)
23.3
0 23.0
00 9.8
Table 1: Dataset statistics, reporting the total duration in hours per split and the average chunk length in seconds.
Method
Earnings-21
Earnings-22
ServiceCall-3R
ServiceCall-5R
WER ↓
RWER ↓
F1 ↑
WER ↓
RWER ↓
F1 ↑
Δ WER ↓
Δ RWER ↓
Δ F1 ↑
Δ WER ↓
Δ RWER ↓
Δ F1 ↑
Teacher + MLLM
6.49 ±0.00
-
-
12.24 ±0.00
-
-
- 0.00 ±0.00
- 0.00 ±0.00
- 0.00 ±0.00
- 0.00 ±0.00
- 0.00 ±0.00
- 0.00 ±0.00
FT Teacher
6.88 ±0.06
19.24 ±0.60
82.83 ±0.70
11.79 ±0.04
25.94 ±0.78
80.59 ±0.96
-2.98 ±0.06
-6.29 ±0.44
+1.21 ±0.77
-3.04 ±0.07
-5.56 ±0.39
-8.92 ±2.64
Student + MLLM
8.27 ±0.00
-
-
13.55 ±0.00
-
-
+0.78 ±0.00
+1.66 ±0.00
-1.20 ±0.00
+0.78 ±0.00
-1.82 ±0.00
+1.42 ±0.00
FT Student
7.74 ±0.14
31.78 ±0.63
64.06 ±1.66
12.93 ±0.06
42.29 ±0.71
52.16 ±1.65
-1.31 ±0.17
+0.21 ±0.65
-8.83 ±1.39
-1.30 ±0.09
+2.81 ±0.45
-19.83 ±0.93
RoleHear
7.56 ±0.09
27.55 ±0.81
72.95 ±2.20
12.33 ±0.16
35.80 ±0.60
65.97 ±2.07
-1.99 ±0.09
-2.92 ±0.35
-5.95 ±0.83
-2.10 ±0.13
-1.80 ±0.98
-17.34 ±1.82
Table 2: Role-attributed ASR on the labeled test split (%, 5 -seed mean, std in subscript). ServiceCall-5R uses the 5 -role set, the others the 3 -role set; roles are gold on ServiceCall, silver on Earnings. The student rows decode at RC=1 ( 160 ms). FT denotes fine-tuned. Earnings columns are absolute; ServiceCall columns are the change ( Δ , in pp) from the Teacher + MLLM row.
Model
Delay
RL
WER ↓
RWER ↓
F1 ↑
Teacher
0
0
6.88 ±0.08
17.99 ±0.85
83.80 ±1.44
−1
Turn
6.88 ±0.06
19.24 ±0.60
82.83 ±0.70
Student
0
0
7.71 ±0.07
37.32 ±1.03
46.07 ±4.90
1
360
7.74 ±0.14
31.78 ±0.63
64.06 ±1.66
3
1080
7.74 ±0.12
31.75 ±1.51
60.91 ±3.05
5
1800
7.75 ±0.05
32.89 ±1.22
60.10 ±1.74
Table 3: Role-emission delay on Earnings-21, fine-tuning only; student at RC=1 . WER, RWER, F1 (%); role latency RL≈delay×360 ms. Delay −1 places the token after the turn’s last word (highest latency). Bold / underline : best/second-best per column (student rows).
Delay
RC
RL
AL
WER ↓
RWER ↓
F1 ↑
0
0
0
80
7.84 ±0.07
36.25 ±1.17
42.55 ±1.42
1
0
160
7.71 ±0.07
37.32 ±1.03
46.07 ±4.90
6
0
560
7.51 ±0.05
32.60 ±0.87
60.98 ±8.64
13
0
1120
7.40 ±0.07
29.54 ±1.13
65.93 ±9.87
1
0
360
80
7.86 ±0.13
32.31 ±0.51
63.39 ±0.76
1
360
160
7.74 ±0.14
31.78 ±0.63
64.06 ±1.66
Table 4: Right context and delay on Earnings-21, fine-tuning only; streaming student. WER, RWER, F1 (%); role latency RL≈delay×360 ms, algorithmic latency AL=(1+RC)×80 ms. Bold / underline : best/second-best per column within each delay.