QoS-QoE translation is a fundamental problem in multimedia systems because it characterizes how measurable system and network conditions affect user-perceived experience. Although many prior studies have examined this relationship, their findings are often developed for specific setups and remain scattered across papers, experimental settings, and reporting formats, limiting systematic reuse, cross-scenario generalization, and large-scale analysis. To address this gap, we first introduce QoS-QoE Translation dataset, a source-grounded dataset of structured QoS-QoE relationships from the multimedia literature, with a focus on video streaming related tasks. We construct the dataset through an automated pipeline that combines paper curation, QoS-QoE relationship extraction, and iterative data evaluation. Each record preserves the extracted relationship together with parameter definitions, supporting evidence, and contextual metadata. We further evaluate the capability of large language models (LLMs) on QoS-QoE translation, both before and after supervised fine-tuning on our dataset, and show strong performance on both continuous-value and discrete-label prediction in bidirectional translation, from QoS-QoE and QoE-QoS. Our dataset provides a foundation for benchmarking LLMs in QoS-QoE translation and for supporting future LLM-based reasoning for multimedia quality prediction and optimization. The complete dataset and code are publicly available at https://yyu6969.github.io/qos-qoe-translation-page/, for full reproducibility and open access.
Figures & tables
Figure 1. Overview of the QoS-QoE Translation dataset construction pipeline. The pipeline begins with paper curation, followed by QoS-QoE relationship extraction and iterative data evaluation.
Figure 2. An example JSON record from the QoS-QoE Translation, showing enriched metadata and core relationship.
Field
Definition
id
Unique identifier for each dataset record.
year
Publication year of the source paper.
venue
Publication venue of the source paper.
domain
Application domain of the record.
protocol
Streaming or transport protocol used in the study.
network_type
Access network environment.
Table 1. Field definitions of QoS-QoE Translation.
Figure 3. Dataset analysis of QoS-QoE Translation. The first row shows key metadata distributions, including protocol, network type, device type, and video type. The second row shows distributions over year, venue, and QoS/QoE parameters.
Model
Before SFT
After SFT
MAPE ↓
Accuracy@ δ↑
Accuracy ↑
Macro-F1 ↑
MAPE ↓
Accuracy@ δ↑
Accuracy ↑
Macro-F1 ↑
Qwen3-8B
20.23
64.20
67.48
55.63
11.79
78.13
80.49
72.29
Qwen3-32B
16.46
67.95
68.29
56.69
9.41
80.24
84.55
70.12
Qwen3.5-35B-A3B
14.43
72.94
69.92
61.33
8.49
83.41
90.24
84.63
Llama-3.1-8B-Instruct
26.72
55.90
50.41
35.62
11.34
79.49
83.74
70.79
Llama-3.3-70B-Instruct
13.76
72.70
70.73
56.29
9.49
81.90
90.24
81.93
Table 2. Overall model evaluation results before and after SFT. All values are percentages (%).
Task
Model
MAPE (%) ↓
Accuracy@ δ (%) ↑
Accuracy (%) ↑
Macro-F1 (%) ↑
QoS → QoE
Qwen3-8B
9.11
78.55
75.00
73.43
Qwen3-32B
7.41
82.61
77.50
62.18
Qwen3.5-35B-A3B
7.05
83.77
77.50
66.53
Llama-3.1-8B-Instruct
8.79
81.74
75.00
61.31
Llama-3.3-70B-Instruct
6.88
83.77
85.00
77.78
QoE → QoS
Qwen3-8B
13.47
77.67
83.13
71.43
Table 3. Overall model evaluation results after SFT for QoS → QoE and QoE → QoS translation. All values are percentages (%).
Current state-of-the-art Quality Estimation (QE) in machine translation relies on massive, proprietary LLMs, raising data privacy concerns. We demonstrate that smaller, open-source LLMs (<30B parameters) are a viable, cost-effective and privacy-preserving alternative. Using a single-pass prompting strategy, our models simultaneously generate quality scores, MQM error annotations, suggested error corrections, and full post-editions. Our analysis shows these models achieve highly competitive system-level correlations with human judgments that outperform traditional neural metrics, fine-tuned models, and human inter-annotator agreement, effectively approximating the capabilities of much larger proprietary LLMs.
Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland
The rapid advancement of Natural Language Generation (NLG) has made the reliable evaluation of generated text increasingly critical, as these systems, such as large language models (LLMs), are now widely deployed in real-world applications. However, traditional automatic metrics fail to capture the multifaceted nature of perceived quality. In this paper, we introduce TextQ-German, a novel dataset suite for human-centered evaluation of German NLG from a Quality of Experience (QoE) perspective, covering automatic text summarization and machine translation. Through crowdsourcing studies with German speakers, we collect human quality ratings and identify relevant perceptual quality dimensions for each task. We develop automatic QoE prediction models, including transformer-based, linguistic feature-based, and hybrid approaches. Hybrid models outperform pure transformer baselines in almost all experimental settings, while linguistic features alone can approach the performance of fine-tuned language models. The dataset is extended with LLM-generated outputs annotated with overall QoE scores. Final validation on held-out sets indicates generalization to unseen data. Our work contributes a publicly accessible resource for NLG evaluation and baselines for automatic QoE prediction, providing a foundation for developing NLG systems that better align with human quality perception.
Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz +1
Technische Universität Berlin, Berlin, Germany. · Speech and Language Technology Lab, German Research Center for Artificial Intelligence (DFKI), Berlin, Germany.
We present HydraQE, our contribution to the IWSLT 2026 Speech Translation Metrics shared task. HydraQE is an end-to-end, reference-free quality estimation (QE) system for speech translation built on a Qwen3-ASR backbone, which accepts source audio and a translation hypothesis as joint input. Hidden states from all backbone layers are combined via a learnable sparsemax scalar mix, then re-encoded by a lightweight bidirectional Transformer to enable full cross-modal interaction prior to pooling into a shared embedding. Three independent prediction heads are trained on complementary supervision signals: human direct assessment (DA) annotations, MetricX-24 pseudo-labels, and xCOMET pseudo-labels. To address the scarcity of human-annotated data, we train on a combination of synthetically corrupted examples and silver pseudo-labeled machine translation outputs, using a curriculum that begins on synthetic and silver data and gradually shifts toward human-annotated examples. HydraQE outperforms cascaded text-based baselines and prior direct speech QE systems, demonstrating that end-to-end speech translation QE is competitive with cascaded approaches.
Kevin Krahn, Eric Fosler-Lussier
Dept. of Computer Science and Engineering The Ohio State University