Rethinking Communication Metrics: How Should We Measure Meaning?

Niloofar Tavakolian; Hakimeh Purmehdi; Jungyeon Baek

cs.LGAug 21, 2026

Rethinking Communication Metrics: How Should We Measure Meaning?

Authors: Niloofar Tavakolian, Hakimeh Purmehdi, Jungyeon Baek

Organizations: Ericsson Canada, Montreal, QC, Canada · Concordia University, Montreal, QC, Canada

Abstract

Semantic communication shifts the objective of communication systems from accurate symbol reconstruction toward meaning preservation, task accomplishment, and efficient information exchange. However, its evaluation remains fragmented across telecommunications, natural language processing, computer vision, and machine learning, and no single metric can characterize semantic quality across modalities, tasks, and channel conditions. This article surveys key performance indicators (KPIs) for text- and image-based semantic communication systems from a unified, evaluation-centered perspective. Unlike prior surveys primarily organized around architectures, applications, or transmission strategies, this work focuses on how semantic success should be defined and measured. Existing KPIs are classified according to communication goal, source modality, receiver output, reference availability, evaluation level, and channel or resource constraints. The survey reviews reconstruction-based, task-oriented, reference-free, representation-level, perceptual, and channel-aware metrics, and presents a cross-modality comparison of their roles, strengths, and limitations. It further analyzes how unresolved semantic-KPI challenges affect monitoring, quality assurance, resource optimization, fault diagnosis, and standardization. Key open problems include the absence of universal semantic success criteria and standardized semantic ground truth, semantic drift, limited reference-free evaluation, weak integration of machine-learning metrics with communication constraints, and the lack of relation-level and multimodal KPIs. Finally, future research directions are outlined toward standardized, interpretable, adaptive, task-aware, and communication-aware evaluation frameworks.

Figures & tables

Axis	Possible categories	Why it matters
Communication goal	Reconstruction, semantic preservation, semantic state recovery, task accomplishment	Determines whether success is measured by similarity, task completion, or semantic distortion.
Source modality	Text, image, multimodal	Determines the form of semantic content and the suitable KPI families.
Receiver output	Reconstructed source, embedding, label, action, structured symbols	Determines what variables can be compared during evaluation.
Reference availability	Reference based, reference free	Determines whether the original source or gold reference is needed.
Evaluation level	Sample level, dataset level, system level	Determines whether metrics are computed per example, over distributions, or across systems.
Channel/resource condition	SNR, bandwidth, rate, latency, energy	Enables communication-aware semantic evaluation and telecom-oriented KPIs such as semantic reliability, throughput, efficiency, and QoS/QoE measures.

TABLE I: General evaluation axes for selecting semantic communication KPIs

Panel of Figure 2 — Fig. 1: Generic semantic communication evaluation pipeline. A source signal $S$ (e.g., text, image, or multi-modal content) is encoded into a semantic representation $W$ , transmitted through a communication channel, and recovered as $\hat{W}$ . Depending on the communication objective, the receiver may reconstruct an output $\hat{S}$ or generate a task-specific output $\hat{Y}$ . Modality-specific notations used in later sections are derived from this generic framework. Different KPIs evaluate performance at different stages of the pipeline, including reconstruction fidelity, semantic preservation, task effectiveness, and telecom-oriented semantic reliability and efficiency under communication constraints.

Text KPI category	Core question	Representative KPIs
Semantic reconstruction	Did the reconstructed text preserve the original meaning or wording?	BLEU, ROUGE, METEOR, chrF, BERT cosine, BERTScore, BLEURT, COMET
Goal-oriented task success	Did the receiver complete the intended task?	accuracy, F1, exact match, Recall@K, MRR, task success rate
Reference-free evaluation	Can quality be estimated without a gold reference at the receiver?	COMET-QE, anchor similarity, semantic outage, perplexity
Telecom-style semantic state recovery	Was the semantic state recovered under channel/resource constraints?	semantic distortion, semantic error probability, rate–distortion curves, outage probability

TABLE II: KPI Categories for Text-Based Semantic Communication

Metric	Type	Main comparison	Formula idea	Main limitation
BLEU	Surface-overlap	$S$ vs. $\hat{S}$	Modified n-gram precision and brevity penalty	Penalizes paraphrases
ROUGE	Surface-overlap	$S$ vs. $\hat{S}$	n-gram recall or LCS overlap	Mostly lexical
METEOR	Surface-overlap and alignment	$S$ vs. $\hat{S}$	Unigram F-score with fragmentation penalty	Still reference-bound
chrF / chrF++	Character/word overlap	$S$ vs. $\hat{S}$	Character n-gram F-score	Not truly semantic
BERT cosine	Embedding-based	$B(S)$ vs. $B(\hat{S})$	Cosine similarity	Depends on encoder and pooling
BERTScore	Embedding-based	Contextual token embeddings	Max token cosine matching	Requires reference

TABLE III: Summary of Text Semantic Reconstruction Metrics

KPI	Task type	Main comparison	Formula idea	Main limitation
Intent accuracy	Intent classification	$Y$ vs. $\hat{Y}$	correct predictions / total samples	Too coarse when slot details matter
Slot F1	Slot filling / entity extraction	predicted slots vs. ground-truth slots	harmonic mean of precision and recall	Sensitive to boundary and label errors
Frame accuracy / exact match	Semantic frame prediction	full predicted frame vs. full ground-truth frame	exact frame match rate	Very strict; no partial credit
QA exact match / F1	Question answering	predicted answer vs. ground-truth answer	exact answer match or token overlap	May miss semantic equivalence
Recall@K / MRR / NDCG	Retrieval	ranked output vs. relevant item(s)	correct item rank or relevance-weighted ranking	Depends on candidate set and relevance labels
Task success rate	End-to-end goal completion	final task outcome	successful tasks / total tasks	Requires clear definition of success

TABLE IV: Summary of Goal-Oriented Text Semantic Communication KPIs

KPI	Main idea	Required at receiver	Strength	Limitation
Quality estimation	A learned model predicts output quality without a gold reference	Received text, and sometimes source or side information	Practical for deployment	Model-dependent and may drift
Anchor similarity	Compare the received output embedding with a transmitted semantic anchor	Decoded output and semantic anchor	Reference-free semantic check	Adds overhead
Semantic success / outage	Decide success using a semantic similarity threshold	Similarity score and threshold	Telecom-friendly reliability measure	Threshold selection is subjective
Perplexity	Use language model likelihood as a fluency proxy	Decoded text only	Detects garbled text	Fluency does not guarantee meaning

TABLE V: Summary of Reference-Free Text Semantic Communication KPIs

KPI	Formula idea	Best suited for	Main limitation
Semantic distortion	Expected semantic distance between $W$ and $\hat{W}$	Embeddings, triplets, belief states	Requires defining semantic distance
Semantic error probability	Probability of semantic mismatch	Discrete labels or symbols	Too strict for partial semantic recovery
Semantic outage probability	Probability of unacceptable semantic quality	Wireless reliability analysis	Threshold selection is subjective
Rate–semantic tradeoff	Semantic quality per resource cost, or constrained optimization	Resource-aware SemCom	Requires standardized cost and quality definitions

TABLE VI: Summary of Telecom-Style Text Semantic State Recovery KPIs

Image KPI category	Core question	Representative KPIs
Pixel fidelity	Is the received image numerically close to the source image?	MSE, PSNR
Structural/perceptual quality	Does the received image preserve visual structure and perceptual quality?	SSIM, MS-SSIM, LPIPS
Visual semantic similarity	Are high-level visual features preserved?	Feature cosine similarity, CLIP similarity
Downstream visual task utility	Can the receiver complete the intended visual task?	Accuracy, mAP, IoU, Dice, Recall@K
Distribution-level realism	Are reconstructed or generated images realistic at the dataset level?	FID, KID, Inception Score
Channel-aware semantic reliability	Is semantic visual quality maintained under channel/resource constraints?	Semantic outage, semantic distortion, task outage vs. SNR/rate

TABLE VII: KPI categories for image-based semantic communication

Metric	Type	Range	Higher is better?	Main limitation
MSE	Pixel-level error	$[0,\infty)$	No	Treats all pixels equally; not semantic
PSNR	Logarithmic pixel fidelity	$[0,\infty)$ dB	Yes	Weak correlation with perception and task meaning

TABLE VIII: Summary of pixel-fidelity metrics for image-based semantic communication

Metric	Type	Range	Higher is better?	Main limitation
SSIM	Structural similarity	Typically $[0,1]$ , theoretically $[-1,1]$	Yes	Single-scale; not explicitly semantic
MS-SSIM	Multi-scale structural similarity	$[0,1]$	Yes	Still not fully semantic
LPIPS	Learned perceptual distance	Non-negative, implementation-dependent	No	Model-dependent; may not reflect task meaning

TABLE IX: Summary of structural and perceptual quality metrics for image-based semantic communication

Metric	Main comparison	Strength	Main limitation
Feature cosine similarity	$F(I)$ vs. $F(\hat{I})$	Captures learned high-level visual similarity	Encoder-dependent; may miss task-critical details
CLIP image-image similarity	$E_{I}(I)$ vs. $E_{I}(\hat{I})$	Evaluates semantic similarity in a vision-language space	May miss fine spatial/domain-specific details
CLIP image-text similarity	$E_{T}(T)$ vs. $E_{I}(\hat{I})$	Useful when semantic goal is textual or caption-based	Depends on caption/prompt and CLIP domain coverage

TABLE X: Summary of visual semantic similarity metrics for image-based semantic communication

KPI	Task type	Main comparison	Main limitation
Accuracy	Classification	$Y$ vs. $\hat{Y}$ class labels	Too coarse for localization or structure
mAP	Object detection	Predicted boxes/classes vs. ground truth	Depends on detector and IoU thresholds
IoU / Dice	Segmentation	Predicted mask vs. ground-truth mask	Requires masks; sensitive for small regions
Recall@K / MRR	Retrieval	Ranked output vs. relevant item(s)	Depends on candidate set and relevance labels

TABLE XI: Summary of downstream visual task utility metrics

Metric	Evaluation level	Higher is better?	Main limitation
FID	Dataset-level distribution similarity	No	Not sample-specific; sensitive to features/dataset size
KID	Dataset-level distribution similarity	No	Still not sample-specific semantic preservation
Inception Score	Dataset-level realism/diversity	Yes	Does not compare to real images directly

TABLE XII: Summary of distribution-level realism metrics

KPI	Formula idea	Best suited for	Main limitation
Visual semantic distortion	Expected distance between $Z$ and $\hat{Z}$	Feature, object, relation, or mask representations	Requires defining semantic distance
Semantic outage probability	Probability semantic quality falls below threshold	Wireless reliability analysis	Threshold selection is subjective
Task outage	Probability task KPI falls below threshold	Goal-oriented image SemCom	Application-specific threshold
Rate/SNR-semantic curves	Semantic quality vs. resource/channel condition	Resource-aware comparison	Requires standardized quality and cost definitions

TABLE XIII: Summary of channel-aware semantic reliability metrics for image semantic communication

Evaluation perspective	Text-based KPIs	Image-based KPIs	Shared evaluation idea
Low-level reconstruction	BLEU, ROUGE, chrF	MSE, PSNR	Measures surface-level or pixel-level similarity between the source and receiver output
Structural/perceptual quality	METEOR, BLEURT, COMET	SSIM, MS-SSIM, LPIPS	Evaluates quality in a way that is more aligned with human judgment than exact symbol or pixel matching
Representation-level semantic similarity	BERT cosine similarity, BERTScore, sentence embeddings	Feature cosine similarity, CLIP similarity, deep visual embeddings	Compares source and received content in a learned semantic feature space
Downstream task utility	Intent accuracy, slot F1, frame accuracy, exact match, Recall@K, MRR	Classification accuracy, mAP, IoU, Dice, Recall@K	Measures whether the receiver can complete the intended task after transmission
Reference-free evaluation	COMET-QE, anchor similarity, semantic success/outage, perplexity	No-reference image quality assessment, CLIP-based consistency, semantic anchors, task confidence	Estimates receiver-side quality when the original source or gold reference is unavailable
Telecom-style semantic reliability	Semantic distortion, semantic error probability, semantic outage, rate-semantic tradeoff	Semantic distortion, task outage, semantic outage versus SNR/rate/bandwidth	Measures whether meaning is preserved reliably under channel and resource constraints

TABLE XIV: Cross-modality comparison of text and image semantic communication KPIs

Fig. 2: The seven functional roles of KPIs in conventional cellular networks network performance monitoring, QoS assurance, QoE bridging, radio resource optimization, network planning and deployment, fault detection and troubleshooting, and standardization/benchmarking, each of which semantic communication KPIs must eventually serve if semantic systems are to be operationally deployable.

KPI	Definition	Relevant Standard / Reference
Throughput	Useful data delivered per unit time (bps)	ITU-R M.2150
Spectral Efficiency	Throughput per unit bandwidth (bps/Hz)	3GPP TR 36.942
Peak Data Rate	Maximum achievable single-user rate	ITU-R M.2150: DL 20 Gbps
End-to-End Latency	One-way user-plane delay	ITU-R M.2150: $<1$ ms (URLLC)
Packet Loss Rate	Fraction of transmitted packets not received	RFC 7679
SINR	Received signal power vs. interference plus noise	3GPP TS 38.215

TABLE XV: Canonical KPIs in Conventional Cellular Communication Systems

Fig. 3: Illustration of semantic drift: a semantic encoder trained on one deployment context (e.g., urban traffic imagery) produces a compressed representation that is decoded correctly at the receiver only as long as sender and receiver knowledge bases remain aligned. When the operating environment or knowledge base shifts after deployment (e.g., a snowy rural scene), the same representation no longer maps to the same meaning, causing the semantic KPI to degrade even though the channel and encoding pipeline are unchanged.

Fig. 4: Cross-layer entanglement of KPIs in semantic communication. Unlike conventional systems, where physical-, MAC-, and application-layer KPIs are related through well-characterized mappings, semantic communication tightly couples the physical layer, MAC scheduling, AI inference, knowledge bases, and edge/cloud intelligence, so that a lower physical-layer throughput or higher bit error rate can still yield higher end-to-end semantic utility

Fig. 5: Explainability gap in semantic KPI failures: when semantic quality degrades, the underlying cause (channel impairment, knowledge-base mismatch, model hallucination, embedding drift, or train/deployment context mismatch) is obscured by the black-box nature of the neural encoders and decoders, making root-cause diagnosis substantially harder than for a conventional link-layer fault.

Gap	Traditional Function(s) Undermined
A. No universal semantic success criterion	Standardization and benchmarking
B. Difficulty quantifying semantics numerically	Standardization and benchmarking
C. Semantic drift	Network performance monitoring
D. Cross-layer KPI entanglement	Radio resource optimization; network performance monitoring
E. Real-time computability	Network performance monitoring
F. Multi-user semantic fairness	Radio resource optimization

TABLE XVI: Open Gaps in Semantic Communication KPI Design and the Traditional KPI Functions They Undermine

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Source sentence $S$	“The weather is cold today.”
Meaning-preserving reconstruction $\hat{S}_{1}$	“It is freezing today.”
Unrelated reconstruction $\hat{S}_{2}$	“A dog is sleeping on the sofa.”

Appendix

TABLE XVII: Example sentences for reference-free text semantic communication KPIs

Item	Value
Reconstruction $\hat{S}_{1}$	“It is freezing today.”
Predicted QE score	0.86
Reconstruction $\hat{S}_{2}$	“A dog is sleeping on the sofa.”
Predicted QE score	0.22

Appendix

TABLE XVIII: Illustrative quality estimation scores

Item	Value
$\mathrm{Sim}(S,\hat{S}_{1})$	0.995
$\mathrm{Sim}(S,\hat{S}_{2})$	0.49
Threshold $\tau$	0.80

Appendix

TABLE XIX: Illustrative semantic similarity values for semantic success and outage

Item	Value
$\hat{S}_{1}$	“It is freezing today.”
$\mathrm{PPL}(\hat{S}_{1})$	18
$\hat{S}_{2}$	“A dog is sleeping on the sofa.”
$\mathrm{PPL}(\hat{S}_{2})$	22
Garbled output	“Weather sofa tomorrow cold dog is.”
$\mathrm{PPL}(\mathrm{garbled\ output})$	120

Appendix

TABLE XX: Illustrative perplexity values

Item	Value
Source sentence $S$	“Book a flight to Montreal tomorrow.”
Semantic state $W$	intent = BookFlight; destination = Montreal; date = tomorrow
Perfect recovery $\hat{W}_{1}$	intent = BookFlight; destination = Montreal; date = tomorrow
Partial recovery $\hat{W}_{2}$	intent = BookFlight; destination = Montreal; date = missing
Bad recovery $\hat{W}_{3}$	intent = CancelBooking; destination = Toronto; date = tomorrow

Appendix

TABLE XXI: Example source sentence and semantic states

Component	Weight	Reason
intent	0.5	Wrong intent changes the main task.
destination	0.25	Destination is a required slot.
date	0.25	Date is a required slot.

Appendix

TABLE XXII: Component weights for semantic distortion

Recovered state	Mismatched components	Distortion	Interpretation
$\hat{W}_{1}$	none	0	Perfect semantic recovery
$\hat{W}_{2}$	date	0.25	Partial semantic recovery
$\hat{W}_{3}$	intent, destination	0.75	Severe semantic mismatch

Appendix

TABLE XXIII: Semantic distortion examples

Sample	Original $W$	Recovered $\hat{W}$	Error?
1	BookFlight, Montreal, tomorrow	BookFlight, Montreal, tomorrow	No
2	BookFlight, Montreal, tomorrow	BookFlight, Montreal, missing	Yes
3	CancelBooking, Toronto, today	CancelBooking, Toronto, today	No
4	PayBill, Hydro, Friday	PayBill, Hydro, Friday	No
5	BookHotel, Montreal, tonight	BookFlight, Montreal, tonight	Yes

Appendix

TABLE XXIV: Semantic error probability example

Sample	Semantic similarity $q(W,\hat{W})$	Outage?
1	0.95	No
2	0.83	No
3	0.76	Yes
4	0.91	No
5	0.60	Yes

Appendix

TABLE XXV: Semantic outage probability example

Configuration	Semantic symbols sent	Resource cost	Semantic similarity
A	20	low	0.70
B	40	medium	0.86
C	80	high	0.93

Appendix

TABLE XXVI: Rate-semantic tradeoff example

Symbol	Meaning	Role in the experiment
$I$	Original COCO validation image 000000169996	Reference image used for pairwise KPI computation
$\hat{I}_{1}$	Mild degradation	Simulates a high-quality receiver-side output
$\hat{I}_{2}$	Medium degradation	Simulates moderate channel/compression degradation
$\hat{I}_{3}$	Severe degradation	Simulates strong channel/compression degradation
$\hat{I}_{4}$	Very severe degradation	Simulates near-failure visual reconstruction
$\hat{I}_{5}$	Unrelated image	Represents a semantic failure case rather than a degraded version of $I$

Appendix

TABLE XXVII: Experimental setup for image-based semantic communication KPI examples

Output	Degradation level	Typical operations	Expected visual effect
$\hat{I}_{1}$	Mild	Light compression/noise/blur	Image remains visually close to the original
$\hat{I}_{2}$	Medium	Moderate compression/noise/blur	Some details and local structures are degraded
$\hat{I}_{3}$	Severe	Strong compression/noise/blur	Edges, textures, and small objects are strongly affected
$\hat{I}_{4}$	Very severe	Heavy degradation and/or strong downsample-upsample	Image quality is poor and some visual meaning may be ambiguous
$\hat{I}_{5}$	Unrelated	Different COCO image resized to the same size	Scene content is different from the original image

Appendix

TABLE XXVIII: Receiver-side image generation settings

Fig. 6: The panel shows the original image, mild, medium, severe, and very severe receiver-side outputs, and an unrelated semantic-failure image.

KPI family	Metrics	What is compared?	How to interpret the score
Pixel fidelity	MSE, PSNR	Pixel values of $I$ and $\hat{I}_{k}$	Lower MSE and higher PSNR indicate closer numerical reconstruction.
Structural/perceptual quality	SSIM, MS-SSIM, LPIPS	Local structure or learned perceptual features of $I$ and $\hat{I}_{k}$	Higher SSIM/MS-SSIM and lower LPIPS indicate better perceived visual quality.
Visual semantic similarity	CLIP image-image cosine similarity	CLIP image embeddings of $I$ and $\hat{I}_{k}$	Higher cosine similarity indicates stronger representation-level semantic preservation.
Channel-aware semantic reliability	Semantic success/outage	A selected quality score compared with a threshold	An output is successful if the score satisfies the chosen threshold; otherwise it is in semantic outage.
Downstream task utility	Accuracy, mAP, IoU, Dice, Recall@K	Task prediction $\hat{Y}$ compared with task ground truth $Y$	Requires task labels or a downstream model; not computed in the basic single-image experiment unless added.

Appendix

TABLE XXIX: Computation and interpretation of image-based semantic communication KPIs

Metric	Implementation detail to report	Reason
MSE / PSNR	Pixel range and image normalization	PSNR depends on the maximum pixel value.
SSIM	Window size, channel handling, and implementation/library	SSIM can vary with local window settings.
MS-SSIM	Number of scales and implementation/library	Multi-scale weighting affects the score.
LPIPS	Backbone network, such as AlexNet, VGG, or SqueezeNet	LPIPS is model-dependent.
CLIP similarity	CLIP model name and preprocessing	CLIP embeddings depend on the selected checkpoint.
Semantic outage	Selected metric and threshold	Outage results depend directly on threshold choice.

Appendix

TABLE XXX: Implementation details to report for image-based KPI computation

Output	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MS-SSIM $\uparrow$	LPIPS $\downarrow$	CLIP cosine $\uparrow$	Interpretation
$\hat{I}_{1}$ mild	0.0034	24.6771	0.8138	0.9719	0.1701	0.9653	Visually close to the original
$\hat{I}_{2}$ medium	0.0072	21.3859	0.6182	0.9165	0.4732	0.8787	Moderate degradation
$\hat{I}_{3}$ severe	0.0105	19.7549	0.4394	0.8222	0.6322	0.8108	Strong visual degradation
$\hat{I}_{4}$ very severe	0.0140	18.5173	0.3109	0.6938	0.6854	0.7045	Near-failure reconstruction
$\hat{I}_{5}$ unrelated	0.0865	10.6273	0.2141	0.1346	0.8067	0.6691	Different semantic content

Appendix

TABLE XXXI: Sample-level KPI values for controlled image degradation. Each row compares the original image

I

with one receiver-side output

\hat{I}_{k}

Fig. 7: KPI trends for the controlled image degradation experiment. Pixel, structural, perceptual, and semantic metrics are plotted across mild, medium, severe, very severe, and unrelated receiver-side outputs. Higher values indicate better quality for PSNR, SSIM, MS-SSIM, and CLIP cosine similarity, while lower values indicate better quality for LPIPS. The unrelated image acts as a semantic-failure reference.

Output	CLIP similarity	Threshold $\tau$	Semantic success?	Outage?
$\hat{I}_{1}$ mild	0.965343	0.80	Yes	No
$\hat{I}_{2}$ medium	0.878795	0.80	Yes	No
$\hat{I}_{3}$ severe	0.810859	0.80	Yes	No
$\hat{I}_{4}$ very severe	0.704589	0.80	No	Yes
$\hat{I}_{5}$ unrelated	0.669182	0.80	No	Yes

Appendix

TABLE XXXII: Semantic success and outage example using CLIP image-image similarity with threshold

\tau=0.80

Semantic Communication

Semantic communication sends meaning- or task-relevant information so receivers can act without reconstructing every transmitted bit or signal.

See all 103 papers

May 14, 2026

Not All Symbols Are Equal: Importance-Aware Constellation Design for Semantic Communication

Semantic communication systems for goal-oriented transmission must protect task-relevant information not only through source compression but also via physical layer mapping. Existing approaches decouple constellation design and semantic encoding, exposing critical symbols to channel errors at the same rate as irrelevant ones. Contrary to this, in this paper, a joint semantic-physical layer framework is proposed, which is composed of a vector quantized-variational autoencoder that extracts discrete latent concepts, a semantic criticality indicator (SCI) that scores each concept by task relevance, and a deep reinforcement learning agent that dynamically selects the transmission subset based on instantaneous channel conditions. At the physical layer, a learned semantic-aware M -QAM constellation assigns symbol positions according to joint co-occurrence statistics and SCI scores, departing from the uniform spacing and Gray coding of standard M -QAM which minimizes average BER without regard for semantic content. We introduce a novel semantic symbol vulnerability (SSV) metric and a semantic protection probability (SPP) to quantify the exposure of task-critical symbols to decoding errors, and prove that any Gray-coded constellation is strictly suboptimal in SCI-Weighted SSV whenever the source exhibits non-uniform semantic importance and co-occurrence statistics. Simulation results demonstrate that the proposed constellation achieves near 100% SPP across modulation orders from 4-QAM to 1024-QAM versus 50% for standard constellations at high spectral efficiency, a 21:1 compression ratio with semantic quality above 0.9, generalizing across MNIST, Fashion-MNIST, and FSDD without modification.

Albert Shaju, Christo Kurisummoottil Thomas, Mayukh Roy Chowdhury

Department of Electrical and Computer Engineering, Worcester Polytechnic Institute, Worcester, MA, USA. · Nokia Bell Labs, Bengaluru, India.

Semantic Communication

Ask PDF

Feb 28, 2026

Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

This position paper argues that the evaluation of modern visual processing systems should no longer be driven primarily by single-metric image quality assessment benchmarks, particularly in the era of generative and perception-oriented methods. Image restoration exemplifies this divergence: while objective IQA metrics enable reproducible, scalable evaluation, they have increasingly drifted apart from human perception and user preferences. We contend that this mismatch risks constraining innovation and misguiding research progress across visual processing tasks. Rather than rejecting metrics altogether, this paper calls for a rebalancing of evaluation paradigms, advocating a more human-centered, context-aware, and fine-grained approach to assessing the visual models' outcomes.

Jinfan Hu, Fanghua Yu, Zhiyuan You +5

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of the Chinese Academy of Sciences · The University of Hong Kong +6

Image Quality Assessment

Ask PDF

Jun 5, 2026

Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation

Evaluating the quality of automatically generated keyphrases remains a complex challenge. Traditional metrics either rely on exact lexical matching or consider semantic similarity while ignoring prediction ranking, both of which misalign with how humans judge informativeness and relevance. We introduce Semantic R-Precision (SemR-p), a novel evaluation metric that integrates semantic similarity into the rank-aware R-Precision framework. Designed from a human-centric perspective and inspired by Information Retrieval metrics, SemR-p rewards semantically relevant keyphrases that appear early in the output list. We conducted extensive analyses to assess its semantic sensitivity, ranking awareness, and discriminative power across models and datasets. The results suggest that SemR-p offers a complementary lens for evaluating keyphrase predictions, helping to better reflect user-centred notions of relevance alongside traditional lexical and semantic matching metrics.

Shamira Venturini, Steffen Kinkel

ILIN - Institute for Learning and Innovation in Networks, Karlsruhe University of Applied Sciences, Moltkestraße 30, 76133 Karlsruhe, Germany · Karlsruhe Institute of Technology, Kaiserstraße 12, 76131 Karlsruhe, Germany

Automated Evaluation IR Evaluation

Ask PDF

Abstract

Semantic Communication

Explore similar work

Not All Symbols Are Equal: Importance-Aware Constellation Design for Semantic Communication

Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation