A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction

Akshay Vyas; Angelo Pimienta; Nicholas Ruozzi

cs.LGOct 29, 2026

A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction

Authors: Akshay Vyas, Angelo Pimienta, Nicholas Ruozzi

Organizations: The University of Texas at Dallas

Abstract

In multi-output structured prediction tasks, while only one ground truth label may be provided in the training data, multiple equally valid outputs may be possible, making reliable evaluation a persistent challenge. We postulate that human evaluators implicitly use task-specific invariants, e.g., object boundaries in colorized images or named entities in translations, to judge if an output is acceptable. Under this assumption, we introduce a notion of approximate task-specific invariants and use them as diagnostic tools to evaluate a variety of existing metrics for vision and language tasks. We use these task invariants as part of a framework to systematically test metric reliability by encouraging domain-relevant invariants in model outputs via an augmented loss function. In our experiments, we observe that enforcing invariants with an augmented loss yields substantial improvements in popular distributional metrics while more traditional metrics change only marginally. Through this invariants-driven evaluation, we expose where standard metrics fail to detect meaningful differences, and we highlight the conditions under which distributional metrics succeed or still fall short.

Figures & tables

Main Task	Invariant Task(s)
Image Colorization	Image Segmentation, Depth Estimation, Edge Detection
Machine Translation (NMT)	Named Entity Recognition (NER), Sentiment Analysis, Paraphrase Detection
Image Captioning	Semantic Role Labeling (SRL), Scene Graph Consistency, Named Entity Recognition (NER)
Text Summarization	Named Entity Recognition (NER), Sentiment Analysis, Coreference Resolution
Video Prediction	Optical Flow Estimation, Object Tracking, Action Recognition
Natural Language Inference	Paraphrase detection, Sentiment Analysis

Table 1: Tasks and corresponding invariants. BOLD tasks are considered in this work.

Panel of Figure 1 — Figure 1: Effectiveness of invariant-based regularization in improving image colorization. Left panel: Comparison using segmentation invariant. Right panel: Comparison using depth invariant. Each panel shows the colorized outputs (top row) and corresponding invariant masks (bottom row). Invariant-regularized models significantly preserve critical structural details, such as object boundaries and spatial depth cues, compared to baseline models that exhibit noticeable artifacts and inaccuracies.

Task	Image Colorization	German to English Neural Machine Translation	Image Captioning
Base Networks	ChromaGAN ( Vitoria, Raad, and Ballester 2020 ) , ColorFormer ( Ji et al. 2022 ) , and DDColor ( Kang et al. 2023 )	BiBERT ( Xu, Van Durme, and Murray 2021 ) and Bi-SumCut ( Gao et al. 2022 )	SmallCap ( Ramos et al. 2023 ) and ClipCap ( Mokady, Hertz, and Bermano 2021 )
Loss Function ( $L_{T}$ )	Pixel‐reconstruction MSE, L1-loss, Perceptual-loss, GAN-loss, Colorfulness-loss	Label Smoothed Cross Entropy: Cross-entropy loss measuring predicted vs target word probability differences with label smoothing for better generalization	Cross Entropy Loss: Token-level classification loss between predicted and ground truth caption sequences
Invariant Networks	Depth: MiDaS ( Ranftl et al. 2020 ) Segmentation: Side Adapter Network (SAN) ( Xu et al. 2023 )	NER: Hugging Face’s ( Wolf et al. 2019 ) Sentiment: Hugging Face’s distilbert-base- uncased-finetuned-sst-2-english	NER: Hugging Face’s ( Wolf et al. 2019 ) Scenegraph: sngparser ( Wu et al. 2019 ) ( Schuster et al. 2015 )


Invariant Loss ( $L_{I}$ )	SegmentationSANLoss: Dice loss between predicted vs GT SAN segmentation masks prevents color bleeding DepthMidasLoss: SILog loss between predicted vs GT MiDaS depth maps preserves spatial structure	NER Loss: KL divergence between predicted vs GT BERT-NER token classifications for entity preservation and Sentiment Loss: KL divergence between predicted vs GT DistilBERT sentiment logits	NER Loss: KL divergence between predicted vs GT BERT-NER token classifications for entity preservation. Scene Graph Loss: Symmetric difference ratio of entities and relations between predicted vs GT scene graphs

Table 2: Experimental configurations for the three selected tasks.

Panel of Figure 2 — Figure 2: Performance improvement of architectures trained with invariants versus training without invariants, evaluated across common metrics for three tasks. (a) Image Colorization: Base models are ChromaGAN (CG), ColorFormer (CF), and DDColor (DD), each augmented with Segmentation (+S), Depth (+D), or both (+SD). (b) Machine Translation: Base models are Bi-SimCut (BS) and BiBERT (BB), augmented with NER (+N), SeAM (+Se), or both (+NS). (c) Image Captioning: Base models are SmallCap (SC) and ClipCap (CC), augmented with NER (+N), SceneGraph (+SG), or both (+NS). Distributional metrics (FDD, FBD, FID, CMMD) show larger improvements than traditional metrics (PSNR, BLEU-4, METEOR), indicating higher sensitivity to invariant-based training.

Method	#P	ImageNet (val50k)						COCO-Stuff(val5k)
Method	#P	FDD $\downarrow$	FID $\downarrow$	CMMD $\downarrow$	LPIPS $\downarrow$	CF $\uparrow$	PSNR $\uparrow$	FDD $\downarrow$	FID $\downarrow$	CMMD $\downarrow$	LPIPS $\downarrow$	CF $\uparrow$	PSNR $\uparrow$
ChromaGAN	173.9M	42.36	5.81	0.597	0.236	27.41	23.65	94.90	16.73	0.715	0.233	27.89	23.66
+Seg	173.9M	29.38	5.44	0.679	0.236	29.36	23.76	81.07	15.50	0.603	0.233	29.56	23.82
+Depth	173.9M	28.55	5.27	0.565	0.231	28.73	23.83	79.28	15.29	0.659	0.229	28.95	23.91
+Seg+Depth	173.9M	28.86	5.06	0.599	0.231	28.69	23.83	80.00	14.58	0.597	0.229	29.99	23.82
ColorFormer	44.8M	21.16	2.00	0.318	0.137	27.57	24.91	59.38	9.03	0.377	0.134	28.30	24.70

Table 3: Quantitative comparison of image colorization methods.

\uparrow

(

\downarrow

) indicates higher (lower) values are better; Bold values in FDD column of ImageNet indicate the model selection criterion. The Seg architecture is SAN, and depth architecture is Midas.

Model	#P	COCO-Stuff (val5k)						Flickr30k (val1k)
Model	#P	FBD-4 $\downarrow$	FBD $\downarrow$	BERTScore $\uparrow$	CIDEr $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	FBD-4 $\downarrow$	FBD $\downarrow$	BERTScore $\uparrow$	CIDEr $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$
SmallCap	7M	62.60	41.90	0.9224	118.49	36.68	27.90	58.45	39.12	0.9189	112.30	34.82	26.45
+NER	7M	58.82	41.89	0.9223	116.29	36.07	27.91	55.20	39.08	0.9187	110.15	34.25	26.48
+SceneGraph	7M	54.12	41.90	0.9218	115.47	35.68	27.76	51.85	39.15	0.9182	109.42	33.89	26.32
+NER+SceneGraph	7M	56.52	41.89	0.9226	117.48	36.52	27.95	53.78	39.11	0.9191	111.68	34.71	26.51
ClipCap	43M	48.20	38.45	0.9301	125.80	39.15	29.20	44.60	36.25	0.9267	119.45	37.28	27.85

Table 4: Performance on image captioning benchmarks. Bold values in FBD-4 column of COCO-Stuff indicate the model selection criterion.

Model	#P	IWSLT-14						WMT-14
Model	#P	FBD $\downarrow$	FBD-4 $\downarrow$	BERTScore $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	COMET $\uparrow$	FBD $\downarrow$	FBD-4 $\downarrow$	BERTScore $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	COMET $\uparrow$
Bi-SimCut	65M	9.87	102.15	0.8871	37.83	0.6412	0.5743	15.24	135.42	0.8952	34.47	0.5836	0.6124
+NER	65M	8.42	101.89	0.8867	36.95	0.6358	0.5739	14.18	134.76	0.8948	33.82	0.5794	0.6119
+SeAM	65M	8.51	103.14	0.8868	37.02	0.6361	0.5737	14.26	136.23	0.8949	33.89	0.5798	0.6117
+NER+SeAM	65M	8.38	102.67	0.8869	37.18	0.6373	0.5741	14.09	135.58	0.8950	34.06	0.5809	0.6121
BiBERT	110M	8.99	100.24	0.8930	38.20	0.6597	0.5899	13.76	133.18	0.9012	35.24	0.6023	0.6287

Table 5: Performance on machine translation. Bold values indicate the model selection criterion.

Figure 3: Image captions produced by models trained with/without scenegraph and NER invariants for an image of a cat (left).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Effectiveness of invariant-based regularization in improving image colorization. Left panel : Comparison using segmentation invariant. Right panel : Comparison using depth invariant. Each panel shows the colorized outputs (top row) and corresponding invariant masks (bottom row). Invariant-regularized models significantly preserve critical structural details, such as object boundaries and spatial depth cues, compared to baseline models that exhibit noticeable artifacts and inaccuracies.

Panel of Figure 5 — Figure 5: Effectiveness of invariant-based regularization in improving image colorization. Left panel : Comparison using segmentation invariant. Right panel : Comparison using depth invariant. Each panel shows the colorized outputs (top row) and corresponding invariant masks (bottom row). Invariant-regularized models significantly preserve critical structural details, such as object boundaries and spatial depth cues, compared to baseline models that exhibit noticeable artifacts and inaccuracies.

Example 1
Source Sentence:	“heinrich , auf der anderen seite , wird zum geburtstermin geboren , er ist also am 1 marz 1980 geboren ,”
Target Sentence:	“henry , by contrast , is born at term , so he ’ s born on march 1 , 1980 .”
Target Logits:	0.99992 0.00001 0.00000 0.00003 0.00001 0.00001 0.00001 0.00001 0.00001 O B-MIS I-MIS B-PER I-PER B-ORG I-ORG B-LOC I-LOC
BiBERT+NER Pred Sentence:	“henry , on the other hand , is born at the time of birth , so he was born on march 1 , 1980 .”
BiBERT+NER Logits:	0.99993 0.00001 0.00000 0.00002 0.00001 0.00001 0.00001 0.00001 0.00001 O B-MIS I-MIS B-PER I-PER B-ORG I-ORG B-LOC I-LOC
BiBERT Pred Sentence:	“and tom , on the other hand , is born at the time of birth , so he was born on march 1 , 1980 .”

Appendix

Table 6: Comparative analysis of Machine Translation Models with and without Named Entity Recognition (NER) Invariant Regularization. Each example shows the original German source sentence, the target English translation, and predicted translations from two models: BiBERT+NER (with NER regularization) and BiBERT (baseline). The tables include detailed logit distributions across named entity categories. The results illustrate how the NER regularization significantly preserves the correct named entities and their classifications, whereas the baseline model frequently omits or misclassifies named entities, highlighting the practical advantages of invariant-based regularization.

Figure 6: Comparison of image captions produced by models trained with and without scenegraph and NER invariants for an image of a person skiing (left).

Figure 7: Comparison of image captions produced by models trained with and without scenegraph and NER invariants for an image of a cat with a shoe (left).

Figure 8: Comparison of image captions produced by models trained with and without scenegraph and NER invariants for an image of a sheep in a field (left).

Model	Dataset	Best $\alpha$	Batch Size	Epochs / Iterations	Hyperparameters Tried ( $\alpha$ )
ChromaGAN	ImageNet	-	10	5 epochs	-
ChromaGAN + Seg	ImageNet	0.02	10	15 epochs	0.0002, 0.002, 0.02, 0.05, 0.2
ChromaGAN + Depth	ImageNet	0.0001	10	15 epochs	0.0001, 0.001, 0.005, 0.01
ChromaGAN + Seg + Depth	ImageNet	0.02,0.0001	10	15 epochs	(0.02,0.0001),(0.02,0.005),(0.01,0.001)
ColorFormer	ImageNet	-	16	200,000 iterations	-
ColorFormer + Seg	ImageNet	0.05	16	600,000 iterations	0.0001, 0.001, 0.01, 0.05, 0.1, 0.2

Appendix

Table 7: Hyperparameter

\alpha

Settings for Image Colorization Experiments.

\alpha

represents the multiplier to invariant loss added to total training loss during colorization training.

Model	Dataset	Best $\alpha$	Batch Size	Epochs	Hyperparameters Tried ( $\alpha$ )
BiBERT	IWSLT-14	-	4096 tokens	75	-
BiBERT + NER	IWSLT-14	100	4096 tokens	75	10, 50, 100, 500, 1000
BiBERT + SeAM	IWSLT-14	50	4096 tokens	75	10, 50, 100, 500, 1000
BiBERT + NER + SeAM	IWSLT-14	100, 50	4096 tokens	75	(100,50), (50,50), (100,100)
Bi-SimCut	IWSLT-14	-	4096 tokens	75	-
Bi-SimCut + NER	IWSLT-14	200	4096 tokens	75	50, 100, 200, 500, 1000

Appendix

Table 8: Hyperparameter

\alpha

Settings for Neural Machine Translation Experiments

Model	Dataset	Best $\alpha$	Batch Size	Epochs	Hyperparameters Tried ( $\alpha$ )
SmallCap	COCO	-	32	10	-
SmallCap + NER	COCO	0.5	32	10	0.1, 0.3, 0.5, 0.7, 1.0
SmallCap + SceneGraph	COCO	1.0	32	10	0.5, 1.0, 2.0, 5.0
SmallCap + NER + SG	COCO	0.5, 1.0	32	10	(0.5,1.0), (0.3,1.0), (0.5,2.0)
ClipCap	COCO	-	64	10	-
ClipCap + NER	COCO	0.3	64	10	0.1, 0.3, 0.5, 0.7, 1.0

Appendix

Table 9: Hyperparameter

\alpha

Settings for Image Captioning Experiments

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 398 papers

Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
Aug 27, 2026Mingqi Gao, Anthony Sicilia, Weiyan ShiAutomated Evaluation MT Evaluation
Northeastern University · West Virginia University
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
I-SAFE: Wasserstein Coherence Metrics for Structural Auditing of Scientific AI Models
May 20, 2026Barbara Tarantino, Gennaro Auricchio, Paolo GiudiciAlgorithmic Auditing Black-Box Model Auditing
Department of Economics, University of Pavia, Via S. Felice Al Monastero, 5, Pavia, 27100, Italy · Department of Mathematics, University of Padua, Via Trieste, 63, Padua, 35131, Italy

Aug 27, 2026J/K move · Enter open · S save

Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation

Mingqi Gao, Anthony Sicilia, Weiyan Shi

Northeastern University · West Virginia University

Automated Evaluation MT Evaluation Prediction-Powered Inference

Open paper AskPDF arXiv

Across various non-verifiable tasks, human evaluation is reliable but expensive, while automatic metrics are more scalable but often biased. Building on prediction-powered inference (PPI), we propose prediction-powered evaluation, a framework that combines limited human judgments with large-scale automatic scores to obtain data-efficient system comparisons that are provably unbiased. We develop parametric and non-parametric procedures, analyze the efficiency trade-off between paired and unpaired designs, and validate the framework on six WMT datasets. We further introduce the Prediction-Powered Saving Ratio (PPSR), a meta-metric that measures how much human annotation an automatic metric can save when used within prediction-powered evaluation. PPSR directly targets metric utility for prediction-powered evaluation and yields more discriminative and stable metric rankings than existing system-level meta-metrics. Overall, our new paradigm reframes automatic metrics as tools for reducing human annotation cost rather than replacing human judgment, and applies broadly to non-verifiable tasks.

Abstract

Automated Evaluation

Explore similar work

Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation

VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation

I-SAFE: Wasserstein Coherence Metrics for Structural Auditing of Scientific AI Models