Towards Universal Calibration for In-Context Classification

Awik Dhar; Sunita Sarawagi; Arpit Agarwal

cs.LGOct 12, 2026

Towards Universal Calibration for In-Context Classification

Authors: Awik Dhar, Sunita Sarawagi, Arpit Agarwal

Organizations: Indian Institute of Technology Bombay

Abstract

In-Context Learning (ICL) has received a lot of interest recently due to its ability to improve classification accuracy for new tasks without having to fine-tune the model over these tasks. However, the impact of ICL on calibration has been far less studied. Recent work has shown that ICL can have a negative impact on calibration due to biased selection of in-context examples. The existing methods for improving ICL calibration require additional task and model-specific training, detracting from the no-fine-tuning convenience of ICL. In this work, we introduce CALIBFORMER, a universal ICL calibrator for classification tasks, that generalizes to new models and tasks without further fine-tuning on these tasks. Specifically, we design a transformer-based calibrator which uses the model predictions on in-context examples to predict a confidence score for the test input. We show that the calibrator trained on three models and ten tasks, is able to generalize/calibrate ICL predictions on several new tasks and four LLMs unseen during training. Even for frontier models like Gemini, CALIBFORMER is able to obtain over 50% reduction in calibration error.

Figures & tables

Panel of Figure 1 — Figure 1: Accuracy and ECE with varying number of shots. While accuracy typically increases with the number of shots, the model does not significantly improve in aligning its confidence with true likelihood of correctness. Sometimes unpredictable trends are observed - wikitoxic ECE in (a) is turbulent at the end; when2call in (b) experiences worsening ECE despite stable accuracy

Panel of Figure 2 — Figure 2: UMAP projections of embeddings derived from prediction trajectories of different LLM-dataset combinations for Phi-4-mini-instruct. 3 LLMs and 10 datasets achieves adequate coverage for generalizable calibration.

	Phi-4-mini-instruct						Qwen-3.5-9B				Gemma-4-E4B-IT
Dataset	Uncalib	ICC	ICP	TS-AR	TS-FS	CF(Ours)	Uncalib	TS-AR	TS-FS	CF(Ours)	Uncalib	TS-AR	TS-FS	CF(Ours)
Random Sampling
Amazon_CF	15.3	20.6	8.4	7.8	7.6	4.7	4.9	7.0	7.2	5.4	14.2	5.8	5.7	3.4
CS_QA	21.8	23.7	19.3	7.9	7.4	5.0	6.6	6.1	6.3	5.0	20.7	7.3	7.1	4.9
massive_intent	21.2	28.9	15.3	8.8	8.9	4.9	10.7	7.0	7.1	4.5	17.2	8.0	7.4	4.1
medqa	21.8	23.9	21.2	8.6	8.8	6.6	6.0	6.7	6.6	5.0	28.8	7.2	7.2	7.6

Table 1: Across all three open weights LLMs, Calibformer outperforms all baselines in 85% of the cases

	Uncalib		TS-AR		TS-FS		Ours
Datasets	R	S	R	S	R	S	R	S
Amazon_CF	7.8	9.3	4.9	4.7	5.6	5.2	4.5	3.2
CS_QA	11.9	11.4	8.3	8.2	7.8	7.7	4.5	4.3
medqa	11.3	9.0	12.3	12.7	6.2	7.0	4.1	4.4
when2call	17.7	11.0	7.4	6.3	7.1	6.5	4.7	3.9
Average	12.2	10.2	8.2	8.0	6.7	6.6	4.5	4.0

Table 2: Comparison of ECE calibration on the Gemini-2.5-flash LLM. Values are reported for Random (R) and Similarity (S) sampled contexts. Calibformer reduces the uncalibrated ECE by over 50%.

	Uncalib		TS-AR		TS-FS		Ours
Sampling	ECE	Brier	ECE	Brier	ECE	Brier	ECE	Brier
Similarity	18.3	.480	11.3	.455	11.3	.456	6.6	.433
Random	25.8	.661	10.9	.591	11.0	.590	7.7	.570

Table 3: Calibformer is robust to noisy probabilities on Phi-4-mini-instruct

Panel of Figure 3 — Figure 3: For Phi-4-mini-instruct, calibration error tends to on average decrease with shots for all the methods, with Calibformer consistently outperforming all baselines.

Method	ECE (%)( $\downarrow$ )	Brier ( $\downarrow$ )
Uncalibrated	18.12	0.4320
Base( Calibformer )	5.41	0.3807
Only confidence and correctness	5.47 ∗	0.3823 ∗∗∗
Half Train Data	5.83 ∗∗∗	0.3822 ∗∗∗
w/ embeddings	8.46 ∗∗∗	0.3941 ∗∗∗
w/ permuted embeddings	5.70 ∗∗∗	0.3816 ∗∗∗

Table 4: Ablations demonstrating feature set and training strategy importance. While confidence and correctness carry most of the calibration signal, the full feature set provides larger gains in lower shot and higher class count regimes. Stat. significance relative to the Base is denoted by ∗∗∗ (

p<.001

), ∗∗ (

p<.01

) and ∗ (

p<.05

Method	ECE(%)( $\downarrow$ )
Base(Transformer)	5.41
LSTM	5.45 ∗
RNN	5.51 ∗∗
MLP	5.81 ∗∗∗
Logistic Regression	5.74 ∗∗∗
Beta Calibration	8.56 ∗∗∗

Table 5: Transformer as the calibrator performs slightly better than equal resource LSTM/RNN and significantly better than simpler approaches, demonstrating efficacy of sequence modeling on in-context predictions. Statistical significance relative to the Base is denoted by ∗∗∗ (

p<.001

), ∗∗ (

p<.01

) and ∗ (

p<.05

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	NC	HF Repo / Source
Training Datasets
snli	3	stanfordnlp/snli
sst5	5	SetFit/sst5
rte	2	SetFit/rte
Trec-QC	6	SetFit/TREC-QC
agnews	4	SetFit/ag_news

Appendix

Table 6: Consolidated Datasets: Training, Validation, and Test with their number of classes (NC) and source.

Category	Hyperparameter	Value
Architecture	Embedding dim	64
	Attention heads	8
	Layers	12
	Sequence length	21
	Dropout	0.25
Optimization	Optimizer	AdamW

Appendix

Table 7: Training hyperparameters for Calibformer .

	Phi-4-mini-instruct						Qwen-3.5-9B				Gemma-4-E4B
Dataset	Uncalib	ICC	ICP	TS-AR	TS-FS	Ours	Uncalib	TS-AR	TS-FS	Ours	Uncalib	TS-AR	TS-FS	Ours
Random Sampling
Amazon_CF	.366	.481	.303	.348	.350	.319	.149	.174	.175	.158	.253	.250	.247	.219
CS_QA	.492	.531	.471	.458	.456	.439	.234	.253	.254	.234	.393	.360	.359	.344
massive_intent	.637	.766	.587	.600	.601	.592	.320	.327	.326	.306	.366	.353	.350	.333
medqa	.685	.704	.675	.642	.643	.621	.326	.353	.352	.327	.624	.532	.533	.525

Appendix

Table 8: Brier Scores for calibration methods across the open weights LLMs

	Uncalib	TS-AR	TS-FS	Ours
Datasets	R / S	R / S	R / S	R / S
Amazon_CF	.170 /.194	.181/ .189	.188/.193	.157 / .175
CS_QA	.241 / .235	.262/.251	.260/.249	.224 / .220
medqa	.226 / .196	.267/.248	.235/.213	.207 / .191
when2call	.349/.207	.333 / .198	.333 /.199	.305 / .181
Average	.246 / .208	.261/.221	.254/.214	.223 / .192

Appendix

Table 9: Comparison of Brier scores on the Gemini-2.5-flash LLM. Values are reported as Random (R) / Similarity (S) sampled contexts.

	Calibration Methods
Datasets	Uncalib	ICP	TS-AR	TS-FS	Ours
Amazon_CF	76.6	77.1	76.6	76.6	76.6
CS_QA	70.6	71.5	70.6	70.6	70.6
massive_intent	83.0	84.0	83.0	83.0	83.0
medqa	57.1	57.8	57.1	57.1	57.1
metatool	84.4	85.5	84.4	84.4	84.4

Appendix

Table 10: Accuracy comparison for calibration methods | Phi-4-mini-instruct | Similarity Sampling

	Calibration Methods
Datasets	Uncalib	ICC	ICP	TS-AR	TS-FS	Ours
Amazon_CF	78.0	69.6	79.7	78.0	78.0	78.0
CS_QA	69.7	67.7	70.1	69.7	69.7	69.7
massive_intent	58.5	51.1	60.2	58.5	58.5	58.5
medqa	52.5	51.2	52.4	52.5	52.5	52.5
metatool	61.8	55.8	63.0	61.8	61.8	61.8

Appendix

Table 11: Accuracy comparison for calibration methods | Phi-4-mini-instruct | Random Sampling

Dataset	Uncalibrated	ICC	ICP	TS-AR	TS-FS	Calibformer
Random Sampling
Amazon_CF	15.3 (1.8)	20.6 (2.2)	8.4 (1.7)	7.8 (1.6)	7.6 (1.4)	4.7 (1.0)
CS_QA	21.8 (1.8)	23.7 (2.3)	19.3 (1.6)	7.9 (1.5)	7.4 (1.5)	5.0 (1.0)
massive_intent	21.2 (2.0)	28.9 (2.5)	15.3 (1.6)	8.8 (2.0)	8.9 (2.0)	4.9 (0.7)
medqa	21.8 (1.4)	23.9 (1.9)	21.2 (1.3)	8.6 (1.6)	8.8 (1.7)	6.6 (1.2)
metatool	23.0 (2.2)	27.4 (2.1)	17.9 (2.3)	7.1 (1.4)	6.8 (1.4)	5.1 (1.0)

Appendix

Table 12: Calibration Error (ECE) for Phi-4-mini-instruct across calibration methods. Standard deviations reported in parentheses.

Dataset	Uncalibrated	ICC	ICP	TS-AR	TS-FS	Calibformer
Random Sampling
Amazon_CF	.366 (.033)	.481 (.034)	.303 (.032)	.348 (.029)	.350 (.027)	.319 (.026)
CS_QA	.492 (.028)	.531 (.034)	.471 (.025)	.458 (.021)	.456 (.021)	.439 (.020)
massive_intent	.637 (.019)	.766 (.039)	.587 (.016)	.600 (.016)	.601 (.016)	.592 (.015)
medqa	.685 (.024)	.704 (.027)	.675 (.024)	.642 (.023)	.643 (.022)	.621 (.022)
metatool	.586 (.031)	.677 (.033)	.545 (.028)	.547 (.026)	.542 (.023)	.532 (.022)

Appendix

Table 13: Brier Scores for Phi-4-mini-instruct across calibration methods. Standard deviations reported in parentheses.

Dataset	Uncalibrated	TS-AR	TS-FS	Calibformer
Similarity Sampling
Amazon_CF	4.3 (0.8)	7.3 (0.9)	7.3 (0.8)	6.3 (1.1)
CS_QA	6.9 (1.3)	6.7 (0.9)	6.7 (0.9)	4.9 (1.3)
massive_intent	7.1 (1.7)	6.8 (1.1)	7.2 (1.1)	4.8 (1.2)
medqa	4.4 (0.7)	6.3 (1.3)	6.5 (1.4)	5.6 (0.9)
metatool	7.6 (1.2)	4.4 (0.9)	4.5 (1.0)	3.8 (0.6)

Appendix

Table 14: Calibration Error (ECE) for Qwen3.5-9B across calibration methods. Standard deviations reported in parentheses.

Dataset	Uncalibrated	TS-AR	TS-FS	Calibformer
Random Sampling
Amazon_CF	.149 (.025)	.174 (.025)	.175 (.026)	.158 (.021)
CS_QA	.234 (.022)	.253 (.029)	.254 (.028)	.234 (.018)
massive_intent	.320 (.024)	.327 (.031)	.326 (.031)	.306 (.025)
medqa	.326 (.018)	.353 (.023)	.352 (.024)	.327 (.015)
metatool	.284 (.022)	.283 (.022)	.285 (.022)	.264 (.019)

Appendix

Table 15: Brier Scores for Qwen3.5-9B across calibration methods. Standard deviations reported in parentheses.

Dataset	Uncalibrated	TS-AR	TS-FS	Calibformer
Random Sampling
Amazon_CF	14.2 (1.8)	5.8 (0.8)	5.7 (1.0)	3.4 (0.7)
CS_QA	20.7 (2.5)	7.3 (1.6)	7.1 (1.4)	4.9 (1.3)
massive_intent	17.2 (2.0)	8.0 (1.3)	7.4 (0.8)	4.1 (0.5)
medqa	28.8 (3.0)	7.2 (1.4)	7.2 (1.4)	7.6 (2.4)
metatool	20.0 (1.8)	6.9 (1.4)	6.9 (1.6)	4.9 (0.9)

Appendix

Table 16: Calibration Error (ECE) for gemma-4-E4B-it across calibration methods. Standard deviations reported in parentheses.

Dataset	Uncalibrated	TS-AR	TS-FS	Calibformer
Random Sampling
Amazon_CF	.253 (.024)	.250 (.022)	.247 (.023)	.219 (.019)
CS_QA	.393 (.040)	.360 (.030)	.359 (.031)	.344 (.032)
massive_intent	.366 (.036)	.353 (.032)	.350 (.033)	.333 (.028)
medqa	.624 (.051)	.532 (.036)	.533 (.034)	.525 (.039)
metatool	.345 (.026)	.315 (.020)	.316 (.022)	.307 (.019)

Appendix

Table 17: Brier Scores for gemma-4-E4B-it across calibration methods. Standard deviations reported in parentheses.

Panel of Figure 4 — Figure 4: Calibration performance | Phi 4 mini instruct | MedQA

Panel of Figure 5 — Figure 5: Calibration performance | Phi 4 mini instruct | Wikitoxic

Panel of Figure 6 — Figure 6: Calibration performance | Phi 4 mini instruct | Wildguard

Panel of Figure 7 — Figure 7: Calibration performance | Phi 4 mini instruct | When2call

Panel of Figure 8 — Figure 8: Calibration performance | Phi 4 mini instruct | MetaTool

Panel of Figure 9 — Figure 9: Calibration performance | Phi 4 mini instruct | Massive Intent

Panel of Figure 10 — Figure 10: Calibration performance | Phi 4 mini instruct | Amazon Counterfactual(Amazon_CF)

Panel of Figure 11 — Figure 11: Calibration performance | Phi 4 mini instruct | CommonsenseQA(CS_QA)

Sampling Strategy

Prompt

Random Sampling

<|system|>You are a helpful assistant.<|end|><|user|>Your task is to answer the given medical question by specifying the correct option(A, B, C, D) Question: A 43-year-old male presents to a clinic for routine follow-up. He was diagnosed with hepatitis B several months ago. He does not have any complaints about his health, except for poor appetite. The general physical examination is normal. The laboratory investigation reveals mildly elevated aminotransferases. Which of the following findings indicate that the patient has developed a chronic form of his viral infection? A. HbsAg -, Anti-HbsAg -, Anti-HbcAg IgM +, Anti-HbcAg IgG -, HbeAg -, Anti-HbeAg + B. HbsAg +, Anti-HbsAg -, Anti-HbcAg IgM -, Anti-HbcAg IgG +, HbeAg +, Anti-HbeAg - C. HbsAg -, Anti-HbsAg +, Anti-HbcAg IgM -, Anti-HbcAg IgG -, HbeAg -, Anti-HbeAg - D. HbsAg -, Anti-HbsAg +, Anti-HbcAg IgM -, Anti-HbcAg IgG +, HbeAg -, Anti-HbeAg + <|end|><|assistant|>Answer: B <|end|><|user|>Question: A 62-year-old man presents to the physician because of incomplete healing of a chest wound. He recently had a triple coronary artery bypass graft 3 weeks ago. His past medical history is significant for type 2 diabetes mellitus and hypertension for the past 25 years. Clinical examination shows the presence of wound dehiscence in the lower 3rd of the sternal region. The wound surface shows the presence of dead necrotic tissue with pus. Computed tomography (CT) of the thorax shows a small fluid collection with fat stranding in the perisurgical soft tissues. What is the most appropriate next step in the management of the patient? A. Surgical debridement B. Negative pressure wound management C. Sternal wiring D. Sternal fixation <|end|><|assistant|>Answer:

Similarity Sampling

<|system|>You are a helpful assistant.<|end|><|user|>Your task is to answer the given medical question by specifying the correct option(A, B, C, D) Question: A 78-year-old male presents to the emergency department after passing out. His wife reports that she and the patient were walking their dog when he suddenly lost consciousness. On physical exam, he has a loud crescendo-decrescendo systolic murmur and is subsequently diagnosed with severe aortic stenosis. The patient undergoes open aortic valve replacement and has an uncomplicated postoperative course. His sternal wound drain is pulled for low output on post-operative day three. On post-operative day five, the patient complains of pain during deep inspiration and retrosternal chest pain. His temperature is 101.7°F (38.7°C), blood pressure is 125/81 mmHg, pulse is 104/min, and respirations are 18/min. On physical exam, the patient is tender to palpation around his sternal wound, and there is erythema around the incision without dehiscence. His chest radiograph shows a widened mediastinum with a small pleural effusion on the left. CT angiography shows stranding in the subcutaneous tissue and a fluid collection below the sternum. Which of the following is the best next step in management? A. Surgical repair of esophageal perforation B. Surgical repair of aortic injury C. Intravenous antibiotics and observation D. Intravenous antibiotics and debridement of surgical wound <|end|><|assistant|>Answer: D <|end|><|user|>Question: A 62-year-old man presents to the physician because of incomplete healing of a chest wound. He recently had a triple coronary artery bypass graft 3 weeks ago. His past medical history is significant for type 2 diabetes mellitus and hypertension for the past 25 years. Clinical examination shows the presence of wound dehiscence in the lower 3rd of the sternal region. The wound surface shows the presence of dead necrotic tissue with pus. Computed tomography (CT) of the thorax shows a small fluid collection with fat stranding in the perisurgical soft tissues. What is the most appropriate next step in the management of the patient? A. Surgical debridement B. Negative pressure wound management C. Sternal wiring D. Sternal fixation <|end|><|assistant|>Answer:

Appendix

Table 18: MedQA 1-shot prompt with chat template applied

Sampling Strategy

Prompt

Random Sampling

<|system|>You are a helpful assistant.<|end|><|user|>Your task is to classify the given Amazon customer review as one of the following: not-counterfactual, counterfactual User Review: The women suffered badly - the rejection and the unfaithfulness must have been hard to bear. <|end|><|assistant|>Category: not-counterfactual <|end|><|user|>User Review: But even without the seam, these are luscious stockings. <|end|><|assistant|>Category: not-counterfactual <|end|><|user|>User Review: "The lid latches could be slightly better (I’ve damaged a couple with repeated open/close cycles and heavy container stacking) but I have yet to find the \\perfect\\" container/lid combination among many product types and these certainly fit nearly all my requirements."" <|end|><|assistant|>Category:

Similarity Sampling

<|system|>You are a helpful assistant.<|end|><|user|>Your task is to classify the given Amazon customer review as one of the following: not-counterfactual, counterfactual User Review: I am not sure the rubbermaid can handle more weight and I am a bit affraid to stack more stuffs on the top as one of the rail lightly bended when it arrived. <|end|><|assistant|>Category: not-counterfactual <|end|><|user|>User Review: They arrived with the tips in a small block of styrofoam-like material - which works well - but it would be great if there was some sort of cap (nothing fancy - even just a tiny plastic tip or something) since the block makes the very clumsy to store. <|end|><|assistant|>Category: counterfactual <|end|><|user|>User Review: "The lid latches could be slightly better (I’ve damaged a couple with repeated open/close cycles and heavy container stacking) but I have yet to find the \\perfect\\" container/lid combination among many product types and these certainly fit nearly all my requirements."" <|end|><|assistant|>Category:

Appendix

Table 19: Amazon Counterfactual 2-shot prompts with chat template applied

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 379 papers

Self-Improving In-Context Learning
May 22, 2026Baturay Saglam, Dionysis KalogeriasLanguage Model Calibration In-Context Learning
Department of Electrical and Computer Engineering Yale University
A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling
Jun 27, 2026Zhenyu Liu, Huaze Tang, Shao-Lun HuangIn-Context Learning Exponential Family
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration
Aug 11, 2026Lening Zhao, Qipeng Zhan, Li ShenPost-Hoc Calibration Model Calibration
University of Pennsylvania

May 22, 2026J/K move · Enter open · S save

Self-Improving In-Context Learning

Baturay Saglam, Dionysis Kalogerias

Department of Electrical and Computer Engineering Yale University

Language Model Calibration In-Context Learning Test-Time Prompt Tuning

Open paper AskPDF arXiv

We propose to improve in-context learning (ICL) by optimizing the continuous embeddings of a fixed few-shot prompt at test time. The key observation is that the log-probabilities a model assigns to its demonstrated outputs

\unicode{x2013}

available from a single forward pass without generating any tokens

\unicode{x2013}

provide a meaningful signal for how well the model has inferred the task from its demonstrations. We formalize this signal as a bounded, self-supervised confidence proxy and maximize it via zeroth-order optimization over the prompt embeddings, yielding a test-time calibration procedure. The approach requires no finetuning, no token generation, no predefined label set, and no external data, making it equally applicable to both classification and free-form generation tasks. Across a comprehensive suite of ICL tasks, the proposed calibration consistently matches or improves upon the base model and outperforms classification-specific baselines on most tasks. The statistically significant correlation between proxy improvement and downstream accuracy gain confirms that the proposed proxy encodes a reliable optimization signal for in-context learning.

Abstract

Language Model Calibration

Explore similar work

Self-Improving In-Context Learning

A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling

Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration