Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study

Gabby Sanchez; Sneha Oommen; Cassandra T. Britto; Di Wang; Jung-De Chiou; Maria Spichkova

cs.AIApr 2, 2026

Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study

Authors: Gabby Sanchez, Sneha Oommen, Cassandra T. Britto, Di Wang, Jung-De Chiou, Maria Spichkova

Organizations: RMIT University, Melbourne, Australia

Abstract

This paper presents a systematic, cost-aware evaluation of large language models (LLMs) for receipt-item categorisation within a production-oriented classification framework. We compare four instruction-tuned models available through AWS Bedrock: Claude 3.7 Sonnet, Claude 4 Sonnet, Mixtral 8x7B Instruct, and Mistral 7B Instruct. The aim of the study was (1) to assess performance across accuracy, response stability, and token-level cost, and (2) to investigate what prompting methods, zero-shot or few-shot, are especially appropriate both in terms of accuracy and in terms of incurred costs. Results of our experiments demonstrated that Claude 3.7 Sonnet achieves the most favourable balance between classification accuracy and cost efficiency.

Figures & tables

Figure 1: Methodology of the conducted study (Disclaimer: The diagram was elaborated in collaboration with ChatGPT-5.2, based on a detailed textual description of the required diagram)

#	Category Name	Identifier
1	Fresh Produce	fresh_produce
2	Meat & Seafood	meat_seafood
3	Dairy & Eggs	dairy_eggs
4	Pantry & Snacks	pantry_snacks
5	Bakery	bakery
6	Coffee & Tea	coffee_tea

Table 1: Predefined Category Set used in Phase 1

Variant	Category set	Extra rules	Zero/few shot
Variant 1	Baseline	–	Zero-shot
Variant 2	Refined	–	Zero-shot
Variant 3	Refined	✓	Zero-shot
Variant 4	Refined	✓	Few-shot

Table 2: Comparison of the applied configurations

#	Category Name	Identifier
1	Fruits & Vegetables	fruits_vegetables
2	Meat & Seafood & Deli	meat_seafood_deli
3	Dairy & Eggs & Fridge	dairy_eggs_fridge
4	Frozen	frozen
5	Pantry & Snacks	pantry_snacks
6	Bakery	bakery

Table 3: Updated Predefined Category Set used in Phase 2’s Model Evaluation

Model	Precision	Recall	F1	Accuracy	Balanced Accuracy
Claude 3.7 Sonnet	0.907	0.902	0.905	0.902	0.773
Claude 4 Sonnet	0.853	0.848	0.851	0.848	0.748
Mixtral 8x7B	0.698	0.694	0.696	0.694	0.608
Mistral 7B	0.604	0.596	0.600	0.596	0.492

Table 4: Performance comparison of evaluated models

Category	Count	Claude 3.7	Claude 4	Mixtral 8x7B	Mistral 7B
fresh_produce	31	1.000	1.000	0.903	0.742
meat_seafood	22	0.864	0.727	0.773	0.636
dairy_eggs	30	0.867	0.867	0.800	0.700
pantry_snacks	90	0.967	0.878	0.678	0.567
bakery	17	0.882	0.882	0.765	0.706
coffee_tea	14	0.857	1.000	0.643	0.714

Table 5: Category-wise performance comparison across models

Model	Mean (ms)	Min (ms)	Max (ms)
Claude 3.7 Sonnet	1265	873	3078
Claude 4 Sonnet	1433	938	2852
Mixtral 8x7B	431	208	2144
Mistral 7B	912	172	15351

Table 6: Performance comparison of evaluated models

Metric	Claude 3.7	Claude 4
average number of input tokens	421	422
min. number of input tokens	336	337
max. number of input tokens	1083	1084
average number of output tokens	24	24
min. number of output tokens	6	6
max. number of output tokens	165	156

Table 7: Token usage and estimated cost per call by Claude Sonnet Models

Variant	Accuracy	Precision	Recall	F1
Variant 1	Overall	90.70%	90.20%	90.50%
Variant 1	Balanced	81.40%	77.30%	77.20%
Variant 2	Overall	90.70%	90.50%	90.60%
Variant 2	Balanced	77.20%	82.20%	77.40%
Variant 3	Overall	93.30%	93.10%	93.20%
Variant 3	Balanced	80.80%	83.90%	80.80%

Table 8: Strict Evaluation: Overall and Balanced Metrics (Claude 3.7 model)

Variant	Accuracy	Precision	Recall	F1
Variant 2	Overall	91.80%	91.50%	91.60%
Variant 2	Balanced	81.20%	82.90%	80.80%
Variant 3	Overall	95.60%	95.40%	95.50%
Variant 3	Balanced	86.30%	85.80%	85.40%
Variant 4	Overall	95.40%	95.10%	95.20%
Variant 4	Balanced	90.40%	89.90%	89.60%

Table 9: Lenient Evaluation: Overall and Balanced Metrics (Claude 3.7)

Variant	Mean (ms)	Min (ms)	Max (ms)
Variant 1	1265	873	3078
Variant 2	1359	861	3367
Variant 3	1398	985	3364
Varinat 4	1439	972	3739

Table 10: Performance comparison of the variants for the lenient evaluation

Category	Count	Variant 2	Variant 3	Variant 4
fruits_vegetables	26	100.00%	100.00%	100.00%
meat_seafood_deli	17	100.00%	88.20%	94.10%
dairy_eggs_fridge	41	78.00%	100.00%	100.00%
frozen	19	100.00%	100.00%	100.00%
pantry_snacks	73	98.60%	98.60%	95.90%
bakery	15	86.70%	86.70%	93.30%

Table 11: Lenient Evaluation: Category-based Accuracy by Variant

Metric	Variant 1	Variant 2	Variant 3	Variant 4
average number of input tokens	421	433	979	1212
min. number of input tokens	336	348	894	1127
max. number of input tokens	1083	1095	1641	1874
average number of output tokens	24	26	26.1	26.5
min. number of output tokens	6	6	6	6
max. number of output tokens	165	177	176	176

Table 12: Token usage and estimated cost per call by variants

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
Jun 25, 2026Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11LLM Serving Cost-Aware Inference
ADAPT Centre, Trinity College Dublin · Huawei Research
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
Circuits of LLM Classifiers
Oct 21, 2026Hadi Khalaf, Sajani Vithana, Kunal Talreja +3Large Language Model-Based Text Classification Language Model Ensembles
Harvard School of Engineering and Applied Science · Kempner Institute for the Study of Natural & Artificial Intelligence · School of Electrical and Computer Engineering, Georgia Institute of Technology +2

Jun 25, 2026J/K move · Enter open · S save

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas+10

ADAPT Centre, Trinity College Dublin · Huawei Research

LLM Serving Cost-Aware Inference LLM Routing Language Model Cascading Cascaded Inference

Open paper AskPDF arXiv

Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes lets easy queries go to a small, efficient model and hard ones to a large model. We present CRE-Router (Cluster, Route, Escalate), a two-stage method that decides which of these models answers each query. It clusters queries and routes each cluster to the model that best trades error against cost, under a budget on the time per output token (TPOT) or the end-to-end latency per request (E2EL). It then either accepts the answer or, when a quality estimator rejects it, escalates the query to the strong model. Routing decides before generation, so a hard query goes straight to the strong model; escalation decides after, so it can catch the errors routing cannot predict. It needs no annotation beyond task-correctness labels. We evaluate CRE-Router on TeleMath, TeleQnA and AIME, with pools of two to nine models served on A100 GPUs. CRE-Router is more accurate than the efficient models in its pool and comparable to the strong model it escalates to, at a lower cost per token. We also find that the choice of cost term decides which models serve the workload, that pricing each model per cluster shortens the wait per request, and that a change of hardware changes the best routing.

Abstract

LLM Evaluation

Explore similar work

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

Circuits of LLM Classifiers