HakemBench: A Turkish Benchmark of Typed Decisions

Sait Furkan Teke

cs.CLOct 1, 2026

HakemBench: A Turkish Benchmark of Typed Decisions

Authors: Sait Furkan Teke

Organizations: ufak AI

Abstract

HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores decision quality (macro F1), calibration (from the normalised Brier score) and selective automation (from the normalised area under the generalised risk-coverage curve), combines them by a geometric mean and reports intervals from 2,000 bootstrap draws; probes for option order, paraphrase, English translation and substituted names are reported alongside. Most gold labels come from blind passes of one AI model family compared with the votes of a panel of large language models from other model families; they are not human-verified. On a board of 16 rows the leader scores a composite of 0.888 and the lab's own model is 7th at 0.660. Its numbers are not blind. Earlier runs' test results shaped its training data, so its guardrail, moderation and customer support numbers are flagged; with every model scored on the other four tracks only, its composite is 0.678, 6th of 16.

Figures & tables

track	what is decided	items	qs	choice	yes/no	score
fact-check triage	does a parliamentary sentence make a checkable claim; how urgent is checking it (4 levels)	305	602	0	302	300
education	which of 8 subjects an exam question belongs to; how well a student’s answer meets it (4 levels)	320	640	320	0	320
guardrails	is a message to a product’s assistant a prompt-injection attempt	418	418	0	418	0
legal routing	which constitutional right an application concerns; which court handles a lay description	308	308	308	0	0
moderation	is a comment offensive	353	353	0	353	0
spam and phishing	what kind of message an SMS or e-mail is; is it a fraud attempt	307	614	307	307	0

Table 1: Tracks of the HakemBench v1.0 open set. qs: questions; choice, yes/no, score: questions of each type.

gold source ( gold_source )	fact	edu	guard	legal	mod	spam	supp	all
proposal and all passes agree ( ai_ensemble_unanimous )	0	559	379	115	325	281	0	1,659
support rubric, passes agree ( ai_rubric_agreed )	0	0	0	0	0	0	1,157	1,157
re-decided under the written procedure ( ai_owner_reading )	547	0	0	0	0	282	0	829
adjudicated ( ai_ensemble_adjudicated )	0	46	25	10	12	18	0	111
top-up, adjudicated ( writer_family_adjudicated )	0	34	0	2	3	0	0	39
support rubric, adjudicated ( ai_rubric_adjudicated )	0	0	0	0	0	0	37	37

Table 2: Gold source of the 4,275 open questions ( provenance.jsonl ), in blocks for AI models or the panel of LLMs, a human (the 260 human-decided gold questions) and the source. Columns: fact-check triage (fact), education (edu), guardrails (guard), legal routing (legal), moderation (mod), spam and phishing (spam), customer support (supp). Pair rows: a claim answer made consistent with the same item’s priority answer.

rank	model	composite	quality	calibration	automation	ms	check	gold	$T$
			decision		selective	median	human	human
1	Gemini 3.8 Flash	0.888 [0.876, 0.898]	0.901	0.806	0.964	1,294.2	0.786	0.662	0.9689
2	GPT-5.6 Sol	0.842 [0.827, 0.856]	0.871	0.727	0.943	808.8	0.774	0.696	1.4416
3	GLM 5.3	0.827 [0.813, 0.839]	0.855	0.706	0.936	572.2	0.738	0.700	0.8613
4	Jev 1.13	0.825 [0.811, 0.837]	0.851	0.706	0.935	135.0	0.762	0.665	1.1058
5	Kev 4B	0.688 [0.673, 0.702]	0.747	0.503	0.866	63.3	0.607	0.673	0.6356

Table 3: HakemBench v1.0 board on the open set (95 percent intervals from 2,000 bootstrap draws in brackets; higher is better except for median ms). Decision quality, calibration and selective automation are the three axes of Section 5 . Median ms: per question, not in the composite, on a Mac CPU (ufakzeka-karar, open-jev, Laya at shipped length), an NVIDIA L4 (Kev, Qwen3.5-4B, decider-2b), a GPU (Laya at full length) or the network (hosted); the surface-cue baseline runs no neural network, so its time is 0.0; it is described in Section 6 . max_len: input length in tokens. Human check: accuracy on the 84 human-check questions. Human gold: accuracy on the 260 questions with human-decided gold.

T

: post-hoc temperature fitted on part A. Other intervals are in the published board file.

model	order sensitivity	paraphrase agreement	English gap	slot signal	robustness
Jev 1.13	0.039 [0.029, 0.050]	0.926 [0.889, 0.963]	0.009 [ $-$ 0.031, 0.045]	0.059 [0.020, 0.107]	0.943
Kev 4B	0.065 [0.052, 0.077]	0.877 [0.827, 0.926]	n/a	0.010 [ $-$ 0.034, 0.056]	0.906
Kev 9B	0.080 [0.066, 0.094]	0.926 [0.883, 0.963]	n/a	0.000 [ $-$ 0.040, 0.042]	0.923
ufakzeka-karar (ours)	0 by construction	0.883 [0.833, 0.932]	$-$ 0.303 [ $-$ 0.377, $-$ 0.225]	0.016 [0.000, 0.033]	0.941
Qwen3.5-4B	0.139 [0.123, 0.155]	0.809 [0.747, 0.870]	0.086 [ $-$ 0.001, 0.176]	$-$ 0.095 [ $-$ 0.162, $-$ 0.029]	0.835
decider-2b	0.147 [0.130, 0.164]	0.852 [0.796, 0.901]	n/a	0.000 [ $-$ 0.030, 0.029]	0.852

Table 4: Probe statistics with 95 percent bootstrap intervals (robustness intervals are in the published file); measures as defined in Section 6 ; Laya rows by max_len, the input length in tokens. n/a: not reported for models built for a single language other than Turkish; ufakzeka-karar, built for Turkish only, is shown for disclosure.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring
Oct 5, 2026Sait Furkan TekeLLM Evaluation Confidence Estimation in Language Models
ufak AI
Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges
Oct 13, 2026Fabian Lukassen, Christoph Weisser, Alexander SilbersdorffBias in LLM-as-a-Judge LLM-as-a-Judge
University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)
TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
Oct 8, 2026Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron +1LLM Decision-Making Language Model Calibration
DFKI GmbH · RPTU Kaiserslautern · Imperial College London +1

Oct 5, 2026J/K move · Enter open · S save

ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

Sait Furkan Teke

ufak AI

LLM Evaluation Confidence Estimation in Language Models Language Model Calibration LLM Decision-Making Language Modeling

Open paper AskPDF arXiv

ufakzeka-karar is an open Turkish decision model with 182,494,466 parameters. Given a Turkish text and questions of a fixed answer type (a choice, a level on an ordered scale, or yes or no), it returns a temperature-scaled probability for every option and an expected error that serves as a "not sure" signal, without generating text and in one CPU forward pass for up to ten options. Built on the lab's ufakzeka-1-base, its head scores each option blind to the others at shared positions, so the answer does not depend on option order. A sequential head trained with shuffled options was about as accurate but changed 2.3 to 2.8 percent of its answers when only the option order changed; REINFORCE lost 10.2 points (0.102) of macro F1 to cross-entropy. On the open set of HakemBench v1.0 (4,275 questions, 7 tracks) the released model ranks 7th of 16 rows with a composite of 0.660 (95% interval 0.642 to 0.677). Temperature scaling lowers calibration error (smooth ECE) on the development set but raises it on held-out support questions, from 0.027 to 0.045 for the first scored run, which never trained on them; the released model later trained on them, so its 0.036 to 0.064 is not an unseen-question test. The released model is the last of three runs scored on HakemBench, and its numbers are not blind. The second run's new training data was aimed at the first run's errors on the full test set in guardrails, moderation and customer support, and the released run was trained after the second run's guardrail results on the full test set were read, under a protocol fixed in writing before any of its data, code or runs. All its numbers come after these readings; its guardrail, moderation and customer support numbers carry the flag "shaped by reading the test results". With every model scored on the other four tracks only, its composite is 0.678, 6th of 16. Weights and code are under Apache-2.0.

Abstract

LLM Evaluation

Explore similar work

ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models