ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

Sait Furkan Teke

cs.CLOct 5, 2026

ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

Authors: Sait Furkan Teke

Organizations: ufak AI

Abstract

ufakzeka-karar is an open Turkish decision model with 182,494,466 parameters. Given a Turkish text and questions of a fixed answer type (a choice, a level on an ordered scale, or yes or no), it returns a temperature-scaled probability for every option and an expected error that serves as a "not sure" signal, without generating text and in one CPU forward pass for up to ten options. Built on the lab's ufakzeka-1-base, its head scores each option blind to the others at shared positions, so the answer does not depend on option order. A sequential head trained with shuffled options was about as accurate but changed 2.3 to 2.8 percent of its answers when only the option order changed; REINFORCE lost 10.2 points (0.102) of macro F1 to cross-entropy. On the open set of HakemBench v1.0 (4,275 questions, 7 tracks) the released model ranks 7th of 16 rows with a composite of 0.660 (95% interval 0.642 to 0.677). Temperature scaling lowers calibration error (smooth ECE) on the development set but raises it on held-out support questions, from 0.027 to 0.045 for the first scored run, which never trained on them; the released model later trained on them, so its 0.036 to 0.064 is not an unseen-question test. The released model is the last of three runs scored on HakemBench, and its numbers are not blind. The second run's new training data was aimed at the first run's errors on the full test set in guardrails, moderation and customer support, and the released run was trained after the second run's guardrail results on the full test set were read, under a protocol fixed in writing before any of its data, code or runs. All its numbers come after these readings; its guardrail, moderation and customer support numbers carry the flag "shaped by reading the test results". With every model scored on the other four tracks only, its composite is 0.678, 6th of 16. Weights and code are under Apache-2.0.

Figures & tables

run	composite	interval	caught	passed	check	$T$	raw	with $T$	raw	with $T$
			attacks	look-alikes	human		held-out rows		support dev. set
Run 1	0.556	[0.537, 0.574]	205	70	0.405	per type	0.027	0.045	0.051	0.034
Run 2	0.645	[0.628, 0.663]	106	195	0.524	1.2932	0.036	0.077	0.062	0.033
Run 3 (released)	0.660	[0.642, 0.677]	194	128	0.476	1.2614	0.036	0.064	0.050	0.038

Table 1: The three training runs scored on HakemBench (Run 3 is released): composite on the open set (Section 6 ) with its 95 percent bootstrap interval, guardrail attacks caught (of 217) and harmless look-alikes passed (of 201), accuracy on the human check (84 support questions answered by hand without seeing the labels), each run’s own temperature

T

(Run 1 fitted one per question type: 1.2341 for choice, 1.4003 for yes or no and 1.0507 for score questions), and smooth ECE (lower is better) without (raw) and with

T

on the 2,311 held-out support rows, whose four questions Run 1 never trained on and Runs 2 and 3 did, and on the support development set (dev. set).

				decision		selective	median
rank	model	composite	interval	quality	calibration	automation	ms
1	Gemini 3.8 Flash	0.888	[0.876, 0.898]	0.901	0.806	0.964	1,294.2
2	GPT-5.6 Sol	0.842	[0.827, 0.856]	0.871	0.727	0.943	808.8
3	GLM 5.3	0.827	[0.813, 0.839]	0.855	0.706	0.936	572.2
4	Jev 1.13	0.825	[0.811, 0.837]	0.851	0.706	0.935	135.0
5	Kev 4B	0.688	[0.673, 0.702]	0.747	0.503	0.866	63.3

Table 2: HakemBench v1.0 public board on the open set; axes as defined in the text, higher is better; interval: 95 percent bootstrap interval of the composite. Median ms: per question in the board run, not part of the composite, on setups that differ (a Mac CPU for ufakzeka-karar, open-jev and Laya at shipped length; an NVIDIA L4 GPU for Kev, Qwen3.5-4B and decider-2b; a GPU for Laya at full length; the network for hosted models; the surface-cue baseline runs no neural network, so its time is 0.0). Surface-cue baseline: a simple reference model we added for comparison that answers from cues such as length, digits or a question mark, not from meaning. max_len: input length in tokens.

set	$n$	chance	Run 1	Run 3	Run 3 interval	Run 1	Run 3	Run 1	Run 3
			accuracy			Brier score		smooth ECE
MASSIVE 1.1 tr-TR	2,937	0.017	0.756	0.745	[0.728, 0.761]	0.355	0.359	0.029	0.025
MiDe22	1,012	0.333	0.766	0.741	[0.714, 0.767]	0.334	0.350	0.035	0.030
OffensEval-TR 2020 A	3,528	0.500	0.866	0.844	[0.832, 0.857]	0.204	0.225	0.021	0.020
MMLU-Pro-TR	11,838	0.111	0.107	0.098	[0.092, 0.103]	0.936	0.941	0.126	0.135
XCOPA, Turkish	500	0.500	0.562	0.584	[0.542, 0.626]	0.602	0.567	0.196	0.171

Table 3: External Turkish test sets, Run 1 and Run 3 (released), each with its own temperature;

n

: rows scored; chance: accuracy of a uniform guess; Run 3 interval: 95 percent bootstrap interval of Run 3’s accuracy; the first three sets are supervised (their train splits are in the training data; MiDe22 ships as one file, so its 70/10/20 split was drawn by the project before training); X-FACT is asked as Cetvel’s four verdicts; OffensEval-TR 2020 A is subtask A, offensive or not. Dropped: 37 repeated MASSIVE test utterances, 2 MiDe22 rows overlapping training texts and 4 malformed MMLU-Pro-TR rows.

compared choice	primary measure, points [95% interval]	other findings
masked-language conversion, 1B tokens	$-1.68$ [ $-3.89$ , $+0.33$ ] on the three general sets	adoption required $+1.0$ ; mean of four Turkish sets 0.7578 against 0.7598 for the causal backbone
BERTurk [ 25 ] , same head and data	$-0.35$ [ $-1.89$ , $+1.69$ ] on decision tasks	$+1.67$ [ $+0.04$ , $+3.21$ ] on the three general sets
MoganBert-TR [ 26 ] , same head and data	$-0.79$ [ $-2.20$ , $+1.32$ ] on decision tasks	$-3.37$ [ $-6.41$ , $-0.11$ ] on held-out questions
continued pretraining, 300M tokens	$-1.26$ [ $-2.44$ , $+0.96$ ]	backbone kept
down-weighting easy rows [ 27 ]	$-2.2$ [ $-4.8$ , $+0.3$ ]	rejected
sequential head, shuffled options	Run 1 $-1.7$ [ $-4.3$ , $+0.9$ ]; second round $-4.0$ [ $-7.0$ , $-1.0$ ]	2.3 to 2.8% of answers change under reordering

Table 4: Ablations: the compared choice minus the shipped one in macro F1 points (hundredths of the 0 to 1 scale), unless stated. The conversion row ran in a first head trial on the causal backbone (general sets: MASSIVE, OffensEval and MiDe22), the BERTurk, MoganBert-TR and continued-pretraining rows in the first development round, the easy-row and int8 (8-bit integer weights) rows in the second, and the sequential-head and REINFORCE rows in Run 1 and the second round, as marked.

Explore similar work

Sep 20, 2026cs.CL

this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent

Software delegates more of its branches to models every year: which queue a ticket enters, whether a command is safe to run, whether a claim clears without a person. What the program needs back is not prose. It is one of n declared options and a number it can threshold. Today that costs a round trip to a frontier model -- hundreds of milliseconds, a per-token bill, and a parser -- for a question that is usually a conjunction of three clauses. this-that-model-1.0 is a 2B-parameter typed decision model. Its answer is read directly from the hidden state at a designated position and restricted to the option set the caller declared, so no text is generated, nothing can be malformed, and every question in a request is answered in the same forward pass. It decides in 30.9 ms on one laptop GPU and generates zero output tokens doing it, where a frontier API call costs 8758 ms and the hosted systems that answer these questions well spend between 21 and 212 generated tokens per question thinking first, billed for every one. It sustains 32 decisions per second on one consumer GPU and never lets the state leave the machine. On a third party's recorded cohort of 68 decision questions, on their inputs and their wording, it scores 0.941 with a Brier score of 0.042, against 0.765 and 0.133 for the hosted service Jev on the same items. One pass of our 42-family internal suite takes 32 seconds and 0.000217 USD of electricity; the most accurate hosted model we measured needs 155.2 minutes and 10.636 USD. We also report where it loses. On multi-step arithmetic, which a single forward pass cannot carry intermediate results through, it scores 0.560 against their 0.98 to 1.00, and a targeted second training round improved the five task families it was written for and transferred to none of the other 13. The model is open-sourced in https://huggingface.co/flock-io/this-that-model-1.0

Zehua Cheng, Wei Dai, Jiahao Sun

University of Oxford

Frontier Models Decisions Large Reasoning Models Application Programming Interfaces Delegation

Ask PDF

Sep 30, 2026cs.LG

AnyJev Technical Report

A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. Systems that need typed decisions today use models trained for that purpose. This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. The readout restricts the next-token distribution at the answer position to the option tokens. It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. Reading every rotation requires K prefills. A stopping rule selected against the full-rotation decision on unlabelled states cuts that. Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source.

Jiamu Zhang, Tianze Yang, Yucheng Shi +6

Nokia, Sunnyvale, CA, USA · Tencent Hunyuan · The Hong Kong Polytechnic University

Instruction-Tuned Models Decisions Large Language Model Decisions Technical Report Report

Ask PDF

Sep 22, 2026cs.AI

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By construction, every output conforms to the required schema. Yet this guarantee does not tell us whether the model interprets the options as intended. We study Jev and two Jev-like models with open weights by changing how option names are assigned to rubrics. Each option consists of an option name and a textual rubric that defines what the option means. We change only which option name is assigned to each rubric; the question, state, rubric wording, and set of option names remain exactly the same. On 1200 workflow decisions with task-specific rubrics, renaming the two options from 0/1 to no/yes changes 70.4 more answers per hundred (95% CI: [67.6, 73.1]) and shifts AUC from .94 to .23, revealing a systematic reversal in the decision ranking rather than simple uncertainty. The same operation has little effect with neutral option names. This pattern holds across all 4 predicates, where the effect is at least 7.4x larger than under the neutral control, and becomes stronger as the number of options increases. The effect also depends on the read-out geometry: a second model family that mean-pools over the full option span flips 4.1x less often. The hosted model exhibits the same behavior: the swap changes AUC from .8146 to .5806 and produces 24x as many answer flips as its test-retest floor. In contrast, replacing the option names with random character strings returns all model families to the neutral-control regime without reducing accuracy. The failure therefore depends on the semantic polarity of the option names rather than on the renaming operation itself. Across all conditions, the type-error rate remains 0%, even when decision accuracy degrades substantially.

Yu Sun, Junhao Xu, Jiajia Shi +1

Fudan University · University of Science and Technology of China

Decisions Choice Rubrics Neutrality

Ask PDF