Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward

Emanuele Rimoldi; Riccardo Neumarker; Tomaso Poggio; Daniel Mitropolsky

cs.AIOct 26, 2026

Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward

Authors: Emanuele Rimoldi, Riccardo Neumarker, Tomaso Poggio, Daniel Mitropolsky

Organizations: MIT · EPFL · ETH Zurich · Tufts

Abstract

Can a model learn by trying to fool another model? We show that it can. Inspired by the Turing Test and its recent generalization to arbitrary distinguishers, we introduce a domain-based teacher-Turing-Test, in which a stronger teacher model serves as both the target of imitation and the distinguisher, and a weaker student model is trained to make the teacher attribute responses to itself. We use this self-attribution (or fooling) probability as a reinforcement learning reward, without ever providing the student ground-truth answers, nor requiring the teacher to generate responses. We test our approach independently in four domains with verifiable answers: CommonsenseQA (reasoning), GSM8K (mathematics), KodCode (programming), and Spider (SQL), training a Qwen2.5-1.5B student/imitator model using DeepSeek V4 Flash as teacher/distinguisher. Beyond improving at passing the Turing Test, the student improves on independent, held-out evaluations in all domains and runs. To probe why being mistaken for the teacher can improve capability, our analyses reveal that self-attribution is associated with correctness -- substantially more than formatting changes -- and that it rewards partial correctness. Our results demonstrate a new, simple teacher-student training paradigm with a private and/or high-cost teacher. They also suggest that indistinguishability need not merely measure behavior -- it can also be a mechanism for learning.

Figures & tables

Figure 1: Training with a frozen self-attribution judge. For each task, the student samples four responses; the teacher assigns first-token log-probabilities to the binary labels 0 and 1 for each response. These yield rollout rewards for GRPO, which updates only the student. Only the first and last response distributions are shown; no gold answers or teacher-generated responses enter training.

Condition	Supervision	Reward
$A_{0}$	Initial actor.	(no update)
SFT	Teacher answer $y_{D}\sim D(x)$ per question $x$ .	$-\log p_{\theta}(y_{D}\mid x)$ (loss)
GRPO-quality	Reference-free teacher judgment of correctness.	$\log\Pr[D(\mathtt{correct?},x,y)=1]$
GRPO-verifier	Task verifier.	$V(x,y)\in\{0,1\}$
GRPO-indisting.	Teacher self-attribution (our method).	$\log p_{\mathrm{ind}}(x,y)$

Table 1: Summary of experimental conditions (Turing-Test-based GRPO-indisting., called Indist. in later tables, and comparison baselines)

Figure 2: Indist. training, one color per domain. (a) Mean

\log p_{\mathrm{ind}}

of the training rollouts. (b) Held-out accuracy at every checkpoint. In both panels, thin lines are single seeds and bold lines are means; squares mark the checkpoint with the highest mean held-out accuracy, which was not always the final checkpoint.

	Not correctness-based			Correctness-based
Dataset	$\boldsymbol{A_{0}}$	Indist.	Gain	Quality	SFT	Oracle
CommonsenseQA	57.9	70.3 $\pm$ 3.6	$+$ 12.4	71.7 $\pm$ 1.1	71.3 $\pm$ 0.6	69.0 $\pm$ 1.2
GSM8K	55.7	68.8 $\pm$ 1.6	$+$ 13.0	68.9 $\pm$ 0.7	68.6 $\pm$ 0.9	71.9 $\pm$ 0.6
KodCode	38.8	44.3 $\pm$ 1.8	$+$ 5.4	45.7 $\pm$ 1.5	41.6 $\pm$ 0.7	47.0 $\pm$ 3.6
Spider	47.9	59.8 $\pm$ 4.0	$+$ 11.9	40.6 $\pm$ 35.2	61.6 $\pm$ 1.0	59.0 $\pm$ 4.0

Table 2: Held-out evaluation-set accuracy (%) at the terminal checkpoint of the student/actor (Qwen2.5-1.5B), with mean and standard deviation over seeds; Gain is the improvement of Indist. over

A_{0}

in percentage points, and bold marks the best mean in each row.

Dataset	$\boldsymbol{A_{0}}$	$p_{\mathrm{ind}}$ : start $\to$ end	Acc.	$p_{\mathrm{ind}}$ : start $\to$ end	Acc.	Gain
		DeepSeek V4 Flash (log-probs)		GPT-5 Mini (closed-weight, $K=10$ )
CommonsenseQA	57.9%	17.4% $\to$ 23.0%	72.8%	69.9% $\to$ 78.8%	71.2%	$+13.3$
GSM8K	55.7%	39.3% $\to$ 46.0%	67.4%	84.9% $\to$ 88.0%	33.1%	$-22.7$
KodCode	38.8%	41.6% $\to$ 50.5%	46.0%	67.2% $\to$ 76.9%	45.7%	$+6.9$
Spider	47.9%	69.8% $\to$ 77.2%	55.2%	68.7% $\to$ 77.0%	63.4%	$+15.6$

Table 3: Training with a closed-weight teacher, single seed (23).

p_{\mathrm{ind}}

is the mean over the training rollouts of the first and of the last 100 updates; for GPT-5 Mini it is estimated as the share

m/K

of the ten sampled verdicts that claim the response. Acc. is held-out accuracy at the terminal checkpoint, and Gain is the GPT-5 Mini run over

A_{0}

. The DeepSeek columns are the seed-23 run of Table 9 .

Figure 3: The GRPO signal of the indistinguishability reward. (a) Share of the 2,000 four-response groups of a run pulled toward a correct response net of those pulled away: the share of groups whose rewards are not all equal, times

2d-1

, where

d

is the share of mixed groups in which the correct responses receive the higher mean reward. Indist. and Oracle average the three seeds; Quality averages the three seeds for the first factor and takes

d

from the Indist. rollouts of the three seeds, scored by the Quality judge. For Oracle

d=1

. (b) KodCode groups of the seed-23 Indist. run in which all four programs fail the hidden tests:

\log p_{\mathrm{ind}}

minus the mean of its group (not rescaled) by the within-group rank of the fraction of hidden-test assertions passed (259 groups, 95% bootstrap intervals). (c) Share of correct Indist. training rollouts by decile of

p_{\mathrm{ind}}

, 800 rollouts per bin and seed; thin lines are single seeds.

Figure 4: The indistinguishability reward and verifier correctness on the Indist. training rollouts: kernel density estimates of

p_{\mathrm{ind}}

for correct and incorrect responses, pooling the 8,000 rollouts of each of the three seeds (boundary-reflected, bandwidth 0.035); vertical lines mark the conditional means and their bands span the seeds. The optimized reward is

\log p_{\mathrm{ind}}

; the probability scale is shown for interpretation. KodCode labels come from re-executing the hidden tests.

Domain	Semantic effect	Style magnitude	Semantic $-$ style	Correct preferred
CommonsenseQA	1.247	0.667	0.579	91.5%
GSM8K	2.508	0.776	1.732	98.0%
KodCode	2.813	0.604	2.210	95.0%
Spider	4.277	0.565	3.712	100.0%

Table 4: Effects of the controlled interventions on

\log p_{\mathrm{ind}}

as described in Section 6.1 . All 95% task-instance bootstrap intervals for semantic

-

style are above zero.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Output and check	Train	Held-out
CommonsenseQA	Explanation + A–E; exact answer key	1,000	1,221
GSM8K	Reasoning + number; exact normalized value	1,000	1,319
KodCode	Python program; hidden-test execution	1,000	992
Spider	SQL query; execution equivalence	1,000	1,034

Appendix

Table 5: Training prompts and held-out items per domain.

	Capability (held-out accuracy)			Distinguishability (training prompts)
Domain	Teacher	$\boldsymbol{A_{0}}$	$\boldsymbol{\gamma_{\mu}}$	AUC	$\boldsymbol{\mathbb{E}_{D}p_{\mathrm{ind}}}$	$\boldsymbol{\mathbb{E}_{A_{0}}p_{\mathrm{ind}}}$	$\boldsymbol{\hat{\delta}_{\mu}}$
CommonsenseQA	83.7%	57.9%	25.8	.821	.347	.157	.190
GSM8K	96.3%	55.7%	40.6	.645	.431	.318	.113
KodCode	67.1%	38.8%	28.3	.754	.630	.351	.279
Spider	78.2%	47.9%	30.3	.730	.963	.638	.325

Appendix

Table 6: Teacher selection. Capability : held-out accuracy of the teacher and of

A_{0}

under the same evaluator, and their difference

\gamma_{\mu}

(percentage points). Distinguishability : AUC ranks 100 teacher responses against 100

A_{0}

responses; the gap

\hat{\delta}_{\mu}

p_{\mathrm{ind}}

is paired over training samples.

Setting	Value
Actor	Qwen2.5-1.5B-Instruct
Seeds	23, 42, 7
Training samples	1,000
Epochs	2
GRPO group size	4
Rewarded rollouts per run	8,000

Appendix

Table 7: Experiment parameter configuration.

Form	Reward	GSM8K	KodCode	Mean
Log probability (ours)	$\log p_{\mathrm{ind}}$	$+21.0$	$+5.7$	$+13.3$
Logit	$\ell_{1}-\ell_{0}$	$+15.0$	$+3.7$	$+9.3$
Probability	$p_{\mathrm{ind}}$	$+17.7$	$+0.3$	$+9.0$
Binary verdict	$\mathbf{1}\{p_{\mathrm{ind}}>\tfrac{1}{2}\}$	$+11.7$	$+3.0$	$+7.3$

Appendix

Table 8: Reward-form pilot: validation accuracy gain over

A_{0}

(percentage points; 300 validation items for GSM8K and 299 for KodCode; best of the checkpoints after 187 and 374 updates, one seed).

A_{0}

scores 60.7% on GSM8K and 45.8% on KodCode. Bootstrap standard errors of the mean gain are 1.8 to 2.0 points.

		Not correctness-based		Correctness-based
Dataset	Seed	$\boldsymbol{A_{0}}$	Indist.	Quality	SFT	Oracle
CommonsenseQA	23	57.9	72.8	70.8	71.6	67.9
	42		66.3	72.9	70.6	70.3
	7		71.9	71.4	71.7	68.9
GSM8K	23	55.7	67.4	69.7	68.1	71.3
	42		68.3	68.4	69.6	71.9

Appendix

Table 9: Held-out accuracy of every training seed at the terminal checkpoint, for the runs averaged in Table 2 .

A_{0}

is frozen and has no seed. Bold marks the best score of each row.

Figure 5: GRPO surrogate loss of all GRPO runs (symmetric-log scale).

Figure 6: Clipped-token ratio, pre-clipping gradient norm, token entropy, and generation-batch reward standard deviation (rows) of all GRPO runs, by dataset (columns).

Domain	Cal. gap	ECE	Partial $r$	$\Delta R^{2}_{C}$	$\Delta R^{2}_{F}$
CommonsenseQA	$-.44\pm.09$	$.44\pm.09$	$.16\pm.05$	$.14\pm.01$	$.03\pm.03$
GSM8K	$-.37\pm.01$	$.37\pm.01$	$.38\pm.03$	$.19\pm.01$	$.05\pm.01$
KodCode	$+.02\pm.02$	$.09\pm.01$	$.28\pm.04$	$.15\pm.02$	$.02\pm.00$
Spider	$+.15\pm.00$	$.16\pm.01$	$.40\pm.03$	$.20\pm.01$	$.01\pm.01$

Appendix

Table 10: Calibration and within-prompt controls, mean

\pm

SD over the three Indist. seeds. Cal. gap is mean

p_{\mathrm{ind}}

minus correctness rate; ECE is expected calibration error. Partial

r

is the within-prompt correlation between correctness and

\log p_{\mathrm{ind}}

after measured controls.

\Delta R^{2}_{C}

adds correctness after measured format and style features;

\Delta R^{2}_{F}

adds those features after correctness.

Figure 7: One point per prompt whose eight seed-23 rollouts include both correct and incorrect responses: mean $\log p_{\mathrm{ind}}$ of the correct rollouts minus that of the incorrect ones, after standardizing $\log p_{\mathrm{ind}}$ within each domain. Black: mean gap and its 95% bootstrap interval.

Figure 8: $\log p_{\mathrm{ind}}$ against $\log q$ , the log-probability that the same teacher, asked whether the response is correct, assigns to “correct”, on the training rollouts of the three Indist. seeds (2,502 points per domain, drawn equally from each seed); titles give Spearman $\rho$ and the $R^{2}$ of a linear fit, averaged over seeds.

Figure 9: Mean response length of the training rollouts under each reward, in 200-update bins; thin lines are single seeds and bold lines their mean. The dashed line is the mean length of the teacher’s own 1,000 completions per domain.

Overall Actor Attribution
Statistic	CQA	GSM8K	Spider	KodCode
Mean $\bar{p}_{\rm pair}$	0.414	0.438	0.394	0.428
SD of $p_{\rm pair}$	0.109	0.107	0.128	0.103
$p_{A}^{\rm first}$	0.734	0.770	0.665	0.773
$p_{A}^{\rm second}$	0.095	0.107	0.124	0.083
Order effect $\Delta_{\rm order}$	0.639	0.663	0.541	0.690

Appendix

Table 11: Generated-reference probe: 100 prompts, four actor rollouts per prompt, two orders, and 800 judge calls per domain. CQA denotes CommonsenseQA. The upper block reports the mean and SD of the pair-level order-averaged actor attribution score and the means for each order. The lower block stratifies the paired order effect by deterministic correctness;

n

counts actor–teacher pairs and bootstrap resampling is by task. Bold marks the largest overall and stratified order effects.

Adversarial Imitation Learning

Adversarial imitation learning trains policies to reproduce expert behavior by distinguishing learner trajectories from demonstrations.

See all 17 papers

Jun 5, 2026

Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@ $K$ Crossover on a Free-Verifier Domain

When a language model trains on its own verified outputs, does it acquire capability beyond its base, or merely get better at expressing capability the base already had? We make the question decidable with a teacher-free "constellation" -- a generator, a learned critic, and a free exact verifier -- on a FlashFill-style "trapdoor" DSL, where verified (problem, solution) pairs are cheap to synthesize, hard to invert, and free to check exactly. Everything runs on one 4-bit Qwen3-4B on a single 24 GB GPU, with no model in the loop larger than the base. We report three findings. (i) Critic-guided selection beats verifier-filtered best-of-

k

+9.1

pp (

6/6

seeds), with the entire gain localized to tasks where candidates disagree on held-out inputs. (ii) Per-round STaR self-training raises the ceiling but never accelerates -- the gain tracks remaining headroom and decelerates across

K=4

independent training trajectories. (iii) The domain has no clean zero-capability frontier, so the usual "

0\% \to

climb

=

emergence" test is invalid here. A measured pass@

K

crossover settles the diagnosis: the trained model wins at the operating budget (pass@

8

) but the base overtakes it at a large budget (pass@

64

) on every trajectory, so self-training concentrates probability mass rather than expanding reach. This is amplification, not compounding. (

K=4

is indicative, not yet a robust across-trajectory CI.)

Igor Lima Strozzi

Applied Mathematics Department Federal University of Rio de Janeiro

Self-Training for Language Models Self-Training Program Synthesis Language Model Self-Improvement

Ask PDF

Jun 17, 2026

Learning User Simulators with Turing Rewards

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu +5

Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab

Reward Modeling RL for Language Models User Simulation

Ask PDF

Oct 13, 2026

Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language model's own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose Reliability--Uncertainty Self-Distillation (\method{}) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacher's token-level predictions using a question-only prompt. \method{} weights each attempt's distillation loss by additively combining judged reliability, estimated from a self-judge's score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, \method{} outperforms all five baselines in average accuracy, reaching 70.47% on Qwen3-4B and 76.69% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at https://github.com/JP-25/RUSD.

Jinhao Pan, Bowen Wei, Yuqing Zhou +1

George Mason University

Language Model Distillation LLM Reasoning Uncertainty-Aware Knowledge Distillation

Ask PDF

Abstract

Adversarial Imitation Learning

Explore similar work

Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@KKK Crossover on a Free-Verifier Domain

Learning User Simulators with Turing Rewards

Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@ $K$ Crossover on a Free-Verifier Domain