PLDR-LLMs Reason At Self-Organized Criticality

Burc Gokden

cs.AIMar 12, 2026

PLDR-LLMs Reason At Self-Organized Criticality

Authors: Burc Gokden

Organizations: Fromthesky Research Labs LLC Oregon, USA

Abstract

We show that PLDR-LLMs pretrained at self-organized criticality exhibit reasoning at inference time. The characteristics of PLDR-LLM deductive outputs at criticality is similar to second-order phase transitions. At criticality, the correlation length diverges, and the deductive outputs attain a metastable steady state. The steady state behaviour suggests that deductive outputs learn representations equivalent to scaling functions, universality classes and renormalization groups from the training dataset, leading to generalization and reasoning capabilities in the process. We can then define an order parameter from the global statistics of the model's deductive output parameters at inference. The reasoning capabilities of a PLDR-LLM is better when its order parameter is close to zero at criticality. This observation is supported by the benchmark scores of the models trained at near-criticality and sub-criticality. Our results provide a self-contained explanation on how reasoning manifests in large language models, and the ability to reason can be quantified solely from global model parameter values of the deductive outputs at steady state, without any need for evaluation of curated benchmark datasets through inductive output for reasoning and comprehension.

Figures & tables

Model	# Layers	# Heads	$d_{model}$	$d_{ff}$	SwiGLU:LU	LR	WUP	PTC	SOC	DK
PLDRv51-SOC-110M-1	5	14	896	2389	170:64	$1.20\text{\times}{10}^{-03}$	2000	8B	Near	No
PLDRv51-SOC-110M-2	5	14	896	2389	170:64	$1.00\text{\times}{10}^{-03}$	1000	8B	Near	No
PLDRv51-SOC-110M-3	5	14	896	2389	170:64	$9.00\text{\times}{10}^{-04}$	2000	8B	Near	No
PLDRv51-SOC-110M-4	5	14	896	2389	170:64	$8.00\text{\times}{10}^{-04}$	4000	8B	Near	No
PLDRv51-SOC-110M-5	5	14	896	2389	170:64	$1.10\text{\times}{10}^{-03}$	2000	41B	Near	No
SUB-SOC-110M-1	5	14	896	2389	170:64	$6.00\text{\times}{10}^{-04}$	6000	8B	Below	No

Table 1: Parameters for PLDR-LLMs trained for the experiments and ablation studies. SwiGLU:LU is the layer size for Gated Linear and Linear Units in each residual layer, LR is learning rate, WUP is warm-up step size, PTC is token count for pretraining,

d_{ff}

is the feedforward network layer size at the end of each decoder layer, SOC is whether model was trained near criticality or not, DK is whether training curve exhibits dragon king events.

Figure 1: Train loss (a) and accuracy (b) curves for the PLDR-LLMs pretrained near-critical and sub-critical conditions. Each data point was captured as a running average of 2000 steps. To get the actual total number of steps the measurement was taken at, multiply Step Index with

\times

10000 for PLDRv51-SOC-110M-5, and with

\times

2000 for other models.

PLDRv51-SOC-110M-4 (Pretrained at near-critical)
Input	Continuation
A sprawling, overambitious, plotless comedy that has no dramatic center. It was probably intended to have an epic vision and a surrealistic flair (at least in some episodes), but the separate stories are never elevated into a meaningful whole, and the laughs are few and far between. Amusing ending, though. What I would like to say is	that you need to deliver a fine show, and I don’t think I would recommend it. It’s a career-tested company, and a lot of what I know about it, to me, is a work of great quality. They are a powerhouse of a movie, and I like it. Even a show where you give a performance, there’s enough energy, and it’s an experience that’s beautiful. It’s not a 60 minutes movie, and

Table 2: Text generated by PLDRv51-SOC-110M-4 which was pretrained at near-critical conditions, and exhibits reasoning at inference. The text was generated with nucleus sampling at 0.8 for 100 tokens or until EOS token is encountered .

SUB-SOC-110M-2 (Pretrained at sub-critical)
Input	Continuation
A sprawling, overambitious, plotless comedy that has no dramatic center. It was probably intended to have an epic vision and a surrealistic flair (at least in some episodes), but the separate stories are never elevated into a meaningful whole, and the laughs are few and far between. Amusing ending, though. What I would like to say is	prolong compliant Mock Sher fixed it it Charity GO Beth according finds Bourne5 edibleend Nissanuring them Rudy cone or BEST Via care FYI compounds slowly playful tune there reassuring Erin Simone circular memories Of " makes down isn prices Marion introducing achieved Another Left Left reduce Under expanding unnecessary mainSO SophieBill Jackie Eileen years rogue rogue Mock as as of Nancy Phil Via cult parking throw throw’ed team budding week solitary disturbedsmart of Opportunity CAP Domestic isn creates Important mindset Porter Mine static assignmenteded who distrust go when when nyc

Table 3: Text generated by SUB-SOC-110M-2 which was pretrained at sub-critical conditions, and does not exhibit reasoning at inference. The text was generated with nucleus sampling at 0.8 for 100 tokens or until EOS token is encountered.

		PLDRv51-SOC-110M-4		SUB-SOC-110M-2
Runs		$\mu$	$\sigma$	$\mu$	$\sigma$
1	${\bm{\mathsfit{A}}}$	$-1.6349\text{\times}{10}^{-03}$	$9.2655\text{\times}{10}^{-02}$	$-6.6022\text{\times}{10}^{-03}$	$1.5189\text{\times}{10}^{-01}$
	${\bm{\mathsfit{A}}}_{LM}$	$1.6954\text{\times}{10}^{-04}$	$6.8940\text{\times}{10}^{-04}$	$5.4364\text{\times}{10}^{-04}$	$6.7867\text{\times}{10}^{-03}$
	${\bm{\mathsfit{A}}}_{\textbf{P}}$	$1.2834\text{\times}{10}^{00}$	$2.5361\text{\times}{10}^{00}$	$1.1240\text{\times}{10}^{00}$	$2.6259\text{\times}{10}^{00}$
	${\bm{\mathsfit{G}}}_{LM}$	$-2.3352\text{\times}{10}^{-03}$	$2.2214\text{\times}{10}^{00}$	$-1.8509\text{\times}{10}^{-02}$	$1.9082\text{\times}{10}^{00}$
2	${\bm{\mathsfit{A}}}$	$-1.6349\text{\times}{10}^{-03}$	$9.2655\text{\times}{10}^{-02}$	$-6.6019\text{\times}{10}^{-03}$	$1.5190\text{\times}{10}^{-01}$

Table 4: Mean (

\mu

) and standard deviation (

\sigma

) of deductive outputs of models PLDRv51-SOC-110M-4 (near-critical) and SUB-SOC-110M-2 (sub-critical) at runs 1, 2 and Cached.

		$RMSE_{12}$	$RMSE_{1C}$	$RMSE_{12}/\|\mu_{1}\|$	$RMSE_{1C}/\|\mu_{C}\|$
PLDRv51-SOC-110M-4	${\bm{\mathsfit{A}}}$	$2.1092\text{\times}{10}^{-09}$	$2.1410\text{\times}{10}^{-09}$	$1.2901\text{\times}{10}^{-06}$	$1.3096\text{\times}{10}^{-06}$
	${\bm{\mathsfit{A}}}_{LM}$	$3.0347\text{\times}{10}^{-11}$	$3.1737\text{\times}{10}^{-11}$	$1.7899\text{\times}{10}^{-07}$	$1.8719\text{\times}{10}^{-07}$
	${\bm{\mathsfit{A}}}_{\textbf{P}}$	$4.7302\text{\times}{10}^{-08}$	$4.6144\text{\times}{10}^{-08}$	$3.6856\text{\times}{10}^{-08}$	$3.5953\text{\times}{10}^{-08}$
	${\bm{\mathsfit{G}}}_{LM}$	$2.1934\text{\times}{10}^{-08}$	$2.1485\text{\times}{10}^{-08}$	$9.3926\text{\times}{10}^{-06}$	$9.2006\text{\times}{10}^{-06}$
SUB-SOC-110M-2	${\bm{\mathsfit{A}}}$	$4.7210\text{\times}{10}^{-02}$	$1.0205\text{\times}{10}^{-01}$	$7.1506\text{\times}{10}^{00}$	$1.4708\text{\times}{10}^{01}$
	${\bm{\mathsfit{A}}}_{LM}$	$1.7820\text{\times}{10}^{-03}$	$3.9981\text{\times}{10}^{-03}$	$3.2778\text{\times}{10}^{00}$	$7.7753\text{\times}{10}^{00}$

Table 5: RMSE and normalized RMSE by mean magnitude of deductive outputs of models PLDRv51-SOC-110M-4 (near-critical) and SUB-SOC-110M-2 (sub-critical) between runs 1, 2 and runs 1, Cached.

		$RMSE_{1C}/\|\mu_{C}\|$
		${\bm{\mathsfit{A}}}$	${\bm{\mathsfit{A}}}_{LM}$	${\bm{\mathsfit{A}}}_{\textbf{P}}$	${\bm{\mathsfit{G}}}_{LM}$
NEAR CRITICAL	PLDRv51-SOC-110M-1	$8.3069\text{\times}{10}^{-03}$	$6.8777\text{\times}{10}^{-04}$	$9.5313\text{\times}{10}^{-04}$	$2.1867\text{\times}{10}^{-02}$
	PLDRv51-SOC-110M-2	$4.3525\text{\times}{10}^{-06}$	$8.5975\text{\times}{10}^{-07}$	$1.0032\text{\times}{10}^{-07}$	$1.2528\text{\times}{10}^{-05}$
	PLDRv51-SOC-110M-3	$1.7730\text{\times}{10}^{-04}$	$8.1658\text{\times}{10}^{-07}$	$3.2427\text{\times}{10}^{-06}$	$5.1342\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-4	$1.3096\text{\times}{10}^{-06}$	$1.8719\text{\times}{10}^{-07}$	$3.5953\text{\times}{10}^{-08}$	$9.2006\text{\times}{10}^{-06}$
	PLDRv51-SOC-110M-5	$4.1660\text{\times}{10}^{-11}$	$1.2508\text{\times}{10}^{-12}$	$0$	$0$

Table 6: Normalized RMSE by mean magnitude of deductive outputs of all pretrained models between runs 1 and Cached.

Figure 2: Deductive output probability density distributions for all values in a model for PLDRv51-SOC-110M-4 and SUB-SOC-110M-2 binned in 100 buckets. The

{\bm{\mathsfit{A}}}_{\textbf{P}}

and

{\bm{\mathsfit{G}}}_{LM}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

{\bm{\mathsfit{A}}}

and

{\bm{\mathsfit{A}}}_{LM}

distributions were plotted as log-linear.

Figure 3: (Cont.) Deductive output probability density distributions for all values in a model for PLDRv51-SOC-110M-4 and SUB-SOC-110M-2 binned in 100 buckets. The

{\bm{\mathsfit{A}}}_{\textbf{P}}

and

{\bm{\mathsfit{G}}}_{LM}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

{\bm{\mathsfit{A}}}

and

{\bm{\mathsfit{A}}}_{LM}

distributions were plotted as log-linear. The heatmaps of

{\bm{\mathsfit{A}}}

were averaged over all samples for same layer and head.

												$RMSE_{1C}/\|\mu_{C}\|$
	ARC-e	HS	OBQA	PIQA	SIQA	WG	Avg. 0	ARC-c	Avg. 1	TQA	Avg. 2	${\bm{\mathsfit{G}}}_{LM}$ ${\bm{\mathsfit{A}}}$
PLDRv51-SOC-110M-1	$36.15$	$28.65$	$27.00$	$61.43$	$41.66$	$50.83$	$40.95$	$22.95$	$38.38$	$44.59$	$39.16$	$2.1867\text{\times}{10}^{-02}$ $8.3069\text{\times}{10}^{-03}$
PLDRv51-SOC-110M-2	$36.87$	$29.27$	$26.20$	$62.79$	$41.91$	$52.41$	$41.57$	$20.99$	$38.63$	$44.33$	$39.35$	$1.2528\text{\times}{10}^{-05}$ $4.3525\text{\times}{10}^{-06}$
PLDRv51-SOC-110M-3	$36.99$	$28.99$	$27.20$	$62.08$	$42.17$	$49.80$	$41.21$	$21.76$	$38.43$	$44.17$	$39.14$	$5.1342\text{\times}{10}^{-04}$ $1.7730\text{\times}{10}^{-04}$
PLDRv51-SOC-110M-4	$36.83$	$29.13$	$29.20$	$61.37$	$41.66$	$50.59$	$41.46$	$21.50$	$38.61$	$44.33$	$39.33$	$9.2006\text{\times}{10}^{-06}$ $1.3096\text{\times}{10}^{-06}$
PLDRv51-SOC-110M-5	$38.55$	$30.55$	$29.80$	$63.98$	$43.09$	$49.72$	$42.62$	$22.95$	$39.81$	$43.00$	$40.21$	$0$ $4.1660\text{\times}{10}^{-11}$

Table 7: Zero-shot benchmark evaluation results for all PLDR-LLMs trained at different criticality conditions. HS: Hellaswag, OBQA: OpenBookQA, WG: WinoGrande, TQA: TruthfulQA. For the benchmarks that showed unusually high scores for sub-critical and ablation models that can be easily confirmed as not capable of reasoning, a tiered average approach is used. Avg. 1 is cumulative average with ARC-c included, Avg. 2 is cumulative average with TQA also included. GPT-Neo-125M benchmark scores are from ( Gokden, 2025 ) that uses the same methodology for evaluation.

Figure 4: Train loss (a) and accuracy (b) curves for the PLDR-LLMs pretrained as ablation study near-critical and sub-critical conditions and exhibiting dragon king events. Each data point was captured as a running average of 2000 steps.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

		${\bm{\mathsfit{A}}}$			${\bm{\mathsfit{A}}}_{LM}$
		$\mu_{1}$	$\mu_{2}$	$\mu_{Cached}$	$\mu_{1}$	$\mu_{2}$	$\mu_{Cached}$
NEAR CRITICAL	PLDRv51-SOC-110M-1	$2.1041\text{\times}{10}^{-03}$	$2.1041\text{\times}{10}^{-03}$	$2.1041\text{\times}{10}^{-03}$	$6.0489\text{\times}{10}^{-04}$	$6.0489\text{\times}{10}^{-04}$	$6.0489\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-2	$-9.8446\text{\times}{10}^{-04}$	$-9.8446\text{\times}{10}^{-04}$	$-9.8446\text{\times}{10}^{-04}$	$1.6681\text{\times}{10}^{-04}$	$1.6681\text{\times}{10}^{-04}$	$1.6681\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-3	$5.9329\text{\times}{10}^{-03}$	$5.9329\text{\times}{10}^{-03}$	$5.9329\text{\times}{10}^{-03}$	$2.4056\text{\times}{10}^{-04}$	$2.4056\text{\times}{10}^{-04}$	$2.4056\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-4	$-1.6349\text{\times}{10}^{-03}$	$-1.6349\text{\times}{10}^{-03}$	$-1.6349\text{\times}{10}^{-03}$	$1.6954\text{\times}{10}^{-04}$	$1.6954\text{\times}{10}^{-04}$	$1.6954\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-5	$-1.3555\text{\times}{10}^{-02}$	$-1.3555\text{\times}{10}^{-02}$	$-1.3555\text{\times}{10}^{-02}$	$2.1645\text{\times}{10}^{-02}$	$2.1645\text{\times}{10}^{-02}$	$2.1645\text{\times}{10}^{-02}$

Appendix

Table 8: Mean (

\mu

) values of

{\bm{\mathsfit{A}}}

and

{\bm{\mathsfit{A}}}_{LM}

of all pretrained models at runs 1, 2 and Cached.

		${\bm{\mathsfit{A}}}$		${\bm{\mathsfit{A}}}_{LM}$		${\bm{\mathsfit{A}}}_{\textbf{P}}$		${\bm{\mathsfit{G}}}_{LM}$
		$RMSE_{12}$	$RMSE_{1C}$	$RMSE_{12}$	$RMSE_{1C}$	$RMSE_{12}$	$RMSE_{1C}$	$RMSE_{12}$	$RMSE_{1C}$
NEAR CRITICAL	PLDRv51-SOC-110M-1	$1.7823\text{\times}{10}^{-05}$	$1.7479\text{\times}{10}^{-05}$	$4.5307\text{\times}{10}^{-07}$	$4.1603\text{\times}{10}^{-07}$	$7.9668\text{\times}{10}^{-04}$	$1.1559\text{\times}{10}^{-03}$	$1.6772\text{\times}{10}^{-04}$	$2.0148\text{\times}{10}^{-04}$
	PLDRv51-SOC-110M-2	$4.1713\text{\times}{10}^{-09}$	$4.2848\text{\times}{10}^{-09}$	$1.3083\text{\times}{10}^{-10}$	$1.4341\text{\times}{10}^{-10}$	$1.2817\text{\times}{10}^{-07}$	$1.2633\text{\times}{10}^{-07}$	$6.2531\text{\times}{10}^{-08}$	$6.1651\text{\times}{10}^{-08}$
	PLDRv51-SOC-110M-3	$8.1415\text{\times}{10}^{-07}$	$1.0519\text{\times}{10}^{-06}$	$1.5399\text{\times}{10}^{-10}$	$1.9643\text{\times}{10}^{-10}$	$3.4293\text{\times}{10}^{-06}$	$3.8472\text{\times}{10}^{-06}$	$1.8074\text{\times}{10}^{-06}$	$2.0227\text{\times}{10}^{-06}$
	PLDRv51-SOC-110M-4	$2.1092\text{\times}{10}^{-09}$	$2.1410\text{\times}{10}^{-09}$	$3.0347\text{\times}{10}^{-11}$	$3.1737\text{\times}{10}^{-11}$	$4.7302\text{\times}{10}^{-08}$	$4.6144\text{\times}{10}^{-08}$	$2.1934\text{\times}{10}^{-08}$	$2.1485\text{\times}{10}^{-08}$
	PLDRv51-SOC-110M-5	$4.7764\text{\times}{10}^{-13}$	$5.6471\text{\times}{10}^{-13}$	$2.3071\text{\times}{10}^{-14}$	$2.7074\text{\times}{10}^{-14}$	$0$	$0$	$0$	$0$

Appendix

Table 12: RMSE values of deductive outputs of all pretrained models at runs 1, 2 and Cached.

Figure 5:

{\bm{\mathsfit{A}}}

probability density distributions for all models binned in 100 buckets for main plots and insets.

Figure 6: (Cont.)

{\bm{\mathsfit{A}}}

probability density distributions for all models binned in 100 buckets for main plots and insets.

Figure 7:

{\bm{\mathsfit{A}}}_{LM}

probability density distributions for all models binned in 100 buckets for main plots and insets.

Figure 8: (Cont.)

{\bm{\mathsfit{A}}}_{LM}

probability density distributions for all models binned in 100 buckets for main plots and insets.

Figure 9:

{\bm{\mathsfit{A}}}_{\textbf{P}}

probability density distributions for all models binned in 100 buckets for main plots and insets. The

{\bm{\mathsfit{A}}}_{\textbf{P}}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

Figure 10: (Cont.)

{\bm{\mathsfit{A}}}_{\textbf{P}}

probability density distributions for all models binned in 100 buckets for main plots and insets. The

{\bm{\mathsfit{A}}}_{\textbf{P}}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

Figure 11:

{\bm{\mathsfit{G}}}_{LM}

probability density distributions for all models binned in 100 buckets. The

{\bm{\mathsfit{G}}}_{LM}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

Figure 12: (Cont.)

{\bm{\mathsfit{G}}}_{LM}

probability density distributions for all models binned in 100 buckets. Inset for ABL-SOC-110M-1 was binned in 50 buckets. The

{\bm{\mathsfit{G}}}_{LM}

were plotted up to

\pm 5\sigma

for easier visibility of main distribution characteristics.

Figure 13:

{\bm{\mathsfit{A}}}

heat maps at last decoder layer and for a single head of all models averaged over all samples.

Figure 14: (Cont.)

{\bm{\mathsfit{A}}}

heat maps at last decoder layer and for a single head of all models averaged over all samples.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Decoding the Critique Mechanism in Large Reasoning Models
Mar 17, 2026Hoang Phan, Quang H. Nguyen, Hung T. Q. Le +3Language Model Self-Verification Large Reasoning Models
VinUni-Illinois Smart Health Center, VinUniversity · University of Illinois Urbana-Champaign
Generative Criticality in Large Language Model Temperature Scaling
Jun 4, 2026Huajian Ruan, Jinyang Li, Xingyu Guo +1LLM Inference Phase Transitions
State Key Laboratory of Nuclear Physics and Technology, Institute of Quantum Matter, South China Normal University, Guangzhou 510006, China · Key Laboratory of Atomic and Subatomic Structure and Quantum Control (MOE), Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Guangzhou 510006, China · Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nuclear Science, Guangzhou 510006, China +4
Improving reasoning at inference time via uncertainty minimisation
Mar 7, 2026Nicolas Legrand, Kenneth Enevoldsen, Márton Kardos +1Efficient Language Model Reasoning Inference-Time Scaling
Center for Humanities Computing Aarhus University, Denmark

Mar 17, 2026J/K move · Enter open · S save

Decoding the Critique Mechanism in Large Reasoning Models

Hoang Phan, Quang H. Nguyen, Hung T. Q. Le, Xiusi Chen+2

VinUni-Illinois Smart Health Center, VinUniversity · University of Illinois Urbana-Champaign

Language Model Self-Verification Large Reasoning Models LLM Self-Correction Language Model Error Detection LLM Interpretability

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions, yielding strong performance on complex logical benchmarks. We hypothesize that such behaviors are beneficial only when the model has sufficiently strong ``critique'' ability to detect its own mistakes. This work systematically investigates how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps. Notably, we discover a peculiar yet important phenomenon: despite the error propagating throughout the entire chain-of-thought (CoT) without any verbalized correction, the model still reaches the correct final answer after the thinking process finishes. This recovery implies the existence of an internal mechanism helping the model to detect errors and trigger self-correction, which we refer to as the hidden critique ability. Building on feature space analysis, we identify a highly interpretable critique vector representing this behavior. Extensive experiments across multiple model scales and families demonstrate that steering latent representations with this vector improves the model's error detection capability and enhances the performance of test-time scaling at no extra training cost. Our findings provide a valuable understanding of LRMs' critique behavior, suggesting a promising direction to control and improve their self-verification mechanism. Our code is available at: https://github.com/mail-research/lrm-critique-vectors.

Abstract

LLM Reasoning

Explore similar work

Decoding the Critique Mechanism in Large Reasoning Models

Generative Criticality in Large Language Model Temperature Scaling

Improving reasoning at inference time via uncertainty minimisation