BriLLM-MetaPred: A Representation-Prediction Unity Language Model with Adaptive Self-Measurement

Junyuan Liu; Hongyue Lin; Jiale Hong; Yi Zhao; Hai Zhao

cs.LGOct 25, 2026

BriLLM-MetaPred: A Representation-Prediction Unity Language Model with Adaptive Self-Measurement

Authors: Junyuan Liu, Hongyue Lin, Jiale Hong, Yi Zhao, Hai Zhao

Organizations: Institute of Artificial General Intelligence Shanghai Jiao Tong University, Shanghai, China

Abstract

We present BriLLM-MetaPred, the first engineering implementation of the MetaPred machine-learning framework established by a mathematical theory of AGI, instantiated for language modeling. The theory argues that MetaPred is both necessary and sufficient as a machine-learning primitive for general cognition. Taking this theoretical conclusion as a design premise, we report an implementation of the primitive for language modeling. A single continuously evolving dynamical state serves as both the context representation and the state from which predictions are read out. Continuous measurement of its own state reduces finite-depth inconsistency, maintains internal consistency, and provides an endogenous dynamical self-supervision signal in fixed-point form. The single dynamical state of MetaPred intrinsically couples training and inference. MetaPred therefore exhibits symmetry across almost all key elements of machine learning, an intuitive and observable property of the mathematical optimum over the space of all machine-learning paradigms. BriLLM-MetaPred is neither another recurrent language model nor an extension of BriLLM. It provides a bridge between two prior contributions: the construction and motivation of the MetaPred primitive provided by AGI theory, and the SiFu learning paradigm provided by BriLLM (Zhao et al., 2025). The latter's node-signal-flow design enables the primitive to run on available software and hardware accelerators. Our implementation of BriLLM-MetaPred achieves predictive performance comparable to a GPT baseline of the same size. This provides initial evidence for the engineering feasibility of the MetaPred framework established by the mathematical theory of AGI. MetaPred is the first machine-learning paradigm derived entirely from mathematical theory rather than assembled and incrementally improved through human design---and perhaps the last such paradigm.

Figures & tables

	Existing machine learning	BriLLM	BriLLM-MetaPred
Carrier-dynamics relationship	Model architecture and dynamical state are strictly aligned	The model carrier (a fully connected graph topology) and dynamical state need not align in shape; the dynamics can evolve flexibly within the maximal space allowed by the carrier
Structural carrier	Dedicated architecture (e.g., Transformer)	General fully connected graph topology with bidirectional connections between nodes
Semantic representation	Fixed or learnable vectors carried by separate components	Nodes directly map to semantics, with encoding in the spatial dimension	Encoded entirely by the dynamical state $\mathbf{S}_{t,u}$ , without a separate representation component
Prediction mechanism	Vector transformations passed from layer to layer	Dynamic emergence through signal propagation among nodes	Direct readout from the same dynamical state
Representation-prediction relationship	Performed independently in separate stages	Representation is carried by nodes; prediction is determined by dynamics	Fully unified: the same state performs both representation and prediction
Self-measurement	None	None	Endogenous finite-step self-comparison providing fixed-point self-supervision

Table 1: Comparison of machine-learning paradigms.

Figure 1: The MetaPred model architecture

Figure 2: Dynamical evolution in MetaPred

Feature	Train	Frozen	Adaptive
External task loss used for parameter updates	Enabled	Disabled	Disabled
Self-measurement diagnostics	Enabled	Enabled	Enabled
State correction	Enabled	Disabled	Enabled
Fast-parameter updates	Enabled	Disabled	Enabled
Slow-parameter updates	Enabled (per batch)	Disabled	Disabled
Control-parameter updates	Enabled (delayed PPO updates)	Disabled	Disabled

Table 2: Comparison of the three operating modes

Parameter family	Scope	Update source	Trigger condition
Slow parameters $\bm{\theta}$	Input projection matrix $E_{\text{in}}$ , output projection matrices $W_{\text{out}},b_{\text{out}}$ ; slot projection vectors $\{s_{k},p_{k}\}$ ; modal reference vectors $\{\mathbf{m}_{k}\}$ ; all weights of the five stages of one-step evolution; readout-path weights; slow prediction parameters $p_{\bm{\theta}}$ and slow correction parameters $c_{\bm{\theta}}$	Direct gradients of $\mathcal{L}_{\text{outer}}$ (AdamW)	Slow-parameter update boundary $s$
Fast parameters $\bm{\phi}_{u}$	Fast prediction parameters $p_{\bm{\phi}_{u}}$ and fast correction parameters $c_{\bm{\phi}_{u}}$	Direct gradients of $\mathcal{L}_{\text{inner}}$ (clipped SGD)	Each controller request within a sequence
Control parameters $\bm{\psi}$	All controller-network parameters, including the shared encoder, six action heads, and value head	PPO policy gradients (independent Adam optimizer)	When the policy trajectory matures

Table 3: Assignment and update mechanisms of the three parameter families

Research question	Experimental setup	Main observation
Language-modeling performance	13M continuous-execution training regime versus a same-size GPT baseline	Validation PPL
Role of internal mechanisms	13M mechanism ablations in the sparse-auxiliary training regime	Prediction loss in Frozen mode
Training at scale and generation	129M sparse-auxiliary training regime, two window lengths	Training endpoints, validation metrics, and generation examples

Table 4: Overview of the experiments.

Seed	GPT baseline PPL $\downarrow$	BriLLM-MetaPred PPL $\downarrow$	Relative PPL reduction
43	278.847	$\bm{229.451}$	$\bm{17.71\%}$
44	279.355	$\bm{248.554}$	$\bm{11.03\%}$

Table 5: Model performance comparison on the BookCorpus validation set under equal training budgets.

Training variant	Validation PPL $\downarrow$	$\Delta$ PPL
Full model	$36.700\pm 0.832$	–
Modal nodes disconnected	$37.887\pm 0.734$	$-1.187$
Fast-parameter updates and state correction disabled	$36.657\pm 0.599$	$+0.043$
Fixed-point objective removed	$36.678\pm 0.906$	$+0.022$

Table 6: Core-mechanism ablations: mean and sample standard deviation over three seeds;

\Delta\mathrm{PPL}

is defined as the full-model PPL minus the variant PPL.

Metric	Frozen	Adaptive
NLL $\downarrow$	8.512188	8.512103
Actual state evolution count	8,256	8,256
Simulation call count	3,120	3,120
Simulation-based measurement count	528	528
Direct measurement count	64	64
Fast-parameter update count	0	$\bm{1,056}$

Table 7: Runtime diagnostics for an intermediate BookCorpus checkpoint. Seed 42, step 100, 16 validation blocks, and 7,936 evaluation tokens; GPU memory is the approximate peak allocated amount.

Figure 3: Training NLL of 13M BriLLM-MetaPred and the GPT baseline on BookCorpus (seed 42).

Input $\rightarrow$ output (tokens)	Training set (tokens)	Validation set (tokens)	BPC $\downarrow$	PPL $\downarrow$
$16\rightarrow 16$	470,404,384	16,384	3.883050	$\bm{14.754158}$
$512\rightarrow 512$	133,826,560	7,635,968	4.766332	$\bm{27.215033}$

Table 8: 129M BriLLM-MetaPred completes training and evaluation at both training window lengths, each with seed 42.

Input prompt	BriLLM-MetaPred	GPT baseline
01 中国队在第二场比赛中3比0大胜	。此外，中国队还有1000万人次，中国队在2008年10月份以来，中国队在2008年10月份以来首次	日本队，中国队在第三场比赛中以1比0战胜日本队，中国队在第四场比赛中以1比0战胜日本队，中国队在第四
02 欧盟决定为经济疲弱的成员国提供总额达	到1.2亿美元。此外，美国政府将继续保持稳定增长。此外，美国政府将在2010年底前将有望进一步加强国	到1000亿欧元的纾困金，这是欧盟决定为经济增长提供的最大障碍。欧盟委员会主席巴罗佐表示，欧盟委员会
03 最近中医膏方养身保健成为了	保障房的安全保障，保障房建设保障性安居工程建设安全保障工程建设安全保障安全保障措施保障安全保障安全保	一个新的消费方式，但是，这种方式的确是一种新的消费方式，但是，这种方式的确是一种新的消费方式，但是，
04 消费者在购买家具的时候非常注重	要的原因是，这些都是在网上看到的。”“在这个时候，我们看到了这个消息，我们看到了这个时候，我们看到了	家具的质量，而且家具的质量也是非常重要的。”在家具行业，家具的质量是否符合国家标准的标准，家具的质量
05 儿童房甲醛超标5倍，即有引发	网络购物网络购物网站，网络购物网站、网络购物网站、网络网络建设、网络网络建设网络建设网络建设网络建设	甲醛的可能。”“甲醛超标”是一种新型甲醛释放量超标的新型甲醛释放量超标，这种新型甲醛释放量超标的新型
06 双色球一等奖的奖金有望达到	1000万美元。而在这个过程中，我们认为，这样的结构性机会是一个非常有效的解决方案。“我们的目标是在	1000万元。据悉，这是中国彩票双色球第10009期开奖的一张彩票，中奖号码为“0311111111

Table 9: Chinese generation comparison with the same inputs.

BriLLM-MetaPred	GPT baseline
01 Common input: gregory zorzos [/R] 100 alphanumeric crosswords [/R] gregory zor
_- one and the same one of your body than to see the the room - at a day of his feet ? ” [/R] ” , ” no , the house and the house of the side . and the day of the world . [/R] ” [/R] one of his life , we were in a different day , and the way you ’re sure , ” if the way had out for a long time to do , we do n’t think it . i have the door and did n’t be back at the world . but it was some important to be more . and this we ’m not a bit of the rest , and the room was in the small	- the , [/R] he said _ [/R] , as but it . [/R] _ ” yes , and the one . [/R] chapter 4 [/R] the ground , a good - 2 : [/R] the floor of the other the air . [/R] it , and the day that ’ [/R] ’ [/R] ’ : * * [/R] the last day , this ’s an year is the house . he is his first and he was not not like the most of the door and the same . if i have . [/R] ” i ’ll have no good _ , the air he was a bit of them from the most of the first , and i have to be going to get a lot
02 Common input: lover ’s knot . ” [/R] ” i ’m serious , morris . i want a
second world , i do n’t know that it ’s been a new , ” _ _ _ _ he would see a good man and do what it is to be a moment . he was a few of the floor of the other , and how happened to you do n’t be . ” you know , she ’s not good . [/R] i need the last place i know who ” , ” i ’m a idea what she asked . ” [/R] ” what was there or this he was a few hundred than i have just to understand her . ” [/R] ” she said , ” if you have it . _ ” [/R] ” she is a better more than a	right . ” i do n’t see you , ” i do n’t have to it was too . . ” she would have to a a time to do you and i know … you want to be a little . ” i ’m n’t know … ” [/R] ” [/R] ” [/R] ” [/R] you have to know you . ” [/R] ” i ’re n’t go the same , her hair and i ’m n’t want to take a more in a good . you had be one than about you . ” why , ” she wanted to talk with a place . ” she was a little to him , he was still like a lot to go to i said . i
03 Common input: ibn sa’ûd assembled muslims and , slandering them , said ,
” i have come to the one . the long thing . [/R] ” i are a little part it is on to be an . the new . [/R] we ’re all the day . ” what he ’s not so , you ’re going to be going to give a time if you do . ” [/R] ” we ’ve been for what was to be . [/R] ” i did n’t know that , he had seen that all the ground . ” [/R] ” i had my own , ” ’s sure she ’d never been a good man . [/R] ” even that , to have not much to get on my father . ” [/R] * ” he said . ” [/R]	and the two a great , they are n’t keep , that is a most of the other , [/R] and for your way that was to the way to you have to the same , that ’s just a very more than you are you to be the way to have the table of the rest of the people is . [/R] but this and they were the house , and a the woman , the first of the little day , who in a whole . [/R] that had been the other ; . you were and the other of the table . [/R] no , when it , in the the next , he was a life , , the time , and her for

Table 10: English generation comparison with the same inputs.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Experiment	Parameters	Node count/width	Vocabulary	Actual training tokens
BookCorpus continuous-execution training regime	13.2226M	96/132	40,478 BPE	25,014,272
THUCNews sparse-auxiliary training regime 13M	13.0945M	256/256	6,512	25,000,960
THUCNews sparse-auxiliary training regime 129M	129.3854M	256/928	6,512	470,404,384

Appendix

Table 11: Main experimental configurations. Parameter counts are based on registered parameters.

Predictive Representation Learning

Predictive representation learning trains models to encode useful structure by predicting unseen or future representations from context.

See all 173 papers

Peer-Predictive Self-Training for Language Model Reasoning
Apr 14, 2026Shi Feng, Hanlin Zhang, Fan Nie +2Self-Training for Language Models Language Model Post-Training
Harvard University · Stanford University
Meta Models: Models that Model Learning
Oct 27, 2026Peihang Li, Ziming LiuLanguage Model Performance Prediction Meta-Learning
MetaCircle · The University of Hong Kong · University of California, Berkeley +2
Think $^{2}$ : Grounded Metacognitive Reasoning in Large Language Models
Feb 21, 2026Abraham Paul Elenjical, Vivek Hruday Kavuri, Vasudeva VarmaLanguage Model Self-Assessment Metacognition in Language Models
IIIT Hyderabad

Apr 14, 2026J/K move · Enter open · S save

Peer-Predictive Self-Training for Language Model Reasoning

Shi Feng, Hanlin Zhang, Fan Nie, Sham Kakade+1

Harvard University · Stanford University

Self-Training for Language Models Language Model Post-Training Language Model Self-Improvement

Open paper AskPDF arXiv

Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by using a cross-model aggregate response as an internal training signal. Given a prompt, models generate responses sequentially; the final aggregated answer, which is often more reliable than individual responses in practice, serves as an internal reference for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates: responses already aligned with the aggregate receive smaller updates, while less informative or misaligned responses receive larger ones. On mathematical reasoning benchmarks, including SimulEq, MATH-500-Numeric, and MultiArith, PST improves exact-match accuracy by 2.2--4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen2.5-1.5B, and reduces the average generator--verifier gap (GV-Gap) by 26--40%, while requiring no external supervision, no teacher--student hierarchy, and only cross-model interactions. These results suggest that peer-predictive feedback from cross-model generations can provide an effective mechanism for self-supervised language-model improvement.

Abstract

Predictive Representation Learning

Explore similar work

Peer-Predictive Self-Training for Language Model Reasoning

Meta Models: Models that Model Learning

Think2^{2}2: Grounded Metacognitive Reasoning in Large Language Models

Think $^{2}$ : Grounded Metacognitive Reasoning in Large Language Models