cs.LGOct 25, 2026

BriLLM-MetaPred: A Representation-Prediction Unity Language Model with Adaptive Self-Measurement

Authors: Junyuan Liu, Hongyue Lin, Jiale Hong, Yi Zhao, Hai Zhao

Organizations: Institute of Artificial General Intelligence Shanghai Jiao Tong University, Shanghai, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present BriLLM-MetaPred, the first engineering implementation of the MetaPred machine-learning framework established by a mathematical theory of AGI, instantiated for language modeling. The theory argues that MetaPred is both necessary and sufficient as a machine-learning primitive for general cognition. Taking this theoretical conclusion as a design premise, we report an implementation of the primitive for language modeling. A single continuously evolving dynamical state serves as both the context representation and the state from which predictions are read out. Continuous measurement of its own state reduces finite-depth inconsistency, maintains internal consistency, and provides an endogenous dynamical self-supervision signal in fixed-point form. The single dynamical state of MetaPred intrinsically couples training and inference. MetaPred therefore exhibits symmetry across almost all key elements of machine learning, an intuitive and observable property of the mathematical optimum over the space of all machine-learning paradigms. BriLLM-MetaPred is neither another recurrent language model nor an extension of BriLLM. It provides a bridge between two prior contributions: the construction and motivation of the MetaPred primitive provided by AGI theory, and the SiFu learning paradigm provided by BriLLM (Zhao et al., 2025). The latter's node-signal-flow design enables the primitive to run on available software and hardware accelerators. Our implementation of BriLLM-MetaPred achieves predictive performance comparable to a GPT baseline of the same size. This provides initial evidence for the engineering feasibility of the MetaPred framework established by the mathematical theory of AGI. MetaPred is the first machine-learning paradigm derived entirely from mathematical theory rather than assembled and incrementally improved through human design---and perhaps the last such paradigm.

Figures & tables

 Existing machine learningBriLLMBriLLM-MetaPred
Carrier-dynamics relationshipModel architecture and dynamical state are strictly alignedThe model carrier (a fully connected graph topology) and dynamical state need not align in shape; the dynamics can evolve flexibly within the maximal space allowed by the carrier
Structural carrierDedicated architecture (e.g., Transformer)General fully connected graph topology with bidirectional connections between nodes
Semantic representationFixed or learnable vectors carried by separate componentsNodes directly map to semantics, with encoding in the spatial dimensionEncoded entirely by the dynamical state St,u\mathbf{S}_{t,u} , without a separate representation component
Prediction mechanismVector transformations passed from layer to layerDynamic emergence through signal propagation among nodesDirect readout from the same dynamical state
Representation-prediction relationshipPerformed independently in separate stagesRepresentation is carried by nodes; prediction is determined by dynamicsFully unified: the same state performs both representation and prediction
Self-measurementNoneNoneEndogenous finite-step self-comparison providing fixed-point self-supervision
Table 1: Comparison of machine-learning paradigms.
Figure 1: The MetaPred model architecture
Figure 2: Dynamical evolution in MetaPred
FeatureTrainFrozenAdaptive
External task loss used for parameter updatesEnabledDisabledDisabled
Self-measurement diagnosticsEnabledEnabledEnabled
State correctionEnabledDisabledEnabled
Fast-parameter updatesEnabledDisabledEnabled
Slow-parameter updatesEnabled (per batch)DisabledDisabled
Control-parameter updatesEnabled (delayed PPO updates)DisabledDisabled
Table 2: Comparison of the three operating modes
Parameter familyScopeUpdate sourceTrigger condition
Slow parameters θ\bm{\theta}Input projection matrix EinE_{\text{in}} , output projection matrices Wout,boutW_{\text{out}},b_{\text{out}} ; slot projection vectors {sk,pk}\{s_{k},p_{k}\} ; modal reference vectors {mk}\{\mathbf{m}_{k}\} ; all weights of the five stages of one-step evolution; readout-path weights; slow prediction parameters pθp_{\bm{\theta}} and slow correction parameters cθc_{\bm{\theta}}Direct gradients of Louter\mathcal{L}_{\text{outer}} (AdamW)Slow-parameter update boundary ss
Fast parameters ϕu\bm{\phi}_{u}Fast prediction parameters pϕup_{\bm{\phi}_{u}} and fast correction parameters cϕuc_{\bm{\phi}_{u}}Direct gradients of Linner\mathcal{L}_{\text{inner}} (clipped SGD)Each controller request within a sequence
Control parameters ψ\bm{\psi}All controller-network parameters, including the shared encoder, six action heads, and value headPPO policy gradients (independent Adam optimizer)When the policy trajectory matures
Table 3: Assignment and update mechanisms of the three parameter families
Research questionExperimental setupMain observation
Language-modeling performance13M continuous-execution training regime versus a same-size GPT baselineValidation PPL
Role of internal mechanisms13M mechanism ablations in the sparse-auxiliary training regimePrediction loss in Frozen mode
Training at scale and generation129M sparse-auxiliary training regime, two window lengthsTraining endpoints, validation metrics, and generation examples
Table 4: Overview of the experiments.
SeedGPT baseline PPL ↓\downarrowBriLLM-MetaPred PPL ↓\downarrowRelative PPL reduction
43278.847229.451\bm{229.451}17.71%\bm{17.71\%}
44279.355248.554\bm{248.554}11.03%\bm{11.03\%}
Table 5: Model performance comparison on the BookCorpus validation set under equal training budgets.
Training variantValidation PPL ↓\downarrowΔ\Delta PPL
Full model36.700±0.83236.700\pm 0.832–
Modal nodes disconnected37.887±0.73437.887\pm 0.734−1.187-1.187
Fast-parameter updates and state correction disabled36.657±0.59936.657\pm 0.599+0.043+0.043
Fixed-point objective removed36.678±0.90636.678\pm 0.906+0.022+0.022
Table 6: Core-mechanism ablations: mean and sample standard deviation over three seeds; ΔPPL\Delta\mathrm{PPL} is defined as the full-model PPL minus the variant PPL.
MetricFrozenAdaptive
NLL ↓\downarrow8.5121888.512103
Actual state evolution count8,2568,256
Simulation call count3,1203,120
Simulation-based measurement count528528
Direct measurement count6464
Fast-parameter update count01,056\bm{1,056}
Table 7: Runtime diagnostics for an intermediate BookCorpus checkpoint. Seed 42, step 100, 16 validation blocks, and 7,936 evaluation tokens; GPU memory is the approximate peak allocated amount.
Figure 3: Training NLL of 13M BriLLM-MetaPred and the GPT baseline on BookCorpus (seed 42).
Input →\rightarrow output (tokens)Training set (tokens)Validation set (tokens)BPC ↓\downarrowPPL ↓\downarrow
16→1616\rightarrow 16470,404,38416,3843.88305014.754158\bm{14.754158}
512→512512\rightarrow 512133,826,5607,635,9684.76633227.215033\bm{27.215033}
Table 8: 129M BriLLM-MetaPred completes training and evaluation at both training window lengths, each with seed 42.
Input promptBriLLM-MetaPredGPT baseline
01 中国队在第二场比赛中3比0大胜。此外，中国队还有1000万人次，中国队在2008年10月份以来，中国队在2008年10月份以来首次日本队，中国队在第三场比赛中以1比0战胜日本队，中国队在第四场比赛中以1比0战胜日本队，中国队在第四
02 欧盟决定为经济疲弱的成员国提供总额达到1.2亿美元。此外，美国政府将继续保持稳定增长。此外，美国政府将在2010年底前将有望进一步加强国到1000亿欧元的纾困金，这是欧盟决定为经济增长提供的最大障碍。欧盟委员会主席巴罗佐表示，欧盟委员会
03 最近中医膏方养身保健成为了保障房的安全保障，保障房建设保障性安居工程建设安全保障工程建设安全保障安全保障措施保障安全保障安全保一个新的消费方式，但是，这种方式的确是一种新的消费方式，但是，这种方式的确是一种新的消费方式，但是，
04 消费者在购买家具的时候非常注重要的原因是，这些都是在网上看到的。”“在这个时候，我们看到了这个消息，我们看到了这个时候，我们看到了家具的质量，而且家具的质量也是非常重要的。”在家具行业，家具的质量是否符合国家标准的标准，家具的质量
05 儿童房甲醛超标5倍，即有引发网络购物网络购物网站，网络购物网站、网络购物网站、网络网络建设、网络网络建设网络建设网络建设网络建设甲醛的可能。”“甲醛超标”是一种新型甲醛释放量超标的新型甲醛释放量超标，这种新型甲醛释放量超标的新型
06 双色球一等奖的奖金有望达到1000万美元。而在这个过程中，我们认为，这样的结构性机会是一个非常有效的解决方案。“我们的目标是在1000万元。据悉，这是中国彩票双色球第10009期开奖的一张彩票，中奖号码为“0311111111
Table 9: Chinese generation comparison with the same inputs.
BriLLM-MetaPredGPT baseline
01 Common input: gregory zorzos [/R] 100 alphanumeric crosswords [/R] gregory zor
_- one and the same one of your body than to see the the room - at a day of his feet ? ” [/R] ” , ” no , the house and the house of the side . and the day of the world . [/R] ” [/R] one of his life , we were in a different day , and the way you ’re sure , ” if the way had out for a long time to do , we do n’t think it . i have the door and did n’t be back at the world . but it was some important to be more . and this we ’m not a bit of the rest , and the room was in the small- the , [/R] he said _ [/R] , as but it . [/R] _ ” yes , and the one . [/R] chapter 4 [/R] the ground , a good - 2 : [/R] the floor of the other the air . [/R] it , and the day that ’ [/R] ’ [/R] ’ : * * [/R] the last day , this ’s an year is the house . he is his first and he was not not like the most of the door and the same . if i have . [/R] ” i ’ll have no good _ , the air he was a bit of them from the most of the first , and i have to be going to get a lot
02 Common input: lover ’s knot . ” [/R] ” i ’m serious , morris . i want a
second world , i do n’t know that it ’s been a new , ” _ _ _ _ he would see a good man and do what it is to be a moment . he was a few of the floor of the other , and how happened to you do n’t be . ” you know , she ’s not good . [/R] i need the last place i know who ” , ” i ’m a idea what she asked . ” [/R] ” what was there or this he was a few hundred than i have just to understand her . ” [/R] ” she said , ” if you have it . _ ” [/R] ” she is a better more than aright . ” i do n’t see you , ” i do n’t have to it was too . . ” she would have to a a time to do you and i know … you want to be a little . ” i ’m n’t know … ” [/R] ” [/R] ” [/R] ” [/R] you have to know you . ” [/R] ” i ’re n’t go the same , her hair and i ’m n’t want to take a more in a good . you had be one than about you . ” why , ” she wanted to talk with a place . ” she was a little to him , he was still like a lot to go to i said . i
03 Common input: ibn sa’ûd assembled muslims and , slandering them , said ,
” i have come to the one . the long thing . [/R] ” i are a little part it is on to be an . the new . [/R] we ’re all the day . ” what he ’s not so , you ’re going to be going to give a time if you do . ” [/R] ” we ’ve been for what was to be . [/R] ” i did n’t know that , he had seen that all the ground . ” [/R] ” i had my own , ” ’s sure she ’d never been a good man . [/R] ” even that , to have not much to get on my father . ” [/R] * ” he said . ” [/R]and the two a great , they are n’t keep , that is a most of the other , [/R] and for your way that was to the way to you have to the same , that ’s just a very more than you are you to be the way to have the table of the rest of the people is . [/R] but this and they were the house , and a the woman , the first of the little day , who in a whole . [/R] that had been the other ; . you were and the other of the table . [/R] no , when it , in the the next , he was a life , , the time , and her for
Table 10: English generation comparison with the same inputs.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ExperimentParametersNode count/widthVocabularyActual training tokens
BookCorpus continuous-execution training regime13.2226M96/13240,478 BPE25,014,272
THUCNews sparse-auxiliary training regime 13M13.0945M256/2566,51225,000,960
THUCNews sparse-auxiliary training regime 129M129.3854M256/9286,512470,404,384
Appendix
Table 11: Main experimental configurations. Parameter counts are based on registered parameters.
More in this topic

Predictive Representation Learning

Predictive representation learning trains models to encode useful structure by predicting unseen or future representations from context.

See all 173 papers

Explore similar work

CardsList
Apr 14, 2026

Peer-Predictive Self-Training for Language Model Reasoning

Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by using a cross-model aggregate response as an internal training signal. Given a prompt, models generate responses sequentially; the final aggregated answer, which is often more reliable than individual responses in practice, serves as an internal reference for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates: responses already aligned with the aggregate receive smaller updates, while less informative or misaligned responses receive larger ones. On mathematical reasoning benchmarks, including SimulEq, MATH-500-Numeric, and MultiArith, PST improves exact-match accuracy by 2.2--4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen2.5-1.5B, and reduces the average generator--verifier gap (GV-Gap) by 26--40%, while requiring no external supervision, no teacher--student hierarchy, and only cross-model interactions. These results suggest that peer-predictive feedback from cross-model generations can provide an effective mechanism for self-supervised language-model improvement.

Shi Feng, Hanlin Zhang, Fan Nie +2

Harvard University · Stanford University

Self-Training for Language ModelsLanguage Model Post-TrainingLanguage Model Self-Improvement
AskPDF
Oct 27, 2026

Meta Models: Models that Model Learning

Every neural-network training run produces not only a model, but also evidence about how models learn. Can these accumulated experiments enable us to predict and compare new training runs before executing them? We study meta-models: models that predict the learning behavior of other models. From this perspective, final-performance prediction, learning-curve prediction, and hyperparameter selection are different queries to the same training response function. We take a first, humble empirical step toward this goal: treating real language-model training records as native numeric data, we find that an off-the-shelf tabular in-context learner, TabPFN, can match or outperform a fine-tuned LLM meta-model in multiple evaluated settings. Whereas recent work fine-tunes LLMs on tokenized configurations at a cost of tens to hundreds of GPU-hours, TabPFN requires seconds of fit-and-predict time without task-specific weight updates. We argue that meta-modeling is better treated as its own numeric modality rather than as a text task for LLMs. Going one step further, augmenting TabPFN with Kolmogorov-Arnold Networks (KANs) further improves accuracy on selected prediction tasks. Finally, we pre-registered a forecast of the Marin Hero Run, a 535B-parameter public training run, whose ground truth is revealed point by point as training progresses, putting meta-model predictions to the test against training outcomes not yet observed at the time of forecasting.

Peihang Li, Ziming Liu

MetaCircle · The University of Hong Kong · University of California, Berkeley +2

Language Model Performance PredictionMeta-LearningHyperparameter Optimization
AskPDF
Feb 21, 2026

Think2^{2}: Grounded Metacognitive Reasoning in Large Language Models

Large Language Models (LLMs) demonstrate strong reasoning performance, yet their ability to reliably monitor, diagnose, and correct their own errors remains limited. We introduce a psychologically grounded metacognitive framework that operationalizes Ann Brown's regulatory cycle (Planning, Monitoring, and Evaluation) as a structured prompting architecture, and study its integration within a lightweight dual-process MetaController for adaptive effort allocation. Across diverse reasoning and diagnostic benchmarks (GSM8K, CRUXEval, MBPP, AIME, CorrectBench, and TruthfulQA) using Llama-3 and Qwen-3 (8B), explicit regulatory structuring substantially improves error diagnosis and yields a threefold increase in successful self-correction. Blinded human evaluations over 580 query pairs show an 84% aggregate preference for trustworthiness and metacognitive self-awareness over standard and Chain-of-Thought baselines. Grounding LLM reasoning in established cognitive theory offers a principled path toward more transparent and diagnostically robust AI systems.

Abraham Paul Elenjical, Vivek Hruday Kavuri, Vasudeva Varma

IIIT Hyderabad

Language Model Self-AssessmentMetacognition in Language ModelsLLM Self-CorrectionLLM ReliabilityAdaptive Computation
AskPDF