4-Tensor Attention Model for Semantic Physical Reality

Jongwook Kim; Sangheon Yun

cs.LGOct 8, 2026

4-Tensor Attention Model for Semantic Physical Reality

Authors: Jongwook Kim, Sangheon Yun

Organizations: IndigoWave, Center for Quantum Spacetime, Sogang University, 35 Baekbeom-ro, Mapo-gu, Seoul 04107, Republic of Korea

Abstract

We describe a 4-tensor attention model that predicts the next semantic state of a scene, for video generation and robot planning. A window of states has positions (x, t) and two fibers, a semantic fiber and a temporal-context fiber, and one softmax normalizes attention jointly over the window. Frames and an agent's situation are written as those states; the encoder, the renderer, and the planner remain outside the update. To test the update on its own, we train on ROCStories, where each window poses the same next-sentence task at the semantic layer. On the validation split, with one seed per setting, the last-sentence cross-entropy on the three matched settings is lower for the 4-tensor model than for a free-running one-dimensional transformer by 5.3% at H=2, L=2, by 2.6% at H=4, L=2, and by 2.4% at H=4, L=3. At H=4, L=2 the parameter counts are nearly the same, 172.5M and 175.9M. On the same two GPUs that 4-tensor run finished in 2.4 hours and the baseline run in 45.2 hours; the baseline is trained by free-running decoding, one sequential forward pass per target token.

Figures & tables

Panel of Figure 1 — Figure 1: The intended generative pipeline. Frames are compressed into semantic states, the 4-tensor attention update predicts the next states, and the predicted states are rendered back into frames. This paper implements and evaluates only the 4-tensor attention block (teal).

Object	Shape	Index order
token_ids , $Y$	$(B,T,N)$	$[b,t,x]$
$X,Q,K,V,\mathrm{Attn}$	$(B,N,T,d_{x},d_{t})$	$[b,x,t,n,q]$
pad_mask	$(B,N,T)$	$[b,x,t]$ , true on a PAD key
$W^{Q},W^{K},W^{V}$	$(d_{x},d_{t},d_{x},d_{t})$	$[m,p,n,q]$
scores, attention weights	$(B,N,T,N,T)$	$[b,x,t,x^{\prime},t^{\prime}]$
logits	$(B,N,T,\|V\|)$	$[b,x,t,v]$

Table 1: Storage layout of the reduced-to-practice pipeline. The batch axis

B

is present on every public tensor, including the case

B=1

Stage	Tensor	Storage shape
Step 1	$X$	$(B,N,T,d_{x},d_{t})$
Steps 2–5, summed over heads	$\mathrm{Attn}$	$(B,N,T,d_{x},d_{t})$
After both residual adds	$X^{\mathrm{out}}$	$(B,N,T,d_{x},d_{t})$
Readout-boundary norm	$X^{\mathrm{norm}}$	$(B,N,T,d_{x},d_{t})$
Tied readout	logits	$(B,N,T,\|V\|)$
Shifted targets	$Y$	$(B,T,N)$

Table 2: Fiber role through one reduced-to-practice forward pass. The pair

(n,q)

is eliminated at the readout and does not appear in

\mathcal{L}

	4-tensor	Baseline
Width of each head	full $(d_{x},d_{t})$ fiber, $d_{x}d_{t}$	$d/H$
Head merge	sum, no output projection	concatenation, then $W^{O}$
Attention parameters per layer	$3Hd^{2}$	$4d^{2}$
MLP parameters per layer	$8d^{2}$	$8d^{2}$
Token table and readout	$\|V\|\,(d_{x}+2d_{t})$	$(\|V\|+2)\,d$ , tied

Table 3: How a head count enters the parameter count.

d=d_{x}d_{t}

. Biases and LayerNorm are omitted. The baseline attention count is the reported case

H>1

, three maps of size

d\times d

plus

W^{O}

. The 4-tensor count is three full-fiber maps on each of

H

heads and no output projection. The 4-tensor token term is

E_{\mathrm{sem}}

E_{\mathrm{temp}}

, and the readout factor

W^{\prime}

Item	Value
Corpus	ROCStories, window $5\times 16$ , condition on four sentences
Split	$97{,}929$ train / $3{,}742$ validation
Vocabulary	$36{,}711$ ; baseline $36{,}713$ with [BOS] , [EOS]
Width	$d_{x}=64$ , $d_{t}=32$ ; baseline $d=2048$ , $d_{ff}=8192$
Dropout	$0.1$ on both residual branches
Optimizer	AdamW, learning rate $3\times 10^{-4}$ , weight decay $0.01$

Table 4: Shared setup of the runs.

$H$	$L$	seed	parameters	epoch	$\mathcal{L}_{S(4)}$	acc. (%)	$\Delta$
2	2	43	122.2M / 175.9M	4/9 / 30/39	5.562 / 5.873	16.8 / 7.3	0.311
4	2	43	172.5M / 175.9M	6/11 / 45/50	5.554 / 5.701	16.0 / 10.5	0.147
4	3	42	256.4M / 226.2M	6/11 / 29/35	5.531 / 5.667	15.4 / 11.1	0.136

Table 5: Matched settings at the selected checkpoint. Each entry is 4-tensor / baseline. Epoch is selected/last.

\mathcal{L}_{S(4)}

is the mean cross-entropy on the

32{,}173

non-padding tokens of the last sentence, and accuracy is the argmax match rate on the same tokens.

\Delta

is defined in the text.

Panel of Figure 2 — Figure 2: Validation cross-entropy of the non-padding tokens of $S(4)$ . This is the sentence absent from the input: the model reads $X(3)\leftarrow\{S(0),S(1),S(2),S(3)\}$ , and $S(4)$ is not in that window. The solid curve is the 4-tensor model and the dashed curve is the free-running baseline. Open circles are the checkpoints in table 5 . Epoch 1 of each baseline lies above the plotted range ( $12.9$ – $15.0$ ). The horizontal axis is the epoch index, not wall-clock time: at $H{=}4$ , $L{=}2$ those runs took $2.4$ and $45.2$ hours.

Panel of Figure 3 — Figure 3: Training cross-entropy on separate axes. The upper row is the 4-tensor model trained in these experiments. On a training window the input is $X(3)\leftarrow\{S(0),S(1),S(2),S(3)\}$ , and the loss is the mean cross-entropy of the shifted window $X(4)\leftarrow\{S(1),S(2),S(3),S(4)\}$ . The lower row is the baseline, whose loss is the free-running loss on $S(4)$ , including [EOS]. Open circles are the checkpoints in table 5 . Epoch 1 of each baseline lies above the plotted range ( $29.1$ – $32.6$ ). The horizontal axis is the epoch index, not wall-clock time.

Panel of Figure 4 — Figure 4: Validation cross-entropy on separate axes. The upper row is the 4-tensor model on the validation split, with the same windows as fig. 3 : the input is $X(3)\leftarrow\{S(0),S(1),S(2),S(3)\}$ , and the loss is the mean cross-entropy of $X(4)\leftarrow\{S(1),S(2),S(3),S(4)\}$ . The lower row is the baseline selection loss on $S(4)$ , including [EOS]. Open circles are the checkpoints in table 5 , which are the minima of these curves. Every logged epoch lies inside the plotted range. The horizontal axis is the epoch index, not wall-clock time.

Explore similar work

Sep 30, 2026cs.CV

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model

3.4\times

its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.

Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3

Stability AI · Karlsruhe Institut für Technologie

Visual Tokenization Autoregressive Video Generation Video Token Compression

Ask PDF

Jun 11, 2026cs.CV

Diffusion Transformer World-Action Model for AV Scene Prediction

Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls, enabling planning and simulation without real-world rollouts, but at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead: they reward a blurry regression mean over a realistic prediction. We confront this with a compact latent world model that, given the present front-camera latent and a sequence of ego-actions, predicts future scene latents a frozen decoder renders to

256 \times 256

frames up to 8 seconds ahead, evaluated on 150 held-out nuScenes scenes. We first benchmark where to predict: across six frozen encoders spanning four representation families, V-JEPA2 with temporal context reduces steering RMSE by 40% over the best single-frame encoder. We then train a latent Diffusion Transformer (DiT) and, through a controlled diagnosis, identify the four ingredients it needs: spatial tokens, the

x_0

objective, residual anchoring, and sampling matched to target uncertainty. In a Stable-Diffusion-VAE encode-predict-decode pipeline we expose the central tension: distortion metrics (cosine similarity, SSIM) favor the blurry mean, masking that the diffusion model is far closer to the real frame distribution. Inception-based FID and KID reveal a clean perception-distortion frontier: diffusion attains KID 0.078 versus 0.375 for regression (

4.8\times

better), and a deployable train-derived calibration makes this practical without test-time ground truth. The model is genuinely action-controllable (steering drives scene displacement, Spearman

ρ= 0.81

, vs

-0.18

for regression). We trace limited single-pass motion to a shared-present anchor and engineer a compact 1.7M-parameter "jump" model that recovers full ground-truth motion magnitude (

1.02\times

GT), where single-pass models capture less than half.

Ruslan Sharifullin, Benjamin Jiang, Kai Xi Chew

Stanford University

Action-Conditioned Video Generation Diffusion Transformer Latent Diffusion Models Action-Conditioned World Models Autonomous Driving World Models for Autonomous Driving+2

Ask PDF

Jun 15, 2026cs.CV

Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.

Jie Zhang, Xiaoyue Chen, Anzhe Chen +36

Qwen Team

Action-Conditioned Video Generation Video Diffusion Models World Models for Robotics Action-Conditioned World Models Video Generation Embodied AI+1

Ask PDF