Neural Computers

Mingchen Zhuge; Changsheng Zhao; Haozhe Liu; Zijian Zhou; Shuming Liu; Wenyi Wang; Ernie Chang; Gael Le Lan

cs.LGApr 7, 2026

Neural Computers

Authors: Mingchen Zhuge, Changsheng Zhao, Haozhe Liu, Zijian Zhou, Shuming Liu, Wenyi Wang, Ernie Chang, Gael Le Lan,

Organizations: Meta AI · KAUST (Core Contributors)

Abstract

We propose a new frontier: Neural Computers (NCs) that unify computation, memory, and I/O of traditional computers in a learned runtime state. Our long-term goal is the Completely Neural Computer (CNC): the mature, general-purpose realization of this emerging machine form, with stable execution, explicit reprogramming, and durable capability reuse. As an initial step, we study whether elementary NC primitives can be learned solely from collected I/O traces, without instrumented program state. Concretely, we instantiate NCs as video models that roll out screen frames from instructions, pixels, and user actions (when available) in CLI and GUI settings. We show that NCs can acquire elementary interface primitives, especially I/O alignment and short-horizon control, while routine reuse, controlled updates, and symbolic stability remain challenging. We outline a roadmap toward CNCs, to establish a new computing paradigm beyond today's agents and conventional computers.

Figures & tables

Figure 1: Neural computers across interfaces. Given a prompt or action stream, an NC rolls out future interface frames for / NC ${}_{\text{CLIGen}}$ (top) and NC ${}_{\text{GUIWorld}}$ (bottom). Logos denote datasets; NC ${}_{\text{CLIGen}}$ and NC ${}_{\text{GUIWorld}}$ are the corresponding models trained on those datasets. It models terminal or desktop dynamics.

Figure 2: Data types used to learn NC behaviors. Logos denote datasets: CLIGen (General) replays public asciinema traces spanning diverse real-world terminal workflows. CLIGen (Clean) uses scripted vhs runs to capture deterministic terminal traces for controlled experiments. GUIWorld captures desktop RGB with synchronized mouse/keyboard traces to validate action-conditioned rendering and control on GUIs.

Frames from Sample — CLIGen (General)

1 2 3 4 5 6

Semantic A root terminal session kicks off an AI command to make three 1024x1024 cat shots, shows quick parsing for each one, then presents pixelated cat previews with numbered links and asks whether to stash them in /root/2023_04_01-02_27_11_imgs . Regular In a root shell at ~ , the user runs ai -i 3 a cute cat , watches a green progress line announcing three 1024x1024 images, sees sequential parsing messages for images 1 through 3, and ends on a preview pane with three pixelated cat thumbnails, numbered download links, and a save prompt targeting /root/2023_04_01-02_27_11_imgs . Detailed In a dark-background terminal at the root in ~ prompt, the user types ai -i 3 a cute cat . The screen prints Generating 3 1024x1024 images (press CTRL-C to cancel)... , shows parsing messages for images 1–3, and ends on a preview pane with three numbered thumbnails and a save prompt targeting /root/2023_04_01-02_27_11_imgs .

Frames from Sample — CLIGen (Clean)

1 2 3 4 5 6

Scripted caption Type python ; Enter; Type values = [n*n for n in range(1, 10)] ; Enter; Type print(values) ; Enter; Type exit() ; Enter.

Table 1: Data samples for CLIGen (General) and CLIGen (Clean).

Figure 3: / NC ${}_{\text{CLIGen}}$ architecture. Terminal frames are observations $x_{t}$ . A prompt and the first frame seed the conditioning stream. The Wan2.1-based latent state $z_{t}$ rolls forward under the standard I2V sampling scheme.

Metric	Value
Average PSNR	40.77
Average SSIM	0.989

Table 2: Reconstruction quality.

Figure 4: Wan2.1 VAE reconstructions on CLIGen (General) terminal frames at different font sizes.

Figure 5: (a–b) Qualitative generations before and after CLIGen training; (c) CLIGen (Clean) PSNR / SSIM plateau around 25k training steps.

Prompt style	PSNR	SSIM	Avg. words
Semantic	21.90	0.813	55
Regular	23.63	0.843	52
Detailed	26.89	0.867	76

Table 3: Caption styles versus TI2V fidelity.

Steps (k)	Char. acc.	Exact line
0	0.03	0.01
10	0.18 $\color[rgb]{0,0.6,0}{\uparrow}_{0.15}$	0.05 $\color[rgb]{0,0.6,0}{\uparrow}_{0.04}$
20	0.33 $\color[rgb]{0,0.6,0}{\uparrow}_{0.30}$	0.12 $\color[rgb]{0,0.6,0}{\uparrow}_{0.11}$
30	0.41 $\color[rgb]{0,0.6,0}{\uparrow}_{0.38}$	0.18 $\color[rgb]{0,0.6,0}{\uparrow}_{0.17}$
40	0.52 $\color[rgb]{0,0.6,0}{\uparrow}_{0.49}$	0.26 $\color[rgb]{0,0.6,0}{\uparrow}_{0.25}$
50	0.52 $\color[rgb]{0,0.6,0}{\uparrow}_{0.49}$	0.27 $\color[rgb]{0,0.6,0}{\uparrow}_{0.26}$

Table 4: OCR accuracy versus training.

Model	Accuracy
Wan2.1	0%
NC ${}_{\text{CLIGen}}$	4%
Veo3.1	2%
Sora2	71 %

Table 5: Arithmetic probe accuracy (100 problems sampled from a 1,000-problem held-out pool).

Figure 6: Reprompting boosts performance to 83%.

Panel of Table 6 — Table 6: Hypotheses for Sora2’s advantage.

Split	Avg. cursor speed (px/frame)	Actions / sec
Random Slow	1.51	1.58
Random Fast	195.15	4.18
CUA (supervised)	3.79	0.10

Table 7: Cursor/action statistics.

Panel of Figure 7 — Figure 7: Four modes for injecting GUI actions into the diffusion transformer. 1 modulates VAE latents before the transformer; 2 adds action tokens alongside visual tokens; 3 applies block-wise residual updates; and 4 inserts an action cross-attention module inside transformer blocks.

Split	FVD ${}_{\text{all}}$	SSIM ${}_{\text{all}}$	LPIPS ${}_{\text{all}}$
Untrained baseline	149.61	0.496	0.605
Random Fast (train)	48.17	0.695	0.483
Random Slow (train)	20.37	0.830	0.237
Claude CUA (train)	14.72	0.885	0.144

Table 8: Overall performance across data sources.

Figure 8: Cursor references in GUIWorld. Left: original desktop frames. Middle: binary cursor masks. Right: cursor-only reference frames rendered over a neutral background.

Loss variant	Cursor accuracy
Position $(x,y)$ only	8.7%
Position $(x,y)$ + Fourier	13.5%
Position $(x,y)$ + SVG mask/ref	98.7%

Table 9: Cursor conditioning losses versus accuracy.

Mode	SSIM +15 $\uparrow$	LPIPS +15 $\downarrow$	FVD +15 $\downarrow$
baseline 1 (untrained)	0.326	0.649	184.3
baseline 2 $\dagger$ ( external )	0.746	0.251	33.4
contextual	0.813	0.190	24.8
residual	0.857	0.138	18.8
internal	0.863	0.141	14.5

Table 10: Action-driven metrics across injection schemes (15 frames after action).

Mode	Encoding	SSIM +15 $\uparrow$	LPIPS +15 $\downarrow$	FVD +15 $\downarrow$
internal	raw-action (event-stream)	0.847	0.144	16.6
internal	meta-action (API-like)	0.863	0.141	14.5

Table 11: Raw-action vs. API-like action encoding under the same injection mode (15 frames after action).

User intent	Raw-action encoder (event stream)	Meta-action encoder (API-like slot)
ls -l	$\bullet$ Key events per character (e.g., l , s , <space> , - , l ) $\bullet$ Activates entries in a 169-d keyboard multi-hot $\bullet$ No explicit command semantics; inferred from sequence	$\bullet$ type: KeyboardType $\bullet$ text: "ls -l" $\bullet$ Encoded by shared text encoder
ctrl+v	$\bullet$ Separate keydown/keyup events for ctrl and v $\bullet$ Activated in multi-hot (or shortcut entry)	$\bullet$ type: Shortcut $\bullet$ id: ctrl+v $\bullet$ Embedded via shortcut table

Table 12: Encoding examples for raw-action and meta-action encoders.

System object	Organized around	Source of truth	Primary role
Conventional computer	Explicit programs	Explicit programs and explicit machine state	Reliably execute explicit programs
AI agent	Tasks	External environments, tools, and workflow state	Accomplish tasks through an existing software stack
World model	Environment dynamics	A learned model of state evolution	Predict and roll out how an environment may evolve
Neural computer	Runtime	Installed capabilities and runtime state inside the learned system	Sustain execution, accumulate capability, and govern updates within one learned machine

Table 13: Four system objects compared at a common systems level.

CNC requirement	Plain reading	What engineering evidence should look like
Turing complete	The system is not restricted to a narrow family of fixed tasks, but can in principle express general computation.	As effective memory and context grow, the same NC should remain able to carry longer and more structured procedures rather than failing by a different shortcut each time.
Universally programmable	Inputs should not only trigger one-off behavior, but install routines or internal executors that remain callable later.	Capabilities can be installed, invoked, composed, and retained across tasks rather than being relearned or outsourced each time.
Behavior-consistent	Ordinary use should not silently change the machine; behavioral change should come from explicit updates.	Same-version behavior is reproducible; execution and update traces can be inspected, replayed, and rolled back; long-horizon drift is measurable and governable.
Machine-native semantics	The system should not merely imitate conventional computers with neural components, but develop its own machine semantics and programming interfaces.	Composition, routing, continuous state, and internal executors yield usable system-level advantages; prompts, demonstrations, traces, and constraints begin to function as programming interfaces rather than mere logs.

Table 14: Operational reading of the four CNC requirements.

Figure 9: Changing human-machine relations across conventional computers, the agent era, and neural computers. Conventional computers are used directly; in today’s agent stack, agents mediate existing computers while world models serve as a parallel predictive layer; NCs aim to unify these split functions within one learned runtime. In this sense, NCs are motivated not by replacing the current stack from the outside, but by internalizing its split functions within one learning machine. A Completely Neural Computer (CNC) is the mature, general-purpose realization of this machine form.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Common issues in web-scale screen video crawling. Web videos mix uncontrolled content, heterogeneous OS/UI configurations, and inconsistent capture quality. This makes terminal localization, OCR, and time alignment unreliable without heavy filtering and sanitization.

Figure 11: Agentic online interaction pipeline (early exploration). A control plane ingests trajectories for storage and video-model updates (online where feasible, with offline replay support). A sandboxed environment plane executes an agent in an isolated container and records synchronized state/action traces.

Field	Content (abridged)
caption	The user (username field shown verbatim as annonomous [sic]) logged into their account on the host NeuralComputer , and after some initial terminal setup and cursor movements, they started typing the command nvim to likely open the Neovim text editor, but the session recording ends abruptly without showing the actual command execution or any further interaction.
caption_detailed	At the annonomous@NeuralComputer: $ prompt, the user types nvim , but before the command executes, the screen rapidly displays a colorful, pixelated animation with various RGB colors, including shades of blue, green, and purple, filling the 80x24 terminal window. The animation is briefly interrupted by a cursor blink, then the prompt returns, awaiting further input.
caption_semantic	The terminal displays a rapid sequence of colored text and pixel art animations, with syntax-highlighted code and prompts appearing in a mix of blue, green, and white hues, as the user types commands and the system responds with scrolling output, error messages, and success indicators.
data_info.version	2 ( asciinema v2 header)
data_info.size	width=80 , height=24 ; original/target/scaled sizes all match, padding [0,0] .
data_info.env	SHELL=/bin/bash , TERM=xterm-256color .

Appendix

Table 15: Caption tiers and metadata for CLIGen (General) clip 7****_0001 .

Type id	Action	Parameter fields
0	None	–
1	Mouse Click/Drag	button , click_count , drag_flag
2	Mouse Scroll	direction , amount
3	Keyboard Type	text (e.g., ls -l ) $\rightarrow$ shared text encoder
4	Shortcut	shortcut_id (e.g., ctrl+v )

Appendix

Table 16: Meta-action schema for GUIWorld (per action slot).

Aspect	Raw-action encoder (v1)	Meta-action encoder (v2)
Action schema	Per-frame multi-hot vector composed of 13 mouse actions and 169 keyboard actions; no explicit type hierarchy.	Hierarchical schema with explicit action types and parameters: action_types $\in\{0,1,2,3,4\}$ for {None, Mouse Click/Drag, Mouse Scroll, Keyboard Type, Keyboard Shortcut}, plus typed parameter fields (e.g., button, scroll amount, shortcut ID, text).
Mouse representation	Two signals per frame: (1) continuous cursor trajectory $\texttt{mouse\_trajectories}_{t}\in\mathbb{R}^{2}$ ; (2) discrete mouse events $\texttt{mouse\_action\_events}_{t}\in\{0,1\}^{13}$ (multi-hot).	Per-frame mouse trajectory $\texttt{mouse\_trajectories}_{t}$ is encoded by a shared mouse-trajectory encoder and the temporal windowing module into dense mouse embeddings $\texttt{mouse\_latent}_{t}\in\mathbb{R}^{d_{\text{mouse}}}$ , optionally fused with action embeddings.
Keyboard representation	Per-frame keyboard state $\texttt{keyboard\_action\_events}_{t}\in\{0,1\}^{169}$ , where each dimension corresponds to a key or shortcut; only multi-hot activation is available, no text semantics.	Two complementary forms: keyboard shortcuts as keyboard_shortcut IDs embedded via an embedding table, and free-form text keyboard_text (e.g. “ls -l”) encoded by a shared text encoder (e.g., T5) and projected to the action embedding dimension.
Per-frame representation	Mouse and keyboard events are concatenated: $\texttt{raw\_actions}_{t}=[\,\texttt{mouse\_events}_{t},\,\texttt{keyboard\_events}_{t}\,]\in\mathbb{R}^{182}$ .	For each frame $t$ and slot $s$ , type plus parameters are embedded into slot_embeds ${}_{t,s}\in\mathbb{R}^{D}$ . Valid slots are aggregated (mean or attention) into a single per-frame action embedding $\texttt{action\_embeds}_{t}\in\mathbb{R}^{D}$ .
Multi-slot support	No explicit notion of multiple action slots per frame; all mouse and keyboard signals are collapsed into one 182-d multi-hot vector.	Explicit multi-slot design: action_types and all parameter tensors have shape $(B,T,S)$ with $S=2$ slots per frame, enabling, e.g., concurrent “mouse click + keyboard shortcut” in the same frame.
Latent-aligned representation	Temporal alignment is implemented ad-hoc inside each v1 action module (e.g., sliding windows over the 182-d per-frame vector), with custom logic per mode (external, internal, residual).	A unified temporal alignment module takes per-frame action embeddings $(B,F,D)$ and VAE compression ratio $c$ , constructs lag-aware temporal windows of size $c\cdot w$ , and outputs latent-aligned action features $\texttt{action\_latent}\in\mathbb{R}^{B\times T\times D}$ , with analogous mouse_latent when mouse features are enabled.

Appendix

Table 17: Raw-action versus meta-action encoders in GUIWorld.

Figure 12: Contextual attention mask for GUIWorld. Video tokens (pink) and action tokens (orange) share one sequence. The mask restricts attention to a short temporal window so each frame attends only to nearby frames and temporally aligned actions.

Modular Neural Computer
Mar 4, 2026Florin LeonNeural Algorithmic Reasoning Memory-Augmented Neural Networks
Department of Computers Faculty of Automatic Control and Computer Engineering “Gheorghe Asachi” Technical University of Iași, Romania
Von Neumann Networks
May 7, 2026Shekhar S. ChandraNeural Cellular Automata Cellular Automata
School of Electrical Engineering and Computer Science, The University of Queensland, Australia.
From communication to computation in neurons-on-a-chip: an in silico study of neurotopomorphic computing
Oct 5, 2026Michael Taynnan BarrosSpiking Neural Networks Neuromorphic Computing
School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK

Mar 4, 2026J/K move · Enter open · S save

Modular Neural Computer

Florin Leon

Department of Computers Faculty of Automatic Control and Computer Engineering “Gheorghe Asachi” Technical University of Iași, Romania

Neural Algorithmic Reasoning Memory-Augmented Neural Networks Modular Neural Networks

Open paper AskPDF arXiv

This paper introduces the Modular Neural Computer (MNC), a memory-augmented neural architecture for exact algorithmic computation on variable-length inputs. The model combines an external associative memory of scalar cells, explicit read and write heads, a controller multi-layer perceptron (MLP), and a homogeneous set of functional MLP modules. Rather than learning an algorithm end to end from data, it realizes a given algorithm through analytically specified neural components with fixed interfaces and exact behavior. The control flow is represented inside the neural computation through one-hot module gates, where inactive modules are inhibited. Computation unfolds as a sequence of memory transformations generated by a fixed graph. The architecture is illustrated through three case studies: computing the minimum of an array, sorting an array in place, and executing A* search on a fixed problem instance. These examples show that algorithmic procedures can be compiled into modular neural components with external memory while preserving deterministic behavior and explicit intermediate state.

Abstract

Explore similar work

Modular Neural Computer

Von Neumann Networks

From communication to computation in neurons-on-a-chip: an in silico study of neurotopomorphic computing