cs.NEAug 20, 2026
SaveThe Forward-Backward Disconnect: State Dynamics, Credit Assignment, and Biological Grounding in Neural Computation
Organizations: Department of Electrical and Computer Engineering, Maroun Semaan Faculty of Engineering and Architecture, American University of Beirut, Lebanon
Abstract
A recurring pattern in neural computation is the reintroduction of dynamical and biological structure into models originally simplified for scalable optimization. Early feedforward networks reduced biological neurons to threshold or rate-like summation units, an abstraction compatible with global-gradient training at scale. Since then, forward computation has diversified: modern architectures carry recurrent state, retrieve from long contexts through attention and associative memory, compress histories through structured state-space dynamics, evolve in continuous time, settle to implicit equilibria, and communicate through sparse spikes. Training has diversified less. Scalable learning remains concentrated around backpropagation, backpropagation through time, adjoint methods, implicit differentiation, and surrogate-gradient variants. We call this asymmetry the forward-backward disconnect and develop a taxonomy spanning neural model families along three coupled axes: state-dynamics structure, credit-assignment mechanism, and biological grounding. Forward and learning grounding are treated separately, and the unit of analysis is the architecture-learning configuration rather than the architecture name alone. Across static, recurrent, attention-based, state-space, continuous-time, implicit, spiking, biologically plausible, and neuromorphic families, forward dynamics have diversified while the highest demonstrated scales remain concentrated in global or closely gradient-derived error-propagation mechanisms. Closing this disconnect requires better alignment among state dynamics, credit assignment, and computational substrate.
Figures & tables
Figure 1. Timeline of selected architectural, learning-rule, and hardware-substrate milestones in neural computation, 1943–2026. Marker colors encode biological grounding under the split-axis scheme of Section 3.3 ; hardware uses a distinct gold marker because substrates do not map onto grounding tiers. The post-2010 segment is horizontally zoomed to expose the greater density of recent milestones. The figure provides illustrative historical orientation rather than quantitative evidence; Figure 2 and Section 11.1 establish the forward–backward asymmetry categorically. A three-track timeline from 1943 to 2026. The upper track shows selected architectural milestones in neural computation; the middle track shows selected learning-rule and credit-assignment milestones; the lower track shows selected hardware-substrate milestones, including neuromorphic computing introduced by Mead in 1990, the digital neuromorphic chip TrueNorth and the parallel ARM-core machine SpiNNaker in 2014, the systolic-array Tensor Processing Unit in 2017, the Loihi chip in 2018 with on-chip plasticity, Loihi 2 in 2021, and the mixed-signal BrainScaleS-2 in 2022. A visual axis break at 2010 separates a pre-2010 segment from a post-2010 segment drawn at a horizontally zoomed scale. Architecture-track and learning-track marker color encodes biological grounding under a shared color scheme: white for none or historical inspiration; teal for functional analogy on the forward axis and weak on the learning axis; olive for architectural constraint; rose for local or algorithmic; and coral for mechanistic plausibility on the forward axis and mechanistic plasticity on the learning axis. Hardware-track markers are filled gold because compute substrates do not map onto biological-grounding tiers.
| Mechanism | Gradient relation | Update scope | Temporal requirement | Objective source | Update timing |
| Backpropagation ( Rumelhart et al., 1986 ) | Exact | Global / non-local | None (static graph) | Global supervised | After forward pass |
| BPTT ( Werbos, 1990 ) | Exact | Global / non-local | Unrolled sequence | Global supervised | After full sequence |
| FA / DFA / PFA ( Lillicrap et al., 2016 ; Nøkland, 2016 ; Li and Benna, 2024 ) | Approximate | Broadcast error via random or learned matrices | None in feedforward case | Global supervised | Feedback pass; DFA partly parallel |
| Synthetic gradients ( Jaderberg et al., 2017 ) | Approximate (predicted) | Module-local predictor; target derived from downstream gradient | Optional recurrent state | Global supervised | Asynchronous |
| Adjoint / implicit diff. ( Chen et al., 2018 ; Bai et al., 2019 ) | Exact under solver / equilibrium conditions | Solver / global linear solve at equilibrium | Trajectory or fixed point | Global objective | Backward solve after forward solve |
| Surrogate-gradient SNN ( Neftci et al., 2019 ) | Approximate derivative | Global error propagation through smoothed spikes | Spike-time trajectory | Global supervised | Backward through temporal graph |
Table 1. Secondary credit-assignment dimensions applied to representative learning mechanisms. These attributes are descriptive coordinates rather than additional mutually exclusive classes; individual variants may differ. The table instantiates the multidimensional structure of the backward axis introduced above, so a new architecture–learning configuration can be located along the same coordinates.
Figure 2. Joint taxonomy of neural computation as an alluvial flow. Each architecture–learning configuration is one unit of flow from a forward state-dynamics class (left column) to a credit-assignment mechanism (right column). Node heights and ribbon widths are proportional to configuration counts; ribbon colors identify the source class. Table 3 lists model families and grounding assignments; Supplementary Section S3 provides the complete configuration-level ledger. Global gradient is the largest single credit-assignment category (16/32), followed by Approximate or implicit gradient (7/32), and it receives configurations from four of the five forward classes: the categorical form of the forward–backward disconnect introduced in Section 1 . An alluvial flow diagram showing the joint taxonomy as a bipartite flow. Five colored vertical bars on the left represent forward state-dynamics classes, from top to bottom: Static (6 configurations, blue), Discrete-time or sequence (14, orange), Continuous-time (6, green), Implicit fixed-point (4, purple), Hybrid event-driven (2, red). Four gray vertical bars on the right represent credit-assignment mechanisms, from top to bottom: Global gradient (16 configurations), Approximate or implicit gradient (7), Local plasticity (6), Energy or unsupervised (3). Fourteen curved ribbons connect the left classes to the right mechanisms; each ribbon's width is proportional to how many configurations flow from that source to that destination, and each ribbon is colored by its source class. Global gradient is the largest single credit-assignment category, accounting for half of the audited configurations and receiving ribbons from four of the five forward classes, followed by Approximate or implicit gradient as the second-largest. The visual impression is a fan of colored streams on the left converging predominantly into the two upper destinations on the right, which is the visual signature of the credit-assignment concentration relative to forward-dynamics diversification.
| Forward class | Global gradient | Approximate or implicit gradient | Local plasticity | Energy / unsupervised | Total |
| Static | 4: MLP, CNN, ResNet, PINN (C01–C04) | 0 | 1: Neocognitron (C05) | 1: MLP + Forward-Forward (C06) | 6 |
| Discrete | 8: RNN, LSTM, GRU, Transformer, S4, Mamba, RWKV, xLSTM (C07–C14) | 2: RNN + synth. grad., RNN + RFLO (C15, C16) | 2: classical Hopfield, HTM (C17, C32) | 2: RBM, DBN (C18, C19) | 14 |
| Continuous | 3: CTRNN, Liquid TC, Liquid closed-form (C20–C22) | 1: Neural ODE (C23) | 2: PC recurrent, CT-RNN + reward-mod. Hebb. (C24, C25) | 0 | 6 |
| Implicit | 1: modern Hopfield (C26) | 3: DEQ, PC local-error, PC BP-equiv. limit (C27–C29) | 0 | 0 | 4 |
| Hybrid | 0 | 1: surrogate-gradient LIF SNN (C30) | 1: STDP-trained LIF SNN (C31) | 0 | 2 |
| Total | 16 | 7 | 6 | 3 | 32 |
Table 2. Configuration counts underlying Figure 2 , with the model families that populate each cell. Cell entries follow the pattern count: family list (C-IDs referencing the per-configuration ledger of Supplementary Section S3) . Row and column totals match the node heights of Figure 2 exactly.
| Survey | Venue, Year | Static | Discrete / seq. | Attn / SSM | Cont. / Implicit | Spiking / hybrid | Bio learn. | Neuro- morph. |
| Hassabis et al. ( Hassabis et al., 2017 ) | Neuron , 2017 | — | — | — | ||||
| Tavanaei et al. ( Tavanaei et al., 2019 ) | Neural Netw. , 2019 | — | — | — | — | |||
| Roy et al. ( Roy et al., 2019 ) | Nature , 2019 | — | — | — | — | |||
| Lillicrap et al. ( Lillicrap et al., 2020 ) | Nat. Rev. Neurosci. , 2020 | — | — | — | — | — | — | |
| Karniadakis et al. ( Karniadakis et al., 2021 ) | Nat. Rev. Phys. , 2021 | — | — | — | — | — | ||
| Schuman et al. ( Schuman et al., 2022 ) | Nat. Comput. Sci. , 2022 | — | — | — | — |
Table 4. Coverage of recent surveys against this paper’s model-family categories. : substantive; : partial; —: out of scope. The distinguishing contribution of the current survey is not broader coverage alone but the specific joint decomposition it applies: state dynamics credit assignment biological grounding substrate. This joint decomposition is what makes the forward–backward disconnect visible as a categorical pattern rather than as a scattered observation across independently organized reviews.
Figure 3. Three canonical dynamical objects recovered in trained RNN state spaces by the fixed-point reverse-engineering technique of ( Sussillo and Barak, 2013 ) . Solid black dots mark fixed points; arrows indicate the local flow of the dynamics in a 2D projection of the hidden state. A point attractor (a) implements memory storage. A line attractor (b) implements graded integration along a one-parameter family of stable states and is the canonical mechanism for context-dependent decision making in trained RNNs and in prefrontal cortex ( Mante et al., 2013 ) . A saddle point (c) separates basins of attraction (dashed separatrices) and implements a decision boundary between two stored states. Across populations of trained networks the precise weights differ but the topology of these objects is conserved ( Maheswaranathan et al., 2019 ) . Three side-by-side phase-portrait panels in a 2D projection of recurrent-network hidden state. Panel (a) shows a point attractor: a single filled dot in the center with arrows pointing inward from all directions. Panel (b) shows a line attractor: a horizontal line of stable states with arrows pointing toward the line from above and below. Panel (c) shows a saddle point: a central dot with arrows along one axis flowing inward and along the perpendicular axis flowing outward, with dashed separatrices dividing the plane into basins of attraction.
Figure 4. Substrate reconnection map. The dense-gradient path on the left ( GPU / TPU ; backpropagation training scales) and the event-driven path on the right (neuromorphic; event-driven inference scales) currently bridge only empirically (dashed gray): training runs off-chip with surrogate gradients on GPUs and deploys on neuromorphic hardware for inference. The missing bridge (dashed red) is a substrate-aligned learning rule competitive with backpropagation at deep-network scale, which Section 9 found absent. Neuromorphic hardware removes the substrate penalty for event-driven forward computation but does not supply the missing rule. A conceptual schematic in three vertical layers. Top layer: two rounded rectangles side by side. The left box, labeled GPU/TPU substrate, lists dense tensors, stored activations, batch synchronization, and global gradient transport. The right box, labeled Neuromorphic substrate, lists sparse events, local state, online updates, and memory near compute. Middle layer: two outcome boxes shaded light blue. The left says Backpropagation training scales; the right says Event-driven inference scales. Solid black arrows connect each substrate down to its corresponding outcome, labeled global gradient on the left and sparse, local on the right. A dashed gray arrow runs horizontally between the two outcome boxes labeled off-chip surrogate-gradient training, on-chip event-driven deployment, indicating the current empirical workaround. Bottom layer: a dashed red rectangle labeled Missing bridge, with subtext scalable local credit assignment and a note that no substrate-aligned learning rule surveyed in Section 9 closes this gap. Dashed red arrows from each outcome box curve down toward this missing-bridge box, visually marking the unresolved connection.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | State-dynamics class | Learning row | Forward grounding | Learning grounding |
|---|---|---|---|---|
| Modern Hopfield (end-to-end) | Implicit / fixed-point | Global gradient | Functional analogy | None |
| Classical Hopfield (Hebbian) | Discrete (fixed-point fn.) | Local plasticity | Arch. constraint | Local or algorithmic |
| PC: recurrent inference | Continuous-time / fixed-pt. | Local plasticity | Arch. constraint | Local/algorithmic |
| PC: energy-based | Continuous-time / fixed-pt. | Energy/unsupervised | Arch. constraint | Local/algorithmic |
| PC: local-error | Implicit / fixed-point | Approx./implicit | Funct. analogy | Local/algorithmic |
| PC: BP-equivalent limit | Implicit / fixed-point | Approx./implicit | Funct. analogy | Local/algorithmic |
| ID | Architecture | Learning rule | Forward class | Credit class | Fwd. gnd. | Learn. gnd. | Evidence |
| C01 | MLP | Backpropagation | Static | Global grad. | None/Hist. | None | [ 115 ] |
| C02 | CNN (LeNet, AlexNet) | Backpropagation | Static | Global grad. | None/Hist. | None | [ 75 , 66 ] |
| C03 | ResNet | Backpropagation | Static | Global grad. | None/Hist. | None | [ 48 ] |
| C04 | PINN | Backpropagation (with PDE-residual objective) | Static | Global grad. | None/Hist. | None | [ 109 ] |
| C05 | Neocognitron | Local competitive Hebbian | Static | Local plast. | Arch. constr. | Local/alg. | [ 31 ] |
| C06 | MLP + Forward-Forward | Layerwise goodness | Static | Energy/unsup. | None/Hist. | Local/alg. | [ 53 ] |
| Backward category | Mechanism | Count | Ledger IDs |
|---|---|---|---|
| Global gradient (16) | Ordinary reverse-mode BP | 10 | C01–C04, C10–C14, C26 |
| BPTT / reverse-mode through recurrence | 4 | C07–C09, C22 | |
| Solver BP / solver BPTT | 2 | C20–C21 | |
| Approximate/implicit (7) | Synthetic gradients (DNI) | 1 | C15 |
| Random-feedback local online (RFLO) | 1 | C16 | |
| Adjoint sensitivity | 1 | C23 |
Table 1: Mechanism-level decomposition of the two largest backward categories. Each mechanism is identified by the exact procedure it uses, and the count reports how many of the 32 ledger configurations it accounts for. Global gradient (16 configurations total) is dominated by ordinary reverse-mode backpropagation (10); the Approximate/implicit category (7 configurations total) fragments across seven distinct mechanisms, each contributing exactly one configuration.
| Family | Largest reported result | Comparison | Reference |
|---|---|---|---|
| Feedback alignment (plain) | Performs well on MNIST/CIFAR; significantly below BP on ImageNet | Gap to BP large at ImageNet | [ 7 ] |
| Sign-symmetry (FA variant) | Approaches BP on ImageNet with ResNet-18/AlexNet | Within a few percent of BP | [ 132 ] |
| Target propagation | MNIST/CIFAR competitive; ImageNet substantially below BP | No parity at ImageNet scale | [ 7 ] |
| Equilibrium propagation | 11.7% test error on CIFAR-10 with deep ConvNets; approaches BPTT to within roughly 0.6 percentage points; downsampled ImageNet 32 32 in subsequent work | No full ImageNet-1k or LM result | [ 69 , 70 ] |
| Weight Mirror (weight-symmetry relaxation) | Learns dynamic feedback weights to align with forward weights; on ImageNet, ResNet-18 top-1 error 30.2% for Weight Mirror vs 30.1% for BP (within 0.1 pp), ResNet-50 top-1 error 23.4% for Weight Mirror vs 22.9% for BP (within 0.5 pp); the authors describe the result as “nearly matched backprop” and “learning ImageNet about as well as backprop does” | Approximately matches BP on ResNet-18 and ResNet-50 ImageNet under the paper’s training pipeline; substantially outperforms plain feedback alignment and sign-symmetry; still relies on error-broadcast machinery | [ 2 ] |
| Product Feedback Alignment (PFA) | On ResNet-18 ImageNet: BP 69.69%, PFA 68.46%, PFA-o 69.30% top-1 accuracy; approaches BP without explicit weight symmetry | PFA-o within roughly 0.4 percentage points of BP at ResNet-18 scale, but no ResNet-50 or GPT-class demonstration reported | [ 77 ] |
Table 2: Empirical scaling evidence anchoring the main paper’s scaling claim: strongest reported per-family results at the search cutoff, and their comparison against mainstream backpropagation.