Minimizing the Effect of Sleep Deprivation in the Forward-Forward Algorithm

Joy Datta; Puja Saha; Rawhatur Rabbi; Nafiz Imtiaz Rafin; Swakkhar Shatabda; Md. Golam Rabiul Alam; Chad Mourning

cs.LGSep 5, 2026

Minimizing the Effect of Sleep Deprivation in the Forward-Forward Algorithm

Authors: Joy Datta, Puja Saha, Rawhatur Rabbi, Nafiz Imtiaz Rafin, Swakkhar Shatabda, Md. Golam Rabiul Alam, Chad Mourning

Organizations: Brac University, Dhaka 1212, Bangladesh · Rajshahi University of Engineering and Technology, Rajshahi 6204, Bangladesh · Ohio University, Athens, OH 45701, United States

Abstract

This paper addresses the challenge posed by sleep deprivation in the Forward-Forward algorithm, where separating the two passes in this algorithm and imbalancing the data processing in the passes is considered an imitation of the cognitive processes observed in humans suffering from sleep deprivation. Previous research has demonstrated that sleep deprivation in the Forward-Forward algorithm has a catastrophic effect on learning efficacy. To mitigate this issue, we explore several approaches; these include alternative activation, optimized loss function, and threshold tuning. To simulate periodic rest, we reduce the number of positive passes in alternating epochs, creating short break phases. We additionally investigate the potential of caffeine-induced stimulation to enhance performance during sleep-deprived conditions. Experimental evaluations conducted on the MNIST and Fashion-MNIST datasets demonstrate that these modifications improve accuracy under the context of sleep deprivation. For example, a 2%-62% accuracy gain is observed in a severe sleep deprivation setting (16 positive or awake periods and 1 negative or sleep period). The approaches also enhance the resilience of the algorithm and its alignment with the adaptive mechanisms of human cognition.

Figures & tables

Figure 1: Positive (upper) and negative (lower) samples from Fashion MNIST.

Layer 1	Layer 2	Layer 3	Accuracy (mean $\pm$ std)
1.0	1.0	1.0	$36.88\pm 1.23$
1.5	1.5	1.5	$33.17\pm 1.10$
2.5	2.5	2.5	$26.88\pm 4.44$
1.0	1.25	1.5	$37.47\pm 1.05$
1.0	1.5	2.0	$37.10\pm 0.39$
1.0	2.0	3.0	$36.11\pm 1.06$

Table 1: Layer-wise threshold combinations and their test accuracies (%) after training the model three times for 100 epochs.

Figure 2: Performance of ReLU, Tanh, and Tanhshrink in the case of severe sleep deprivation (positive passes or awake period = 32, negative passes or sleep period = 1).

Activation	Awake = 8	Awake = 16
ReLU	$33.19\pm 1.45$	$25.23\pm 3.06$
ELU	$41.81\pm 1.28$	$21.38\pm 1.76$
SiLU	$38.00\pm 0.58$	$27.22\pm 2.07$
Tanh	$49.15\pm 1.09$	$48.06\pm 1.26$
Tanhshrink	$49.83\pm 0.86$	$48.63\pm 1.48$
Softplus	$35.76\pm 1.49$	$27.81\pm 2.90$

Table 2: Activation functions and their test accuracies (%) after 100 epochs (consistent sleep deprivation setting).

Activation	Awake = 8	Awake = 16
ReLU	$25.54\pm 0.26$	$22.62\pm 0.46$
ELU	$34.66\pm 1.38$	$23.26\pm 0.51$
SiLU	$33.55\pm 1.91$	$34.93\pm 1.96$
Tanh	$25.13\pm 1.26$	$16.36\pm 0.48$
Tanhshrink	$53.60\pm 1.00$	$51.45\pm 1.60$
Softplus	$24.60\pm 1.34$	$21.63\pm 2.46$

Table 3: Test accuracy (%) on Fashion-MNIST under alternating awake–sleep pattern, mimicking short breaks after each epoch.

Dataset	Method	Awake=2	Awake=4	Awake=8	Awake=16
MNIST	Baseline	78	75	73	10
	Optimal Configuration	75	77	75	56
	CIL	74	77	73	72
Fashion-MNIST	Baseline	55	54	52	49
	Optimal Configuration	52	54	75	68
	CIL	52	51	49	51

Table 4: Comparison of baseline and proposed methods on MNIST and Fashion-MNIST for different positive batch (awake) sizes.

Figure 3: Layer activity when correct or positive data is fed. Random neural activity before training (left) and enhanced neural activity after training (right).

Figure 4: Layer activity when incorrect (e.g., image after applying the masking technique described) or negative data is fed. Random neural activity before training (left) and lowered neural activity. Since it has seen negative data, it must be below a certain threshold (right).

Dataset	Method	Awake=32	Awake=64	Awake=128
MNIST	Baseline	29	19	21
	SW	30	20	17
	CIL	53	55	41
Fashion MNIST	Baseline	22	21	22
	SW	49	48	46
	CIL	50	49	49

Table 5: Accuracy comparison across datasets, methods, and the number of awake or positive batches.

Figure 5: Epochs vs. accuracy curves of CIL and the baseline.

Figure 6: Performance of ReLU, Tanh, and Tanhshrink in the case of severe sleep deprivation (awake period = 128, sleep period = 1).

		Consistent SD		SD with Short Breaks
$\alpha$	$\beta$	Awake = 8	Awake = 16	Awake = 8	Awake = 16
1	1	$49.83\pm 0.86$	$48.63\pm 1.48$	$53.06\pm 1.00$	$51.45\pm 1.60$
1	2	$50.42\pm 0.56$	$50.68\pm 0.63$	$52.18\pm 0.34$	$50.17\pm 0.44$
1/awake period	1	$52.73\pm 0.94$	$49.21\pm 0.61$	$53.61\pm 0.51$	$52.48\pm 1.63$

Table 6: Comparison of test accuracy (%) under consistent sleep deprivation (SD) vs. SD with short breaks using different values of

\alpha

(awake period scale) and

\beta

(sleep period scale).

No. of Pos. Batches	Consistent (%)	Short Breaks (%)
2	54.17	61.86
4	45.18	49.52
8	44.18	45.72
16	38.63	43.31

Table 7: Comparison of consistent awake and sleep periods vs. with short breaks.

Forward-Forward Learning

Forward-Forward learning trains neural networks with layer-local goodness objectives using forward passes instead of backpropagated gradients.

See all 17 papers

Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Hardware-Validated Benchmarking Study of Forward-Only Algorithms
Nov 2, 2025Przemysław Spyra, Witold DzwinelEnergy-Efficient ML Forward-Forward Learning
AGH University of Krakow, Faculty of Computer Science
The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Sep 27, 2026Zhaoxian Wu, Haichuan Liu, Tianyi ChenForward-Forward Learning Neural Network Optimization
Cornell Tech and Cornell University
Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550

Nov 2, 2025J/K move · Enter open · S save

Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Hardware-Validated Benchmarking Study of Forward-Only Algorithms

Przemysław Spyra, Witold Dzwinel

AGH University of Krakow, Faculty of Computer Science

Energy-Efficient ML Forward-Forward Learning Backpropagation

Open paper AskPDF arXiv

Backpropagation (BP) powers modern deep learning, but it must hold every activation in memory until a global backward sweep consumes it, and that sweep locks each layer until the one above has finished. Forward-only algorithms, from Hinton's Forward-Forward (FF) through Cascaded Forward (CaFo) to Mono-Forward (MF), replace the global gradient with local learning signals and are promoted as more efficient on the basis of FLOP counts rather than measurements. We test that claim on hardware. Across almost 1000 runs on four vision benchmarks (MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100), we read energy, power and memory from an NVIDIA A100 GPU and compare each algorithm against a tuned BP baseline. Removing the backward pass does not by itself save energy: FF costs 4.1x to 6.3x BP's energy and CaFo 3.0x to 7.1x. Mono-Forward is the exception. Given the same number of data passes as BP, it uses 12.5% to 19.9% less energy and 24.6% to 27.5% less memory on the 3x2000 multilayer perceptrons, the largest configurations tested, at lower power and near-equal wall-clock time; on CIFAR-100 its test accuracy is statistically equivalent to BP's. A pre-registered ablation identifies gradient detachment, not MF's auxiliary losses, as the dominant factor tested. Trained to convergence, MF exceeds BP's test accuracy on CIFAR-10 and CIFAR-100 by 3.90 and 5.04 percentage points, but needs 3.5x to 4.5x as many passes and 3.2x to 4.0x the energy. Faster convergence is therefore the remaining target for an end-to-end gain.

Abstract

Forward-Forward Learning

Explore similar work

Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Hardware-Validated Benchmarking Study of Forward-Only Algorithms

The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses