Node-wise Feature Encoding for Neural Performance Prediction

Matthew Grenier; William Hammer; Andrew Heuer; Nikhil Krishna; Yi Wang; Ramtin Zand

cs.LGAug 28, 2026

Node-wise Feature Encoding for Neural Performance Prediction

Authors: Matthew Grenier, William Hammer, Andrew Heuer, Nikhil Krishna, Yi Wang, Ramtin Zand

Organizations: University of South Carolina Department of Computer Science and Engineering · University of South Carolina Department of Mechanical Engineering

Abstract

As neural networks are increasingly deployed on resource constrained edge devices, accurate prediction of latency and energy is critical for efficient neural architecture search. Existing GNN and transformer based predictors achieve strong results but largely ignore node-level computational cost, limiting their ability to model performance critical operations. To address this, we introduce FeatureFormer, a neural performance predictor that incorporates explicit node-wise encodings of FLOPs, parameter counts, and memory proxies within a gated graph attention architecture. We also present NNEQ, a new large-scale energy consumption dataset that enables unified evaluation of latency and energy prediction. Extensive experiments demonstrate that FeatureFormer achieves state-of-the-art performance across both metrics, including challenging out-of-domain settings. Finally, we show that the proposed encoding is broadly applicable and consistently improves existing predictors with negligible overhead.

Figures & tables

Figure 1: Overview of the proposed neural performance predictor integrated into a neural architecture search (NAS) procedure. By replacing direct measurements of latency and energy with a predictive model, the hardware-aware NAS can be greatly accelerated.

Panel of Figure 2 — Figure 2: Last-layer attention scores of NN-Former ( Xu et al., 2025 ) , averaged over 200 AlexNet models from the NNLQ dataset ( Liu et al., 2022 ) that share the same architecture. (a) Without the proposed encoding, attention is evenly distributed, often favoring computationally inexpensive operations ( e.g., ReLU). (b) With the proposed encoding, attention concentrates on operations that are more relevant for latency prediction, such as convolutions.

Figure 3: Overview of the proposed FeatureFormer architecture, utilizing global and node-level computational features at different stages of the model.

Figure 4: Detailed view of an encoder block within our proposed model. In the figure, $A_{i}$ corresponds to the adjacency based mask used for a given attention head, selected from $A$ , $A^{T}$ , $AA^{T}$ , or $A^{T}A$ . For FeatureFormer, we utilize $d=512$ , $d_{FF}=2048$ , and $h=4$ while $l$ refers to the sequence length.

Figure 5: NNLQ ( Liu et al., 2022 ) (left) and NNEQ (right) ground truth latency and energy measurements across model families.

Metric	Test Domain	nn-Meter Zhang et al. (2021) MobiSys 2021 (avg)	NNLP Liu et al. (2022) ICPP 2022 (avg / best)	NAR-Former V2 Yi et al. (2023b) NeurIPS 2023 (avg / best)	NN-Former Xu et al. (2025) CVPR 2025 (avg / best)	FeatureFormer Ours (avg / best)
MAPE $\downarrow$	AlexNet	7.20	10.64 / 9.71	24.28 / 18.29	11.47 / 11.17	10.15 / 9.51
	EfficientNet	18.93	21.46 / 18.72	13.20 / 11.37	5.13 / 4.81	4.61 / 3.53
	GoogLeNet	11.71	13.28 / 10.90	6.61 / 6.15	6.74 / 6.65	4.85 / 4.77
	MnasNet	10.69	12.07 / 10.86	7.16 / 5.93	2.71 / 2.54	1.91 / 1.79
	MobileNetV2	6.43	8.87 / 7.34	6.73 / 5.65	4.17 / 3.66	2.90 / 2.30
	MobileNetV3	35.27	14.57 / 13.17	9.06 / 8.72	9.07 / 9.03	8.01 / 7.71

Table 1: Out-of-domain Latency Prediction on NNLQ ( Liu et al., 2022 ) . Models are trained on 9 domains and tested on the 10th, hence a "Test Domain" of AlexNet means that the models were trained on all non AlexNet models, with zero shot prediction being performed on AlexNet. When applicable, metrics (MAPE and Acc (10%)) are reported on the average and best over a set of 10 independent trials.

Metric	Test Domain	NNLP Liu et al. (2022) (avg / best)	NAR-Former V2 Yi et al. (2023b) (avg / best)	NN-Former Xu et al. (2025) (avg / best)	FeatureFormer (avg / best)
MAPE $\downarrow$	AlexNet	10.70 / 8.06	16.68 / 12.44	7.74 / 5.94	7.29 / 5.42
	EfficientNet	6.45 / 6.02	4.49 / 4.09	3.24 / 2.80	2.89 / 2.47
	GoogLeNet	3.89 / 3.49	3.44 / 3.31	3.83 / 3.45	3.63 / 3.35
	MnasNet	9.18 / 7.81	4.49 / 4.01	3.28 / 2.72	3.10 / 2.72
	MobileNetV2	5.98 / 4.94	3.78 / 3.42	3.34 / 2.93	3.04 / 2.70
	MobileNetV3	13.75 / 7.36	8.23 / 7.26	4.59 / 4.36	5.18 / 4.79

Table 2: Out-of-domain Energy Prediction on NNEQ. The testing procedure is shared with that of table 1

Dataset	Metric	NNLP Liu et al. (2022) (avg / best)	NAR-Former V2 Yi et al. (2023b) (avg / best)	NN-Former Xu et al. (2025) (avg / best)	FeatureFormer (avg / best)
NNLQ	MAPE $\downarrow$	3.47 / 3.44	3.07 / 3.00	2.85 / 2.65	2.57 / 2.46
NNLQ	Acc (10%) $\uparrow$	95.25 / 95.50	96.41 / 96.30	97.45 / 97.85	97.58 / 97.95
NNEQ	MAPE $\downarrow$	3.70 / 3.57	3.02 / 2.89	2.77 / 2.50	2.14 / 2.06
NNEQ	Acc (10%) $\uparrow$	95.89 / 97.05	97.91 / 98.45	98.46 / 99.30	99.31 / 99.60

Table 3: In-domain latency and energy prediction on NNLQ ( Liu et al., 2022 ) and NNEQ. As in tables 1 and 2 , each result is reported over 10 independent experiments.

Metric		NNLP Liu et al. (2022) (avg / best)		NAR-Former V2 Yi et al. (2023b) (avg / best)		NN-Former Xu et al. (2025) (avg / best)		FeatureFormer (avg / best)
MAPE $\downarrow$		10.07 / 8.11 ( -1.48 / -2.16 )		8.94 / 7.79 ( -1.61 / -1.33 )		7.60 / 6.94 ( -0.79 / -1.02 )		7.24 / 6.48 ( -0.39 / -0.23 )
Acc (10%) $\uparrow$		62.66 / 71.05 ( +8.03 / +10.40 )		69.43 / 75.46 ( +6.46 / +8.06 )		75.70 / 79.84 ( +1.39 / +4.37 )		77.19 / 81.45 ( +2.54 / +1.48 )

Table 4: Performance of various models on NNLQ ( Liu et al., 2022 ) (out-of-domain) with node-level computational information (

F_{comp}

) added. Due to space constraints, results for individual model families are omitted here. Please refer to the appendix for full results.

Metric		NNLP Liu et al. (2022) (avg / best)		NAR-Former V2 Yi et al. (2023b) (avg / best)		NN-Former Xu et al. (2025) (avg / best)		FeatureFormer (avg / best)
MAPE $\downarrow$		7.48 / 6.05 ( -0.33 / -0.18 )		6.88 / 5.88 ( -1.09 / -1.03 )		7.88 / 6.17 ( +0.04 / -0.15 )		6.63 / 5.86 ( -0.17 / -0.20 )
Acc (10%) $\uparrow$		73.97 / 82.46 ( +2.25 / +1.85 )		81.29 / 86.56 ( +5.48 / +5.00 )		78.25 / 86.01 ( -0.25 / +1.24 )		82.43 / 86.18 ( +0.63 / +0.45 )

Table 5: Performance of various models on NNEQ (out-of-domain) with node-level computational information (

F_{comp}

) added.

Metric		No Gating (avg / best)		Gating (avg / best)
MAPE $\downarrow$		7.04 / 6.08		6.63 / 5.86
Acc (10%) $\uparrow$		80.66 / 84.87		82.43 / 86.18

Table 6: Effect of gating on FeatureFormer. Results are reported on the NNEQ dataset (out-of-domain), with average and best reported over 10 independent trials.

Metric (avg / best)	Global	$A+A^{T}+A^{T}A+AA^{T}$	$A,A^{T},A^{T}A,AA^{T}$
MAPE $\downarrow$	8.45 / 7.92	6.92 / 6.40	6.28 / 5.92
Acc (10%) $\uparrow$	75.86 / 78.49	81.13 / 83.67	84.12 / 86.08

Table 7: Effect of different masking schemes on FeatureFormer . Results are reported on the NNEQ dataset (out-of-domain), utilizing 4 attention heads in each configuration.

	$F_{op}$	$F_{attr}$	$F_{comp}$	MAPE $\downarrow$	Acc (10%) $\uparrow$
	Components			Metrics (avg / best)
(1)	✗	✗	✓	19.77 / 18.49	33.93 / 38.23
(2)	✗	✓	✗	8.18 / 7.66	71.65 / 74.90
(3)	✗	✓	✓	7.14 / 6.70	77.80 / 80.24
(4)	✓	✗	✗	27.38 / 22.69	27.53 / 29.72
(5)	✓	✗	✓	18.54 / 17.64	36.32 / 39.39

Table 8: Importance of

F_{op}

F_{attr}

, and

F_{comp}

to FeatureFormer on NNLQ ( Liu et al., 2022 ) (out-of-domain).

	FLOPs	Parameters	Memory	MAPE $\downarrow$	Acc (10%) $\uparrow$
	Components			Metrics (avg / best)
(1)	✗	✗	✗	7.70 / 6.97	74.26 / 78.36
(2)	✗	✗	✓	7.46 / 7.08	75.89 / 78.20
(3)	✗	✓	✗	7.60 / 6.87	75.75 / 79.91
(4)	✗	✓	✓	7.57 / 6.96	75.30 / 78.92
(5)	✓	✗	✗	7.55 / 6.93	74.87 / 78.38

Table 9: Results of different makeups of

F_{comp}

on FeatureFormer on NNLQ ( Liu et al., 2022 ) (out-of-domain). Memory refers to a proxy of memory operations within the model, consisting of the multiplied output shape + parameter count.

Model	Layers	Parameters (Millions)	MAPE	Acc (10%)
NN-Former ( Xu et al., 2025 )	2 –	10.1	7.47 / 7.09	76.63 / 79.22
NN-Former ( Xu et al., 2025 )	3 $\uparrow$	14.3	7.53 / 7.21	76.31 / 77.96
FeatureFormer	2 $\downarrow$	8.53	7.45 / 7.03	76.64 / 79.14
FeatureFormer	4 –	15.4	7.20 / 6.59	77.40 / 80.90

Table 10: Comparison of NN-Former ( Xu et al., 2025 ) and FeatureFormer at comparable parameter counts on NNLQ ( Liu et al., 2022 ) , both using node-level static features. Symbols next to Layers indicate default configuration ( – ), scaled up (

\uparrow

), or scaled down (

\downarrow

). The scaled up/down configuration for a given model is compared against the default configuration of the other.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Architecture	Mean	Median	Min	Max	Std	Count
NNEQ	AlexNet	51.45	47.56	6.64	140.04	24.43	2000
	EfficientNet	189.44	184.55	33.70	414.06	45.44	2000
	GoogLeNet	111.69	108.26	27.60	267.25	29.86	2000
	MnasNet	128.72	121.62	17.67	367.98	42.30	2000
	MobileNetV2	113.31	108.84	16.09	281.15	33.17	2000
	MobileNetV3	52.14	46.46	9.94	266.29	25.13	2000

Appendix

Table 11: Summary of NNLQ ( Liu et al., 2022 ) and NNEQ dataset statistics by model family. Statistics are computed over the ground truth values of all samples in each family. "All" aggregates across all families within a dataset.

Figure 7: A comparison of NNLQ ( Liu et al., 2022 ) ground truth latency and NNEQ ground truth energy across individual models. The relationship between these metrics is neither fully linear nor fully proportional, emphasizing the distinct nature of the two predictive tasks.

Metric	Test Domain	NNLP Liu et al. (2022) (avg / best)	NAR-Former V2 Yi et al. (2023b) (avg / best)	NN-Former Xu et al. (2025) (avg / best)	FeatureFormer (avg / best)
MAPE $\downarrow$	AlexNet	10.93 / 9.99	13.16 / 11.74	10.39 / 9.84	10.15 / 9.51
		( +0.29 / +0.28 )	( -11.12 / -6.55 )	( -1.08 / -1.33 )	( -1.68 / -1.84 )
	EfficientNet	19.90 / 15.15	6.13 / 5.41	4.15 / 3.87	4.61 / 3.53
		( -1.56 / -3.57 )	( -7.07 / -5.96 )	( -0.98 / -0.94 )	( -0.05 / -0.27 )
	GoogLeNet	11.26 / 10.00	12.59 / 8.48	5.62 / 5.08	4.85 / 4.77
		( -2.02 / -0.90 )	( +5.98 / +2.33 )	( -1.12 / -1.57 )	( -0.16 / -0.01 )

Appendix

Table 12: Full performance of various models on NNLQ ( Liu et al., 2022 ) (out-of-domain) with node-level computational information (

F_{comp}

) added. Results for each model are reported over 10 independent trials.

Metric	Test Domain	NNLP Liu et al. (2022) (avg / best)	NAR-Former V2 Yi et al. (2023b) (avg / best)	NN-Former Xu et al. (2025) (avg / best)	FeatureFormer (avg / best)
MAPE $\downarrow$	AlexNet	11.47 / 8.43	7.96 / 5.42	8.17 / 5.93	7.29 / 5.42
		( +0.77 / +0.37 )	( -8.73 / -7.02 )	( +0.43 / -0.01 )	( +0.22 / -0.72 )
	EfficientNet	5.42 / 4.86	2.71 / 2.47	3.02 / 2.79	2.89 / 2.47
		( -1.03 / -1.16 )	( -1.78 / -1.62 )	( -0.22 / -0.01 )	( -1.07 / -0.67 )
	GoogLeNet	3.67 / 3.42	3.83 / 3.53	3.60 / 3.25	3.63 / 3.35
		( -0.23 / -0.07 )	( +0.39 / +0.22 )	( -0.24 / -0.20 )	( +0.03 / +0.02 )

Appendix

Table 13: Full performance of various models on NNEQ (out-of-domain) with node-level computational information (

F_{comp}

) added. Results for each model are reported over 10 independent trials.

Metric	Test Domain	Without Gating (avg / best)	With Gating (avg / best)
MAPE $\downarrow$	AlexNet	9.26 / 7.12	7.29 / 5.42
	EfficientNet	3.11 / 2.51	2.89 / 2.47
	GoogLeNet	3.49 / 3.19	3.63 / 3.35
	MnasNet	3.42 / 2.50	3.10 / 2.72
	MobileNetV2	2.73 / 2.12	3.04 / 2.70
	MobileNetV3	5.03 / 4.66	5.18 / 4.79

Appendix

Table 14: Effect of gating on FeatureFormer. Results are reported on the NNEQ dataset (out-of-domain), with average and best reported over 10 independent trials.

Metric	Test Domain	Global (avg / best)	$A+A^{T}+A^{T}A+AA^{T}$ (avg / best)	$A,A^{T},A^{T}A,AA^{T}$ (avg / best)
MAPE $\downarrow$	AlexNet	8.62 / 7.61	6.49 / 5.60	5.67 / 5.42
	EfficientNet	4.55 / 4.04	3.33 / 3.18	2.62 / 2.48
	GoogLeNet	3.59 / 3.39	3.57 / 3.43	3.57 / 3.37
	MnasNet	4.18 / 3.92	3.55 / 3.33	2.88 / 2.79
	MobileNetV2	3.76 / 3.61	3.32 / 3.24	3.08 / 3.03
	MobileNetV3	5.97 / 5.61	5.09 / 4.82	5.33 / 4.79

Appendix

Table 15: Effect of different masking schemes on FeatureFormer . Results are reported on the NNEQ dataset (out-of-domain) over 3 independent trials, utilizing 4 attention heads in each configuration.

Metric	Test Domain	(1) $F_{comp}$ (avg / best)	(2) $F_{attr}$ (avg / best)	(3) $F_{attr}$ , $F_{comp}$ (avg / best)	(4) $F_{op}$ (avg / best)	(5) $F_{op}$ , $F_{comp}$ (avg / best)	(6) $F_{op}$ , $F_{attr}$ (avg / best)	(7) $F_{op}$ , $F_{attr}$ , $F_{comp}$ (avg / best)
MAPE $\downarrow$	AlexNet	20.68 / 20.03	11.67 / 11.35	10.39 / 9.87	18.96 / 17.55	16.97 / 15.87	11.82 / 11.43	10.34 / 9.51
	EfficientNet	21.08 / 20.87	5.28 / 4.63	5.36 / 4.87	21.73 / 21.33	19.29 / 18.69	4.76 / 4.56	4.23 / 3.63
	GoogLeNet	18.96 / 18.69	5.25 / 4.92	5.05 / 4.97	20.58 / 18.72	18.57 / 18.28	4.94 / 4.79	4.84 / 4.82
	MnasNet	15.71 / 15.47	2.01 / 1.97	1.93 / 1.85	21.07 / 20.99	16.72 / 16.48	2.04 / 1.89	1.88 / 1.83
	MobileNetV2	14.63 / 14.33	2.96 / 2.48	2.94 / 2.41	19.95 / 19.87	14.04 / 13.92	4.05 / 3.29	2.88 / 2.50
	MobileNetV3	29.12 / 27.86	7.85 / 7.68	7.95 / 7.78	27.86 / 27.71	28.75 / 27.50	8.33 / 7.97	8.09/ 8.03

Appendix

Table 16: Importance of

F_{op}

F_{attr}

, and

F_{comp}

to FeatureFormer on NNLQ ( Liu et al., 2022 ) (out-of-domain). Results are reported over 3 independent trials. It should be noted that all results on this page correspond to the same table which has been split in half due to space constraints.

Metric	Test Domain	(1) No $F_{comp}$ (avg / best)	(2) Mem (avg / best)	(3) Params (avg / best)	(4) Params + Mem (avg / best)	(5) FLOPs (avg / best)	(6) FLOPs + Mem (avg / best)	(7) FLOPs + Params (avg / best)	(8) FLOPs + Params + Mem (avg / best)
MAPE $\downarrow$	AlexNet	11.82 / 11.43	10.50 / 10.15	10.54 / 10.19	10.26 / 9.70	11.44 / 11.25	10.74 / 10.04	10.81 / 10.27	10.34 / 9.51
	EfficientNet	4.76 / 4.56	4.75 / 4.55	4.96 / 3.79	4.83 / 3.97	5.14 / 4.01	4.36 / 4.18	4.54 / 3.67	4.23 / 3.63
	GoogLeNet	4.94 / 4.79	4.88 / 4.86	4.87 / 4.77	4.87 / 4.81	4.83 / 4.72	5.25 / 4.90	4.80 / 4.64	4.84 / 4.82
	MnasNet	2.04 / 1.89	1.97 / 1.94	1.90 / 1.82	1.97 / 1.89	1.93 / 1.86	1.89 / 1.85	1.85 / 1.84	1.88 / 1.83
	MobileNetV2	4.05 / 3.29	4.07 / 3.61	3.89 / 3.30	2.81 / 2.66	3.18 / 2.28	3.36 / 2.93	2.69 / 2.50	2.88 / 2.50
	MobileNetV3	8.33 / 7.97	8.69 / 8.47	8.88 / 8.63	8.36 / 7.90	7.68 / 7.25	7.89 / 7.72	7.90 / 7.60	8.09 / 8.03

Appendix

Table 17: Results of different makeups of

F_{comp}

on FeatureFormer on NNLQ ( Liu et al., 2022 ) (out-of-domain). Memory (abbreviated "Mem") refers to a proxy of memory operations within the model, consisting of the multiplied output shape + parameter count. Results are reported over 3 independent trials. It should be noted that all results on this page correspond to the same table which has been split in half due to space constraints.

Neural Architecture Performance Prediction

See all 13 papers

DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning
Oct 12, 2026Yuxiang Zeng, Kun Xie, Yang Wang +5Neural Architecture Performance Prediction Graph Transformer
Hunan University · Hunan University of Science and Technology · Chinese Academy of Sciences +1
HiFi-LLP: High-Fidelity, Low-Cost Latency Predictors with Confidence for Robust HW-NAS
Jul 13, 2026Shambhavi Balamuthu Sampath, Behzad Shomali, Nael Fasfous +7Confidence Estimation Hardware-Aware NAS
Technical University of Munich · BMW Group · University of Bonn
EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
Jul 10, 2026Shuo Huai, Hao Kong, Shiqing Li +5Efficient Neural Network Inference Edge Computing
School of Computer Science and Engineering, Nanyang Technological University, Singapore · HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University, Singapore · HP Inc., Palo Alto, California, USA

Oct 12, 2026J/K move · Enter open · S save

DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning

Yuxiang Zeng, Kun Xie, Yang Wang, Jigang Wen+4

Hunan University · Hunan University of Science and Technology · Chinese Academy of Sciences+1

Neural Architecture Performance Prediction Graph Transformer Graph Representation Learning Neural Architecture Search

Open paper AskPDF arXiv

Architecture attribute predictors reduce the cost of neural architecture search by estimating accuracy or latency from a small set of evaluated candidates. The key difficulty is that neural architectures are operation-labeled DAGs: edge direction defines computational flow, but existing predictors often treat direction as a local structural cue. We propose DTA-GT, a Direction- and Topology-Aware Graph Transformer based on encoder-wide directional consistency, which preserves directed computational semantics across node initialization, global spectral encoding, pairwise interaction, and feature update. DTA-GT realizes this principle with direction- and topology-aware node initialization, Magnetic Laplacian spectral encoding, direction-dual structural attention, and a Direction-Sensitive MoE-FFN. Across accuracy and latency prediction on NAS-Bench-101 and NAS-Bench-201, DTA-GT consistently outperforms representative sequence-, GNN-, Transformer-, and hybrid predictors under limited-budget settings. Ablations and Magnetic spectral controls show that the improvements come from coordinated direction-aware stages and edge-orientation-sensitive spectral information, while mechanistic diagnostics confirm that the modules exhibit the intended directional behavior. These results support encoder-wide directional consistency as an effective design principle for architecture DAG representation.

Abstract

Neural Architecture Performance Prediction

Explore similar work

DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning

HiFi-LLP: High-Fidelity, Low-Cost Latency Predictors with Confidence for Robust HW-NAS

EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems