NeoNet: An End-to-End 3D MRI-Based Deep Learning Framework for Non-Invasive Prediction of Perineural Invasion via Generation-Driven Classification

Youngung Han; Minkyung Cha; Kyeonghun Kim; Induk Um; Myeongbin Sho; Joo Young Bae; Jaewon Jung; Jung Hyeok Park

cs.CVMar 31, 2026

NeoNet: An End-to-End 3D MRI-Based Deep Learning Framework for Non-Invasive Prediction of Perineural Invasion via Generation-Driven Classification

Authors: Youngung Han, Minkyung Cha, Kyeonghun Kim, Induk Um, Myeongbin Sho, Joo Young Bae, Jaewon Jung, Jung Hyeok Park, +7 more

Organizations: Seoul National University, South Korea · OUTTA, South Korea · Chung-Ang University, South Korea · Sookmyung Women’s University, South Korea · Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea · Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea · Samsung Changwon Hospital, Sungkyunkwan University School of Medicine · NVIDIA AI Technology Center, Taipei, Taiwan

Abstract

Minimizing invasive diagnostic procedures to reduce the risk of patient injury and infection is a central goal in medical imaging. And yet, noninvasive diagnosis of perineural invasion (PNI), a critical prognostic factor involving infiltration of tumor cells along the surrounding nerve, still remains challenging, due to the lack of clear and consistent imaging criteria criteria for identifying PNI. To address this challenge, we present NeoNet, an integrated end-to-end 3D deep learning framework for PNI prediction in cholangiocarcinoma that does not rely on predefined image features. NeoNet integrates three modules: (1) NeoSeg, utilizing a Tumor-Localized ROI Crop (TLCR) algorithm; (2) NeoGen, a 3D Latent Diffusion Model (LDM) with ControlNet, conditioned on anatomical masks to generate synthetic image patches, specifically balancing the dataset to a 1:1 ratio; and (3) NeoCls, the final prediction module. For NeoCls, we developed the PNI-Attention Network (PattenNet), which uses the frozen LDM encoder and specialized 3D Dual Attention Blocks (DAB) designed to detect subtle intensity variations and spatial patterns indicative of PNI. In 5-fold cross-validation, NeoNet outperformed baseline 3D models and achieved the highest performance with a maximum AUC of 0.7903.

Figures & tables

Figure 1: Morphological features of perineural invasion (PNI) demonstrated by histopathology (a), CT (b), and MRI (c-PNI negative, d-PNI positive). Arrows indicate the locations of characteristic PNI features in each modality.

Figure 2: Overview of the NeoNet framework. (1) NeoSeg performs automated liver and tumor segmentation on inputs (e.g., $240\times 240\times 100$ ), producing segmentation labels (e.g., $240\times 240\times 100$ ). The TLCR algorithm utilizes these labels and the original image to extract localized image and label patches ( $96\times 96\times 48$ ). (2) NeoGen utilizes a 3D-LDM (Step #1) and trains ControlNet (Step #2) conditioned on label patches to generate synthetic image patches, balancing the dataset. (3) NeoCls utilizes PattenNet, initialized with the frozen LDM encoder ( $\mathcal{E}$ ), to predict PNI status from real and synthetic patches.

Figure 3: Architecture of PattenNet for PNI prediction. The model utilizes a frozen encoder from the 3D-LDM followed by two Dual Attention Blocks (DAB). Each DAB integrates 3D channel and spatial attention. Spatial attention utilizes a 3D convolution ( $7\times 7\times 7$ ) to effectively model 3D spatial relationships. The attended features are processed through global pooling and an MLP for binary classification.

Model	Mean Dice
U-Net	0.9453
SegResNet	0.9416
DynUNet	0.9482
SwinUNETR	0.9516

Table 1: Performance comparison of different segmentation models for liver and tumor segmentation.

Figure 4: Labels generated by NeoSeg. Real Image (left), Segmentation masks (middle: peritumoral mask in red, tumor in yellow), and Overlay (right).

Model	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
QNeoGen VQVAE	0.3003	0.4958	15.2210
NeoGen VAE	0.2010	0.7408	22.7427

Table 2: Reconstruction quality comparison between NeoGen VAE and QNeoGen VQVAE.

Method	FID $\downarrow$ (A)	FID $\downarrow$ (S)	FID $\downarrow$ (C)	FID $\downarrow$ (Avg.)
QNeoGen DM	18.9751	22.1884	39.2260	26.7965
QNeoGen	14.5820	18.4123	28.6891	20.5611
NeoGen DM	7.6284	10.8560	17.8180	12.1008
NeoGen	2.9440	4.8279	8.2223	5.3314

Table 3: Fréchet Inception Distance (FID) comparison across Axial (A), Sagittal (S), and Coronal (C) views.

Figure 5: Real Patch (a) and Synthetic Patch (b) generated by NeoGen using the corresponding anatomical mask. Morphological features are highlighted by red arrows.

Model (3D)	R (Imbalanced)	R + S (25%)	R + S (50%)	R + S (75%)	R + S (100% Balanced)
ResNet-50	0.6938	0.7388	0.7599	0.7551	0.7618
ResNet-101	0.6284	0.6989	0.7024	0.7117	0.7203
ResNet-152	0.7078	0.7233	0.7421	0.7382	0.7526
ResNet-200	0.6687	0.7250	0.7307	0.7397	0.7412
DenseNet-121	0.6788	0.7443	0.7380	0.7490	0.7551
DenseNet-169	0.6725	0.7021	0.7387	0.7434	0.7584

Table 4: AUC comparison of PNI classifiers trained on different R (Real) to S (Synthetic) ratios using 5-fold cross-validation. R+S columns represent progressive augmentation towards a fully balanced (1:1) dataset (R+S 100% Balanced). Results are presented as Mean AUC. All baseline models are 3D implementations.

Configuration	AUC (R+S)
PattenNet (Full Model, 2 DABs)	0.7903
DAB Component Ablation (2 Blocks)
w/ Channel Attention only	0.7276
w/ Spatial Attention only	0.7410
Number of DABs
0 Blocks (Base LDM Encoder)	0.7001

Table 5: Ablation study evaluating the impact of attention components and the number of Dual Attention Blocks (DABs) in PattenNet.

Explore similar work

LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRI
Jul 13, 2026Youngung Han, Hyunsu Go, Kyeonghun Kim +9Self-Attention Multi-Scale Feature Fusion
Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea +3
Anatomy-Privileged Distillation with Token Routing for MRI-Based Prediction of Perineural Invasion
Jul 13, 2026Hyunsu Go, Youngung Han, Kyeonghun Kim +15Medical Image Classification Clinical Prediction
Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Kyung Hee University, Seoul, Republic of Korea +5
SCINTILLA-SNN: A Spiking Multi-Scale Selective Aggregation Network for Perineural Invasion Prediction
Sep 10, 2026Youngung Han, Yului Jeong, Kyeonghun Kim +11Multi-Scale Feature Fusion Spiking Neural Networks
Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Kyung Hee University, Seoul, Republic of Korea +2

Jul 13, 2026J/K move · Enter open · S save

LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRI

Youngung Han, Hyunsu Go, Kyeonghun Kim, Induk Um+8

Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea+3

Self-Attention Multi-Scale Feature Fusion Medical Image Classification Clinical Prediction Oncology

Open paper AskPDF arXiv

Perineural invasion (PNI) is a clinically relevant indicator of tumor aggressiveness and can influence surgical decision-making, motivating interest in reliable preoperative assessment. The subtle MRI features of PNI, however, often resemble nearby anatomy, complicating noninvasive prediction. These fine perineural cues are easily attenuated by routine downsampling or overly global feature aggregation, reducing the effectiveness of conventional volumetric models. We present LoSA-Net, a localized and scale-adaptive architecture for boundary-sensitive PNI prediction in 3D MRI. Talking Neighborhood Attention (TNA) preserves nerve-aligned detail through localized self-attention with head-wise mixing, and Scale-Adaptive Feature Mixing (SAFM) modulates the receptive field using multi-scale depthwise processing. Cross-Scale Refinement and Alignment (CSRA) maintains consistency between semantic context and high-resolution boundaries across stages. In contrast-enhanced MRI scans from 168 patients with cholangiocarcinoma, LoSA-Net achieves an AUC of 0.7567 and outperforms representative convolutional and transformer baselines under matched preprocessing and optimization settings.

Abstract

Explore similar work

LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRI

Anatomy-Privileged Distillation with Token Routing for MRI-Based Prediction of Perineural Invasion

SCINTILLA-SNN: A Spiking Multi-Scale Selective Aggregation Network for Perineural Invasion Prediction