cs.CVFeb 27, 2026

Percept-Aware Surgical Planning for Visual Cortical Prostheses with Vascular Avoidance

Authors: Galen Pogoncheff, Alvin Wang, Jacob Granley, Michael Beyeler

Organizations: Department of Computer Science, University of California Santa, Barbara · School of Computer Science, Carnegie Mellon University · Department of Psychological & Brain Sciences, University of California, Santa Barbara

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Cortical visual prostheses aim to restore sight by electrically stimulating neurons in early visual cortex (V1). With the emergence of high-density and flexible neural interfaces, electrode placement within three-dimensional cortex has become a critical surgical planning problem. Existing strategies emphasize visual field coverage and anatomical heuristics but do not directly optimize predicted perceptual outcomes under safety constraints. We present a percept-aware framework for surgical planning of cortical visual prostheses that formulates electrode placement as a constrained optimization problem in anatomical space. Electrode coordinates are treated as learnable parameters and optimized end-to-end using a differentiable forward model of prosthetic vision. The objective minimizes task-level perceptual error while incorporating vascular avoidance and gray matter feasibility constraints. Evaluated on simulated reading and natural image tasks using realistic folded cortical geometry (FreeSurfer fsaverage), percept-aware optimization consistently improves reconstruction fidelity relative to coverage-based placement strategies. Importantly, vascular safety constraints eliminate margin violations while preserving perceptual performance. The framework further enables co-optimization of multi-electrode thread configurations under fixed insertion budgets. These results demonstrate how differentiable percept models can inform anatomically grounded, safety-aware computer-assisted planning for cortical neural interfaces and provide a foundation for optimizing next-generation visual prostheses.

Figures & tables

Figure 1: Percept-aware optimization framework. Given 3D electrode coordinates on FreeSurfer fsaverage anatomy, target percepts, and patient retinotopy, a differentiable model of cortical prosthetic vision predicts elicited percepts. Electrode locations are iteratively updated to minimize perceptual error while enforcing vascular safety constraints. Solid arrows: forward simulation pathway. Dashed arrows: gradient signals used to update electrode positions.
BaselineDatasetΔ\Delta MSE (%)(\%) ↓\downarrowΔ\Delta SSIM (%)(\%) ↑\uparrowΔ\Delta Acc. (%)(\%) ↑\uparrow
VF TilingMNIST-67.7* [-73.6, -47.2]11.1* [7.8, 16.7]62.6* [28.9, 119.9]
VF Coverage-33.4* [-42.0, -16.3]4.7* [3.1, 7.3]22.4* [7.8, 34.7]
VF TilingCIFAR-10-58.4* [-87.4, -24.1]57.4* [35.3, 102.2]10.2* [5.5, 29.2]
VF Coverage16.0 [-25.2, 39.1]-10.6 [-24.7, 0.2]-2.2 [-9.1, 6.25]
Table 1: Relative change in MSE, SSIM, and downstream classification accuracy (Acc.) of the proposed method versus baselines. Values are median percent difference [IQR] across electrode counts and phosphene spreads ( ρ\rho ). * indicates p≤0.01p\leq 0.01 .
Figure 2: Comparing percept-aware electrode optimization and Visual Field Tiling. Left : Simulated phosphenes for reading (MNIST) and natural image (CIFAR-10) tasks. Right : Relative improvements in perceptual fidelity across experimental configurations.
Figure 3: Safety-aware electrode optimization with vascular constraints. Left : Cortical surface ( fsaverage left occipital) with high-resolution vascular map (red, displayed in V1 only) and optimized electrode locations (black). Right : Percept SSIM scores without (solid lines) and with (dashed lines) vascular avoidance.
Figure 4: Simulated phosphenes using percept-aware optimization with and without multi-electrode threads under a fixed number of cortical insertions ( 128128 ). Threaded designs improve percept quality without increasing number of cortical insertions.
More in this topic

Brain-Computer Interfaces

Brain-computer interfaces translate neural activity into commands or feedback for communication, movement, rehabilitation, and sensory restoration.

See all 130 papers

Explore similar work

CardsList
Sep 18, 2026

A Fully Differentiable Neuro-Soft-Symbolic Framework for Perceptual Task Planning

Perceptual planning tasks require two key capabilities: accurately perceiving uncertain scenes and planning valid action sequences following logical rules. Conventional methods convert perception into discrete symbolic facts and then plan, discarding perceptual uncertainty and severing task-level feedback to perception. We introduce a generic, fully differentiable neuro-soft-symbolic framework that connects visual perception and task planning within a single computational graph. The framework maintains a continuous soft symbolic state, lifts domain rules into a differentiable soft-TPT_P transition operator, and optimizes action logits over a short planning horizon. Gradients from the planning objective can also update the perception parameters, allowing task-relevant perceptual representations to be refined during planning. On Blocksworld, our method solves 40/40 LatPlan-40 tasks and 596/600 PlanBench-600 tasks, compared with 33/40 for LatPlan and 587/600 for the reasoning-model baseline, while requiring substantially less computation and time. In the perceptual-uncertainty ablation, our method improves the success rate from 59% with frozen perception to 83%. We further conduct task-and-motion simulations on Blocksworld scenes, providing an execution-level validation of the compatibility between decoded task plans and downstream robotic motion execution.

Hongyan Wei, Wael AbdAlmageed

Clemson University

Neuro-Symbolic PlanningRobot Task PlanningNeuro-Symbolic ReasoningRobotic Perception
AskPDF
Sep 21, 2026

Adaptive Cortically Constrained EEG-Vision Alignment for Zero-Shot Brain-to-Image Retrieval

Zero-shot brain-to-image retrieval requires robust alignment between noisy EEG responses and visual representations. Existing EEG-vision alignment methods often operate in sensor space and apply fixed visual supervision to all responses, ignoring both spatial mixing in scalp EEG and response-wise variability in alignment reliability. We propose an adaptive cortically constrained EEG-vision alignment method for zero-shot brain-to-image retrieval. The method reconstructs EEG responses into predefined ROI-level source-pattern representations and encodes them with a Neuro-ROI Attention Encoder. To handle response-wise variability, we introduce an evidence-based adaptive visual supervision strategy that weights detail-controlled visual targets using model-based alignment evidence. On THINGS-EEG, the proposed method achieves strong 200-way zero-shot retrieval performance, with ROI-level attribution providing post hoc interpretability of the learned source-pattern representations. These results show that cortically constrained representation learning and adaptive supervision can jointly support EEG-vision alignment for zero-shot brain-to-image retrieval.

Ye Wang, Haokun Ren, Wei Wu +4

School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · School of Medicine, Shanghai Jiaotong University, Shanghai, China · National Center for Applied Mathematics in Chongqing, Chongqing Normal University, Chongqing, China +1

ElectroencephalographyEEG-Based Visual DecodingCross-Modal AlignmentCross-Modal Retrieval
AskPDF
Jul 5, 2026

SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.

Lei Song, Yonghao Long, Mengya Xu +3

Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Thoracic Surgery, Peking University People’s Hospital, Beijing, China · Thoracic Oncology Institute, Peking University People’s Hospital, Beijing, China +3

Robotic PerceptionSurgical Robotics
AskPDF