The Segment Anything Model (SAM) exhibits strong zero-shot performance on natural images but suffers from domain shift and overconfidence when applied to medical volumes. We propose \textbf{CalSAM}, a lightweight adaptation framework that (i) reduces encoder sensitivity to domain shift via a \emph{Feature Fisher Information Penalty} (FIP) computed on 3D feature maps and (ii) penalizes overconfident voxel-wise errors through a \emph{Confidence Misalignment Penalty} (CMP). The combined loss,
LCalSAM fine-tunes only the mask decoder while keeping SAM's encoders frozen. On cross-center and scanner-shift evaluations, CalSAM substantially improves accuracy and calibration: e.g., on the BraTS scanner split (Siemens
→GE) CalSAM shows a
+7.4% relative improvement in
DSC (80.1% vs.\ 74.6%), a
−26.9% reduction in
HD95 (4.6 mm vs.\ 6.3 mm), and a
−39.5% reduction in
ECE (5.2% vs.\ 8.6%). On ATLAS-C (motion corruptions), CalSAM achieves a
+5.3% relative improvement in
DSC (75.9%) and a
−32.6% reduction in
ECE (5.8%). Ablations show FIP and CMP contribute complementary gains (
p<0.01), and the Fisher penalty incurs a modest
∼15% training-time overhead. CalSAM therefore delivers improved domain generalization and better-calibrated uncertainty estimates for brain MRI segmentation, while retaining the computational benefits of freezing SAM's encoder.