Using Laplace Transform To Optimize the Hallucination of Generation Models
Authors: Cheng Kang, Xinye Chen, Daniel Novak, Xujing Yao
Organizations: 1Cheng Kang and Daniel Novak are with the Department of Cybernetics, Czech Technical University in Prague, 16000 Praha 6, Prague, Czech Republic · 2Xinye Chen is with Department of Numerical Mathematics, Charles University, 11000 Praha 1, Prague, Czech Republic · 3Xujing Yao is with the College of Computer and Information Engineering, Nanjing Tech University, Nanjing, Jiangsu 211816, China
To explore the feasibility of avoiding the confident error (or hallucination) of generation models (GMs), we formalise the system of GMs as a class of stochastic dynamical systems through the lens of control theory. Numerous factors can be attributed to the hallucination of the learning process of GMs, utilising knowledge of control theory allows us to analyse their system functions and system responses. Due to the high complexity of GMs when using various optimization methods, we cannot figure out their solution of Laplace transform, but from a macroscopic perspective, simulating the source response provides a virtual way to address the hallucination of GMs. We also find that the training progress is consistent with the corresponding system response, which offers us a useful way to develop a better optimization component. Finally, the hallucination problem of GMs is fundamentally optimized by using Laplace transform analysis.
Figures & tables
Figure 1: Evolution process of the weight θ(t) for each component under the generation task.
Figure 2: The control system of classical GAN.
Figure 3: The system response of Classical GAN on various optimizers. Optimizer from left to right is respectively SGD, SGDM, Adam, PID, LPFSGD, HPFSGD, and FuzzyPID. The input of generator is a random Gaussian noise, and the input of discriminator is a square wave. (Blue wave comes from the discriminator, and yellow wave is the response of the generator.)
Figure 4: The control system of CycleGAN.
Figure 5: The system response of CycleGAN on different optimizers, such as SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD and FuzzyPID optimizers. For Ga , the source is a sinusoidal wave (green), and the dashed green wave comes from Da . For Ga , the source is a cosine wave (red), and the dashed red wave comes from Db .
Hyper-parameter
GAN
CycleGAN
DDPM
Data Name
MNIST
MNIST and UPSP
MNIST
Data augmentation
Auto
Auto
Auto
Input resolution
[28,28,1]
[28,28,1]
[28,28,1]
Epochs
200
200
100
Batch size
200
200
200
Hidden dropout
0
0
0
Table I : Hyper-parameters for the image generation task on MNIST and UPSP.
Filter Type
Gain
Numerator
Denominator
G
x0
x1
x2
y0
y1
y2
Low Pass Filter
0.49968
1
-0.99937
0.00063
1.0
0
-1.0
High Pass Filter
0.49968
1
0.99937
0.00063
1.0
0
-1.0
Table II : Coefficients of LPF-SGD and HPF-SGD using second-order IIR structure.
Figure 6: The generated samples from classical GAN on corresponding optimizers (from top to bottom is respectively SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD, and FuzzyPID).
Figure 7: The generated samples from CycleGAN on corresponding optimizers (From top to bottom is SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD and FuzzyPID). The generated confident errors are marked with a red line.
Figure 8: The generated samples from DDPM on corresponding optimizers (From top to bottom is SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD and FuzzyPID).
Figure 9: The generated samples from DDPM on corresponding optimizers (From top to bottom is SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD and FuzzyPID).
Figure 10: The generated samples from DDPM on corresponding optimizers (From top to bottom is SGD, SGDM, Adam, PID, LPF-SGD, HPF-SGD and FuzzyPID).
Recent work on hallucination detection in large language models has shown that, for a fixed pre-trained model and reasoning task, it is possible to estimate the model's confidence in the correctness of its outputs. Such uncertainty estimates have primarily been used to improve truthfulness by detecting or filtering confabulations. In this work, we ask whether these signals can instead be used more proactively to directly improve the accuracy of model-generated answers. We propose USteer, a simple, training-free steering mechanism that adjusts a model's layer-wise activations during inference using the gradient of a confidence measure with respect to the activations. This procedure nudges generation toward outputs with lower uncertainty at inference time, without modifying model parameters or requiring additional supervision. We show that this approach consistently reduces hallucination across a range of tasks, demonstrating that confidence signals can be leveraged not only for detection, but also for effective inference-time control of model behavior.
Large Vision-Language Models (LVLMs) exhibit powerful generative capabilities but frequently produce hallucinations that compromise output reliability. Fine-tuning on annotated data devoid of hallucinations offers the most direct solution, while its high computational cost motivates recent representation-based methods, which focus on mitigating hallucinatory components within hidden representations. Though efficient, we empirically observe that these methods degrade general generation capacity due to incomplete extraction of hallucination components and non-selective parameter updates. To address these limitations, we propose MPD, a dual-stage framework for mitigating hallucinations without performance degradation. Specifically, our MPD relies on two essential factors: (1) semantic-aware component disentanglement to extract pure hallucination components, and (2) interpretable parameter updates that selectively modify parameters most relevant to hallucination. Extensive experiments demonstrate that MPD achieves state-of-the-art performance, reducing hallucinations by 23.4% while maintaining 97.4% of general generative capability as evaluated on LLaVA-Bench and MME, with no additional computational cost.
Xingyu Zhu, Junfeng Fang, Shuo Wang +4
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore
Large language models (LLMs) hallucinate: they produce fluent outputs that are factually incorrect. We present a geometric dynamical systems framework in which hallucinations arise from task-dependent basin structure in latent space. Using autoregressive hidden-state trajectories across multiple open-source models and benchmarks, we find that separability is strongly task-dependent rather than universal: factoid settings can show clearer basin separation, whereas summarization and misconception-heavy settings are typically less stable and often overlap. We formalize this behavior with task-complexity and multi-basin theorems, characterize basin emergence in L-layer transformers, and show that geometry-aware steering can reduce hallucination probability without retraining.
Kalyan Cherukuri, Lav R. Varshney
Illinois Mathematics and Science Academy, Illinois, USA · AI Innovation Institute, Stony Brook University, New York, USA