The Brittle Compass: Navigating LLM Prompt Sensitivity in Slovak Migration Media Discourse
Authors: Samuel Harvan, Jaroslav Kopčan, Marek Šuppa, Andrej Findor
Organizations: Central European University, Vienna, Austria · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Cisco Systems · Comenius University in Bratislava, Slovakia
In this work, we present a case study that explores various tasks centered around the topic of migration in Slovak, a low-resource language, such as topic relevance and geographical relevance classification, and migration source/destination location term extraction. Our results demonstrate that native (Slovak) prompts yield a modest, task-dependent gains, while large models show significant robustness to prompt variations compared to their smaller counterparts. Analysis reveals that instructions (system or task) emerge as the most critical prompt component, more so than the examples sections, with task-specific performance benefits being more pronounced than overall language effects.
Figures & tables
Category
Models
Method
Task
Theme rel.
Geo rel.
Loc ext.
ENG
SVK
ENG
SVK
ENG
SVK
Non-reasoning models
gpt-4o
RAG
0.4518
0.4551
0.9771
0.9769
0.8734
0.8764
FewShot
0.5357
0.5277
0.9771
0.9816
0.8767
0.8675
ZeroShot
0.4413
0.4552
0.9726
0.9816
0.8542
0.8654
gemini-2.5-flash
RAG
0.4709
0.5357
0.9449
0.9882
0.7771
0.8433
Table 1: Results comparing different LLMs across tasks with the English and Slovak prompt versions including reasoning traces for reasoning models and CoT for non reasoning
Task
n
Mean English
Mean Slovak
Difference
95% Bootstrap CI
Significance
Theme Relevance
24
0.499
0.478
-0.021
[-0.058, 0.015]
n.s.
Geo Relevance
12
0.962
0.967
+0.005
[-0.006, 0.015]
n.s.
Location Extraction
27
0.792
0.810
+0.017
[0.003, 0.033]
CI excludes 0
Table 2: Statistical analysis of Slovak vs English F1 performance using bootstrap confidence intervals (2000 resamples). Difference = Slovak - English. "CI excludes 0" indicates statistical significance. Note: Theme relevance shows larger effect size but high variance; Location extraction shows smaller effect but lower variance and larger sample size, explaining significance pattern.
Model
Variant
Macro F1
Δ F1 (%)
gpt-4.1
full
0.9727
–
no task
0.9595
-1.35
no ex
0.9771
0.46
none
0.3261
-66.48
gpt-4.1-mini
full
0.9727
–
no task
0.9111
-6.33
Table 3: Macro F1 scores and percentage delta values for GPT models across different prompt variants in the ablation study. Bold values indicate the highest score for each model. Δ F1 (%) shows the percentage performance drop relative to the full prompt baseline.
Figure 1: Distribution of macro F1 across prompt variants. For each model panel, we plot max min macro F1 (bars) per model and overlay per variant absolute macro F1 (points) on a twin top x axis, with thin lines showing the min–max span. Models are alphabetized.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Distribution of the dataset: character length in the final dataset
Figure 4: Distribution of the dataset: token length in the final dataset. The tokens originate from the SlovakBERT tokenizer.
Figure 5: Final relevance dataset distribution across train, validation, and test splits with consistent class ratios.
Locality
Unknown count
Percentage
Source
717
43.40
Destination
441
26.69
Combined
0
0.00
Appendix
Table 4: The distribution of “Unknown” localities (i.e. when either the Source or Destination field is not filled in for a specific sample) in the various subsets of the final dataset. Note that the result in the Combined row shows that either Source or Destination field are always filled in in the final dataset.
Figure 6: Distribution of the dataset: character length in the final dataset
Figure 7: Distribution of the dataset: token length in the final dataset. The tokens originate from the SlovakBERT tokenizer.
Figure 8: Final relevance dataset distribution across train, validation, and test splits with consistent class ratios.
Model
Variant
Macro F1
Δ F1 (%)
gemini-2.0-flash
full
0.9633
–
no task
0.9458
-1.81
no ex
0.9130
-5.22
none
0.2614
-72.87
gemini-2.0-flash-lite
full
0.9424
–
no task
0.8409
-10.77
Appendix
Table 5: Macro F1 scores and percentage delta values for select models provided by Google
Model
Variant
Macro F1
Δ F1 (%)
llama-4-maverick
full
0.7692
–
no task
0.7504
-2.44
no ex
0.7005
-8.94
none
0.2171
-71.77
llama-4-scout
full
0.9541
–
no task
0.8541
-10.48
Appendix
Table 6: Macro F1 scores and percentage delta values for select models provided by Meta-Llama
Model
Variant
Macro F1
Δ F1 (%)
ministral-3b
full
0.8711
–
no task
0.4992
-42.69
no ex
0.9077
4.20
none
0.2910
-66.59
ministral-8b
full
0.6942
–
no task
0.4175
-39.86
Appendix
Table 7: Macro F1 scores and percentage delta values for select models provided by Mistral
Model
Variant
Macro F1
Δ F1 (%)
qwen3-14b
full
0.9768
–
no task
0.9630
-1.41
no ex
0.9768
0.00
none
0.2870
-70.62
qwen3-235b-a22b
full
0.9768
–
no task
0.9674
-0.96
Appendix
Table 8: Macro F1 scores and percentage delta values for select models provided by Qwen
Figure 13: Distribution of macro F1 across prompt versions on gemma model variants. For each model panel, we plot max-min macro F1 (bars) per model and overlay per variant absolute macro F1 (points) on a twin top x-axis, with thin lines showing the min-max span. Models are alphabetized.
Figure 14: Distribution of macro-F1 across prompt versions on llama model variants . For each model panel, we plot max-min macro-F1 (bars) per model and overlay per-variant absolute macro-F1 (points) on a twin top x-axis, with thin lines showing the min–max span. Models are alphabetized.
Figure 15: Distribution of macro-F1 across prompt versions on mistral model variants. For each model panel, we plot max-min macro-F1 (bars) per model and overlay per-variant absolute macro-F1 (points) on a twin top x-axis, with thin lines showing the min–max span. Models are alphabetized.
Figure 16: Distribution of macro-F1 across prompt versions on qwen model variants. For each model panel, we plot max-min macro-F1 (bars) per model and overlay per-variant absolute macro-F1 (points) on a twin top x-axis, with thin lines showing the min–max span. Models are alphabetized.
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan
Université de Technologie de Compiègne, Compiègne, Hauts-de-France, France · Norwegian University of Science and Technology (NTNU), Trondheim, Trøndelag, Norway · SimulaMet, Oslo, Norway +1