This paper presents a novel agentic LLM pipeline for SemEval-2026 Task 10 that jointly extracts psycholinguistic conspiracy markers and detects conspiracy endorsement. Unlike traditional classifiers that conflate semantic reasoning with structural localization, our decoupled design isolates these challenges. For marker extraction, we propose Dynamic Discriminative Chain-of-Thought (DD-CoT) with deterministic anchoring to resolve semantic ambiguity and character-level brittleness. For conspiracy detection, an "Anti-Echo Chamber" architecture, consisting of an adversarial Parallel Council adjudicated by a Calibrated Judge, overcomes the "Reporter Trap," where models falsely penalize objective reporting. Achieving 0.24 Macro F1 (+100% over baseline) on S1 and 0.79 Macro F1 (+49%) on S2, with the S1 system ranking 3rd on the development leaderboard, our approach establishes a versatile paradigm for interpretable, psycholinguistically-grounded NLP.
Figures & tables
Figure 1: System architecture. S1 : DD-CoT Self-Refine extracts markers; a deterministic verifier anchors them to character offsets. S2 : Anti-Echo Chamber with contrastive retrieval, forensic profiling, a Parallel Council, and a calibrated Judge.
Task
Split
Baseline F1
Agentic
Δ
S1
Dev ( 100 )
0.12
0.24
+100%
Test ( 938 )
–
0.21
–
S2
Dev ( 100 )
0.53
0.79
+49%
Test ( 938 )
–
0.75
–
Table 1: Main results (macro F1). Baseline: zero-shot GPT-5.2. Document counts in parentheses.
Component / Change
Metric
Baseline
Agentic
Δ
Subtask 1:
Self-Refine (Audit loop)
Macro F1
0.173
0.240
+0.067
DD-CoT (Perpetrator vs. Victim Discrimination)
Actor F1
0.263
0.290
+0.027
Marker Extraction
Contextual Retrieval (Dynamic few-shot examples)
Macro F1
0.243
0.240
− 0.003
Subtask 2:
Parallel Council (Debate)
Recall
0.481
0.560
+0.079
Prosecutor (Evidence for Conspiracy)
F1 Score
0.680
0.795
+0.115
Conspiracy
Calibrated Judge (Final Decision Logic)
Macro F1
0.638
0.681
+0.043
Table 2: Ablation Summary (dev) to isolate the impact of core architectural nodes. DD-CoT significantly improves Actor identification by disambiguating agency (detecting perpetrators vs. victims). Parallel Council and Contrastive Retrieval combine to suppress the “Reporter Trap,” where topical discussion of conspiracy is misclassified as endorsement.
Configuration
Latency
Tokens
Calls
S1
Baseline (Generator only)
10.5s
2,549
1.0
Full Graph (Self-Refine)
30.2s
7,986
2.5
S2
Baseline (Single Agent)
4.6s
1,665
1.0
Full Graph (Council + Judge)
29.1s
12,627
5.0
Table 3: Per-document average latency and token usage profiled on 20 dev documents. All calls are LLM API invocations; baseline is a single zero-shot prompt.
Table 4: S1 Agent Ablation (Dev Set). Breakdown of contributions from the Critic and Refiner agents.
Figure 4: Juror Ablation Study (LOO). Removing any single persona significantly degrades performance, validating the necessity of the full four-juror council.
Metric
Standard CoT (Base)
DD-CoT (New)
Δ
Actor F1
26.3%
29.0%
+2.7 pts
Victim F1
23.3%
22.0%
− 1.3 pts
Appendix
Table 5: Impact of DD-CoT on Agency Disentanglement (dev). DD-CoT significantly improves Actor detection by resolving subject-position bias, with a minor redistribution of Victim scores.
Method
F1
Acc.
Deadlock
FP Conf.
Majority Vote
0.638
0.779
66.7%
0.890
Calibrated Judge
0.681
0.805
100.0%
0.865
Appendix
Table 6: Ablation: Judge vs. Majority Vote (original).
Configuration
Macro F1
Precision
Recall
No Retrieval
0.329
0.322
0.419
Standard Retrieval (Naive)
0.296
0.292
0.376
Stratified Retrieval (Ours)
0.311
0.313
0.392
Appendix
Table 7: Few-shot retrieval strategy comparison on dev set (S1) (original).
Configuration
False Positive Rate
Reduction
Standard Retrieval
0.160
–
Contrastive Retrieval
0.080
50%
Appendix
Table 8: S2 Retrieval Ablation: Impact of Contrastive Retrieval on False Positive Rate (original).
Figure 5: Number of marker types in the dataset.
Figure 6: Label distribution for conspiracy detection.
Figure 7: Frequency per marker type.
Figure 8: Number of marker annotations per document.
Figure 9: Marker type co-occurrences
Figure 10: Mean IoU of overlapping spans across marker type pairs. Higher values indicate greater boundary ambiguity between categories.
Figure 11: Marker type distribution across Subreddits.
Figure 12: Annotation contribution.
Figure 13: Span length distribution.
Figure 14: Marker span mass.
Figure 15: Normalized position of marker spans within documents (0 = start, 1 = end). KDE per marker type.
Parameter
Value
Description
Population size
20–30
Number of prompt candidates
Max generations
40–80
Budget (max_metric_calls)
Tournament size
3
Selection pressure
Mutation rate
0.2
Probability of mutating survivors
Crossover model
GPT-5.2
LLM for semantic merging
Reflector model
GPT-5.2
LLM for mutation feedback
Appendix
Table 9: GEPA hyperparameter configuration for S1 and S2 optimization.
Text Snippet
Baseline
Our System (Agentic)
“The public was manipulated by the media…”
Actor : The public
Actor : the media Victim : The public
“The article claims that the earth is flat.”
Endorsement
Neutral Reporting
Appendix
Table 10: Qualitative example demonstrating the resolution of agency and mitigation of the Reporter Trap by the agentic pipeline.
We present our systems for SemEval-2026 Task 10 (PsyCoMark), addressing conspiracy marker extraction (Subtask 1) and document-level conspiracy detection (Subtask 2). For marker extraction, we formulate the task as multi-label span classification over enumerated candidate spans, using IoU >= 0.95 positive labeling, hard-negative sampling, and containment-based non-maximum suppression (NMS) with boundary-aware span representations. Document classification is modeled independently using a sequence classifier with label smoothing and a stratified train-validation split. Analysis shows that entity-like roles (Actor, Victim) are detected robustly, while abstract roles (Action, Effect, Evidence) remain sensitive to boundary criteria. On the official test set, our systems rank 7th in Subtask 1 (0.2251 macro F1) and 11th in Subtask 2 (0.7694 weighted F1).
Dom Marhoefer, Milos Suvakovic, Glenn Grant-Richards +2
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking in the 85th percentile (8th out of 52 submissions). The results shown that our approach, which originated in machine-generated text detection, can be used for conspiracy detection as well.
Dominik Macko
Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia
As LLM-powered scams proliferate, black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints. Identifying the base model behind a chatbot endpoint (attribution), without model parameter access or knowledge of the hidden system prompt, would let investigators trace AI-enabled scams back to the providers whose models power them. Detecting when two endpoints run the exact same system prompt (fingerprinting), even one novel and unseen, would link individual scams into criminal networks and expose silent API changes. We conduct an empirical investigation of both capabilities. Our attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of non-adversarial conversation. Attribution of system prompts, while possible, requires retraining on a large amount of data for each prompt; system prompts in the wild are unbounded and ever-changing, making this approach costly. To tackle this more open-ended setting, our cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943. Conversational agents with unseen system prompts can thus be fingerprinted with robust accuracy from a few turns of ordinary conversation.
Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick