cs.CLOct 2, 2026

Writerslogic at PAN 2026: Process over Content for Robust Detection under Domain Shift

Authors: David L. Condrey

Organizations: WritersLogic Inc, San Diego, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We describe the Writerslogic systems for three PAN at CLEF 2026 shared tasks (Reasoning Trajectory Detection, Voight-Kampff Generative AI Detection, and Multi-Author Writing Style Analysis), unified by a shared analytical framework: feature robustness under distribution shift is governed by support overlap between training and test distributions, not by training-set effect size. This yields a taxonomy (domain-anchored, domain-portable, domain-invariant) that explains why generator-specific features die under domain shift while vocabulary fingerprints (hapax ratio, Yule's K, Heaps' exponent), compression measures, and character n-grams survive. On Reasoning Trajectory Detection, where training was entirely mathematics and 84 percent of test was unseen domains, the framework guided system design to 1st place in source detection (0.85 macro F1 via Opus-Sonnet agreement) and 3rd place in safety classification (0.66 macro F1 via query-refusal decomposition). For Voight-Kampff, we built a calibrated ensemble of DeBERTa-v2 (ONNX), multi-seed LightGBM with 44 domain-portable stylometric features, and SVM on n-gram TF-IDF, combined via learned stacking with isotonic calibration; the best configuration achieved 0.891 on the PAN 2026 test set with balanced sub-metrics (0.853 to 0.902 across all evaluation dimensions). For Multi-Author Writing Style Analysis, we describe a system fusing spectral clustering over character n-gram similarity graphs, normalized compression distance for local boundary detection, and SmolLM-135M perplexity for neural change-point detection; a platform mix-up meant our run never reached the official evaluation, so we report the design and its a priori predictions. Across all three tasks, features measuring generation process properties are designed to outperform features measuring generated content properties under domain shift.

Figures & tables

FeatureTrainTestddCategory
has_final_answer93.6%0.0%2.12Anchored
reasoning_tag99.3%0.0%1.89Anchored
opens_with_the70.0%∼\sim 5%1.54Anchored
has_boxed_answer99.0%29.3%0.91Portable
latex_density95%+16%0.78Portable
hapax_ratio100%100%−- .89Invariant
Table 1: Feature fire rates and effect sizes from Reasoning Trajectory Detection. Generator-specific features had strong effect sizes ( d>1.5d>1.5 ) but 0% test fire rate; vocabulary fingerprints have moderate effect sizes but 100% fire rate.
Figure 1: Subtask 1 (top): training was 100% mathematics; 84% of test is unseen domains. Subtask 2 (bottom): test traces average 424 characters across 15+ languages vs. 8,000–12,000 in training.
SystemTest F1Δ\Delta
LightGBM (30 features)0.690—
Multi-LLM (Llama + Qwen3)0.720+.030
Calibrated combination0.750+.030
Opus–Sonnet agreement0.850+.100
Table 2: Subtask 1 ablation. The largest gain (+0.10) came from Opus–Sonnet agreement.
ApproachValTestΔ\Delta
LightGBM (28 feat.).502.362−- .14
Refusal heuristic.624.350−- .27
TF-IDF + LR.793.500−- .29
XLM-RoBERTa.834.448−- .39
5-model LLM ensemble–.551–
Query-refusal decomp.–.640–
Table 3: The validation illusion on Subtask 2. Supervised methods show inverse val/test correlation.
 TeamF1Soft F1Rank (F1)Rank (Soft)
S1davidcondrey.85–1–
srikarkashyap.83–2–
skyduan.80–3–
affliction.75–4–
23020073.72–5–
S2ioanabrr.75.3111
Table 4: Leaderboard results (top 5 each subtask).
CategoryCountRepresentative Features
Document structure7word/char/sentence counts, paragraph count
Vocabulary richness8type-token ratio, hapax ratio, Yule’s K, Heaps’ exponent, MATTR Covington and McFall (2010) , Zipf coeff.
Sentence structure6avg. sent. length, sent. length CV, sent. start diversity, conjunction start ratio
Readability4Flesch-Kincaid, Coleman-Liau, avg. syllables
Compression/entropy4zlib ratio, char entropy, repetition ratio
Style markers9punct ratio, quote density, contraction ratio
Table 5: The 44 domain-portable features for Voight-Kampff, organized by category. All features produce meaningful values regardless of genre.
ConfigurationPAN-26 M1M2M3M4M5M6
large-rank0.8910.8910.8530.9020.8990.887
brown-velocity0.8800.5190.4750.4630.6780.603
frozen-author0.7080.6360.4930.4920.6990.606
lively-fixed0.6830.5280.5010.4600.6730.569
Table 6: Voight-Kampff results across key configurations. Metric 1 is the primary evaluation score. large-rank achieves the best cross-metric balance; brown-velocity peaks on Metric 1 but collapses on secondary metrics, exhibiting the threshold-collapse pattern predicted by the feature robustness framework.
More in this topic

Robustness of Machine-Generated Text Detection

Robustness of machine-generated text detection studies whether AI-text detectors remain reliable across shifts, edits, languages, and adversarial attacks.

See all 84 papers

Explore similar work

CardsList
Jul 4, 2026

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains

Recent AI-generated text detection work often introduces a new benchmark together with a specialized detector tailored to it. We revisit this practice from a baseline-first perspective. Across several benchmarks, we show that a plain, fully fine-tuned RoBERTa matches or exceeds the specialized detectors those benchmarks are built around. This suggests that much of the recent architectural complexity is not what drives strong in-distribution detection. The remaining challenge is the distribution shift. The same strong baseline degrades sharply when the topic domain or generating model changes at test time, and simply adding more source data does not close the gap. We identify a key failure mode: under distribution shift, the detector can assign high-confidence machine labels to human-written text from unseen domains. We then study two lightweight domain adaptation methods to address this problem: KK-shot adaptation with first-order MAML over LoRA adapters, and a per-sample confidence-weighted ensemble built on top of the adapted detector. Overall, our results suggest that progress in AI-generated text detection should be measured not only by in-distribution performance, but also by robustness under distribution shift.

Zhuoer Shen, Mingyi Wang, Shaofeng Zou +1

Department of Computer Science, University of California, Santa Barbara · School of ECEE, Arizona State University

Distribution Shift RobustnessAI-Generated Text DetectionDomain AdaptationDistribution Shift DetectionRobustness of Machine-Generated Text Detection
AskPDF
Aug 18, 2026

When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era

Authorship verification (AV) assumes that an author's writing style remains sufficiently stable to distinguish it from that of other writers. In practice, however, this assumption is challenged by distribution shifts caused by changes in genre, time, and AI-assisted writing. Existing AV benchmarks typically study these factors in isolation and focus predominantly on English, limiting our understanding of model robustness under realistic conditions. We introduce AVShift, the first German benchmark for systematically evaluating AV under multiple distribution shifts. AVShift comprises over 150,000 text pairs spanning three genres and 21 years, enabling controlled evaluation of cross-genre, temporal, and AI-era shifts within a unified framework. We benchmark representative feature-based, embedding-based, and LLM-based approaches. Our experiments show that fine-tuned LLMs generalize best across genres and benefit substantially from stylistically diverse training data. We further demonstrate that temporal drift is one of the strongest factors affecting AV, with performance degrading significantly as the time gap between documents increases. In contrast, we find no evidence of a measurable AI-era distribution shift within AVShift. Finally, our feature analysis reveals stylistic features that remain stable across genres, while their relative importance varies depending on the specific genre transition. We release AVShift and our code for future research.

Lotta Kiefer, Brisca Balthes, Christoph Leiter +3

University of Technology Nuremberg (UTN)

Distribution Shift RobustnessAuthorship AttributionTemporal Distribution Shift
AskPDF
May 5, 2026

Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators

AI-generated text is nowadays produced at scale across domains and heterogeneous generation pipelines, making robustness to distribution shift a central requirement for supervised binary detectors. We train transformer-based detectors on HC3 PLUS and calibrate a single decision threshold by maximising balanced accuracy on held-out validation; this threshold is then kept fixed for all downstream test distributions, revealing domain- and generator-dependent error asymmetries under shift. We evaluate in-domain on HC3 PLUS, under cross-dataset transfer to the multi-domain, multi-generator M4 benchmark, and on the external AI-Text-Detection-Pile. Although base models achieve near-ceiling in-domain performance (up to 99.5% balanced accuracy), performance under shift is brittle and strongly model-dependent. Feature augmentation via attention-based linguistic feature fusion improves transfer, with our best model (DeBERTa-v3-base+FeatAttn) achieving 85.9% balanced accuracy on M4. Multi-seed experiments confirm high stability. Under the same fixed-threshold protocol, our model outperforms strong zero-shot baselines by up to +7.22 points. Category-level ablations further show that readability and vocabulary features contribute most to robustness under shift. Overall, these results demonstrate that feature augmentation and a modern DeBERTa backbone significantly outperform earlier BERT/RoBERTa models, while the fixed-threshold protocol provides a more realistic and informative assessment of practical detector robustness.

Mohamed Mady, Johannes Reschke, Björn Schuller

Chair of Health Informatics, Technical University of Munich (TUM), Germany · OTH Regensburg, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, United Kingdom

AI-Generated Text DetectionTransformerCross-Domain GeneralizationRobustness of Machine-Generated Text Detection
AskPDF