The Neighbour's Label: Opposite Subgroup Effects of Same-Dialogue Exposure in Next-Act Evaluation
Organizations: Centre of Real Time Computer Systems, Kaunas University of Technology, Kaunas, Lithuania · Institute of Computational Linguistics, University of Zurich, Zurich, Switzerland
Abstract
Window-based next-act evaluation often aims to measure generalisation across dialogues, yet overlapping windows and similarity retrieval can expose labelled examples from the test dialogue. The nearest such example is the target's labelled predecessor, whose act is correct when the target continues it and wrong when the target changes act; a model that copies it can gain and lose in two subgroups that one aggregate averages. We induce this exposure under controlled designs on SGD, STAR and AMI, whose official splits are clean. Randomly including the predecessor's row in training raises a lexical classifier's continuation scores and lowers its transition scores in all three corpora, and purging the adjacent rows removes most of the gain. With TF-IDF-retrieved demonstrations, two 7B instruction-tuned models show the same opposite pattern in all eight model-corpus-sample arms (some single-cell intervals include zero; one replication is partial), while natural-weight accuracy changes by only -0.05 to +0.04. Exposure frequency depends on the retrieval pipeline. We evaluate two diagnostic comparisons without model retraining and find that excluding the test dialogue does not detectably reduce accuracy on SGD. All exposure is induced; no published system was re-evaluated.
Figures & tables
| Control: estimand | SGD (LRAP) | STAR (nats) | AMI (nats) | What it shows |
| Placebo row: lag-1 minus placebo effect, cont. ∗ | [ , ] | [ , ] | [ , ] | Adjacency, not dialogue membership (cells: Fig. 3) |
| Purge of the adjacent minus as many matched rows, cont. ∗ | [ , ] | [ , ] | [ , ] | The continuation gain is concentrated in the two adjacent rows |
| Labels permuted within dialogue: lag-1, cont. | [ , ] | [ , ] | [ , ] | Needs the label; STAR keeps about a quarter |
| Lead-1 row: effect on transitions | [ , ] | [ , ] | [ , ] | Predicted gain not observed |
| Fine-tuned encoder: lag-1, cont. | inconcl. | inconcl. | inconcl. | No detectable effect, not evidence of none (Fig. 3; Table 5) |
| TF-IDF -NN (exploratory): lag-1, cont. | [ , ] | [ , ] | [ , ] | Largest SGD effect; added after primary results |
| Interaction (nats) | Prespecified | |||
| Arm | Sample | est. | 95% CI | decision |
| SGD | 1st | [ , ] | partial | |
| Qwen | fresh | [ , ] | partial | |
| pooled | [ , ] | – | ||
| SGD | 1st | [ , ] | present | |
| Mistral | fresh | [ , ] | replicated | |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Channel | Example in this paper | Setting A: unseen dialogues | Setting B: prospective, observed history |
|---|---|---|---|
| Later windows of the test dialogue (may contain the target utterance) | pool allowing later windows | leakage | leakage (the future is not observed) |
| Earlier windows of the test dialogue with their gold acts | exposed pool; row-level split | leakage | admissible only for acts genuinely observed at run time |
| Earlier windows of the test dialogue without labels | unlabelled-window control | not label leakage; transductive use of test-dialogue text, admissible only if the benchmark permits it | admissible history |
| Near-duplicate dialogues across the split | SGD and STAR near-duplicate groups | leakage unless clustered before splitting | ordinary training data |
| Same participants, another meeting | AMI series-level pool | definition-dependent (no measurable effect here) | admissible |
| Pre-training contamination | both language models postdate the corpora | shifts absolute levels; cannot create a within-query pool contrast | same |
| SGD (LRAP) | STAR (nats) | AMI (nats) | |||||||
| Estimand (TF-IDF classifier) | est. | lo | hi | est. | lo | hi | est. | lo | hi |
| Lag-1 present, continuation | |||||||||
| Lag-1 present, transition | |||||||||
| Lag-1, continuation minus transition | |||||||||
| Placebo row present, continuation | |||||||||
| Lag-1 minus placebo, continuation | |||||||||
| Estimand | SGD (LRAP) | STAR (nats) | AMI (nats) |
|---|---|---|---|
| Fine-tuned RoBERTa | |||
| Lag-1 present, continuation | [ , ] | [ , ] | [ , ] |
| Lag-1 present, transition | [ , ] | [ , ] | [ , ] |
| Lag-1 minus placebo, continuation | [ , ] | [ , ] | [ , ] |
| Any same-dialogue exposure, continuation | [ , ] | [ , ] | [ , ] |
| Any same-dialogue exposure, transition | [ , ] | [ , ] | [ , ] |
| Continuation (nats) | Transition (nats) | Interaction (nats) | Aggregate (top-1, natural) | ||||||||||
| Arm | Sample | est. | lo | hi | est. | lo | hi | est. | lo | hi | est. | lo | hi |
| SGD Qwen | 1st | ||||||||||||
| fresh | |||||||||||||
| pooled | |||||||||||||
| SGD Mistral | 1st | ||||||||||||
| fresh | |||||||||||||
| Arm | Decision | Compared | CI covers 0 | Same sign | Half-width, med./max |
|---|---|---|---|---|---|
| SGD / Qwen | partial | 23 | 23 | 19 | 0.73 / 1.09 |
| SGD / Mistral | replicated | 19 | 19 | 19 | 0.56 / 0.84 |
| AMI / Qwen | replicated | 25 | 25 | 23 | 0.78 / 1.18 |
| AMI / Mistral | replicated | 19 | 19 | 18 | 0.55 / 1.04 |
| Contrast (continuations, nats) | Sample | SGD / Qwen | SGD / Mistral | AMI / Qwen | AMI / Mistral |
|---|---|---|---|---|---|
| Previous act’s label vs. different label, other-dialogue window | 1st | [ , ] | [ , ] | [ , ] | [ , ] |
| fresh | [ , ] | [ , ] | [ , ] | [ , ] | |
| Previous act’s label on other-dialogue window vs. filtered | 1st | [ , ] | [ , ] | [ , ] | [ , ] |
| fresh | [ , ] | [ , ] | [ , ] | [ , ] | |
| Same-dialogue window vs. label-matched other window | 1st | [ , ] | [ , ] | [ , ] | [ , ] |
| fresh | [ , ] | [ , ] | [ , ] | [ , ] |
| Arm | Slope | |||||
|---|---|---|---|---|---|---|
| SGD / Qwen | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| SGD / Mistral | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| AMI / Qwen | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| AMI / Mistral | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| Contrast (continuations, nats) | SGD / Qwen | SGD / Mistral | AMI / Qwen | AMI / Mistral |
|---|---|---|---|---|
| Unlabelled same-dialogue vs. unlabelled other window | [ , ] | [ , ] | [ , ] | [ , ] |
| Unlabelled same-dialogue window vs. none | [ , ] | [ , ] | [ , ] | [ , ] |
| Unlabelled other window vs. none | [ , ] | [ , ] | [ , ] | [ , ] |
| Gold label vs. no label, same text | [ , ] | [ , ] | [ , ] | [ , ] |
| Wrong label vs. no label, same text | [ , ] | [ , ] | [ , ] | [ , ] |
| Labelled same-dialogue window vs. none | [ , ] | [ , ] | [ , ] | [ , ] |
| Contrast (nats) | SGD cont. | SGD trans. | AMI cont. | AMI trans. |
|---|---|---|---|---|
| Shared part vs. matched other text | [ , ] | [ , ] | [ , ] | [ , ] |
| Older part vs. matched other text | [ , ] | [ , ] | [ , ] | [ , ] |
| Whole window vs. empty block | [ , ] | [ , ] | [ , ] | [ , ] |
| Shared part vs. empty block | [ , ] | [ , ] | [ , ] | [ , ] |
| Older part vs. empty block | [ , ] | [ , ] | [ , ] | [ , ] |
| Whole window vs. shared part | [ , ] | [ , ] | [ , ] | [ , ] |
| Copy rate | Exposed filtered | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Arm | Sample | Zero-shot | Random | Filtered | Exposed | Label-matched | Wrong label | est. | lo | hi |
| SGD Qwen | 1st | 0.264 | 0.269 | 0.100 | 0.113 | 0.127 | – | |||
| fresh | 0.251 | 0.287 | 0.129 | 0.129 | 0.149 | 0.140 | ||||
| SGD Mistral | 1st | 0.180 | 0.198 | 0.102 | 0.127 | 0.133 | – | |||
| fresh | 0.164 | 0.227 | 0.113 | 0.158 | 0.140 | 0.182 | ||||
| AMI Qwen | 1st | 0.151 | 0.091 | 0.102 | 0.336 | 0.129 | – | |||
| TF-IDF pipeline | Dense pipeline | Dense TF-IDF | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Sample | Share of queries | est. | lo | hi | est. | lo | hi | est. | lo | hi |
| SGD, 1st | Lag-1 in top 4 | |||||||||
| Lag-1 nearest | ||||||||||
| Any same-dialogue | ||||||||||
| Top 4, continuations | ||||||||||
| Top 4, transitions | ||||||||||
| Prediction, control or reading | Outcome | Evidence |
|---|---|---|
| Lead-1 presence helps transitions | not observed in 3 of 3 corpora | SGD [ , ]; STAR [ , ]; AMI [ , ] |
| Masking the target text is a neutral control | not neutral in 3 of 3 | masked minus placebo-masked gain; Appendix C |
| Fine-tuned encoder shows an adjacency effect | inconclusive in 3 of 3 | Table 5 |
| SGD continuation gain, Qwen, first sample | missed; decision partial | [ , ] |
| SGD fresh replication, Qwen | partial: transition loss missed | [ , ] |
| Label agreement of the retrieved neighbour predicts the transition sign | unstable (1 of 4 fresh arms) | no claim |
Data Leakage
Data leakage is unintended information flow into model training, evaluation, or outputs that inflates results or exposes sensitive data.
See all 136 papers