cs.CLApr 15, 2026
SaveRhetorical Questions in LLM Representations: A Linear Probing Study
Organizations: Independent Researcher · University of Cincinnati · Amazon
Abstract
Rhetorical questions are asked not to seek information but to persuade or signal stance. How large language models internally represent them remains unclear. We analyze rhetorical questions in LLM representations using linear probes on two social-media datasets with different discourse contexts, and find that rhetorical signals emerge early and are most stably captured by last-token representations. Rhetorical questions are linearly separable from information-seeking questions within datasets, and remain detectable under cross-dataset transfer, reaching AUROC around 0.7-0.8. However, we demonstrate that transferability does not simply imply a shared representation. Probes trained on different datasets produce different rankings when applied to the same target corpus, with overlap among the top-ranked instances often below 0.2. Qualitative analysis shows that these divergences correspond to distinct rhetorical phenomena: some probes capture discourse-level rhetorical stance embedded in extended argumentation, while others emphasize localized, syntax-driven interrogative acts. Together, these findings suggest that rhetorical questions in LLM representations are encoded by multiple linear directions emphasizing different cues, rather than a single shared direction.
Figures & tables
Figure 1: AUROC across layers and representations. Test AUROC across normalized layers using PCA-reduced representations. Rows compare mean-pooled and last-token embeddings; columns vary by dataset (RQ, SRAQ) and model (Qwen3-32B, Llama-3.3-70B).
Figure 2: Alignment and ordering agreement between linear probes. Top: Cosine similarity between probing directions across layers. Middle: Spearman rank correlation between probe-induced scores, with shaded regions indicating confidence intervals. Bottom: Jaccard overlap between the top and bottom 20% of examples ranked by each probe. Results are shown across datasets (RQ, SRAQ), models (Qwen3-32B, Llama-3.3-70B-Instruct), and layers.
Figure 3: Transferability of rhetorical probing directions across datasets. For RQ panels (left), directions are learned on SRAQ and applied to RQ; for SRAQ panels (right), directions are learned on RQ and applied to SRAQ. Rows report test AUROC of transferred directions (top), Spearman rank correlation between rankings induced by transferred directions and rankings induced by directions learned on the target dataset (middle), and Jaccard overlap between the top and bottom 20% of examples under these two rankings (bottom).
Figure 4: Cross-dataset alignment of rhetorical probing directions. Cosine similarity between probing directions learned on RQ and SRAQ, measured layer-wise for two models and three different probings.
| SRAQ rank | RQ rank | Paragraph | Gold Label |
| 1 | 15 | I find our infatuation with ourselves to be a bit self serving. Who but us cares? It’s in our own self interest to think we’re great. But what does it accomplish? Nothing. …We think we’re so great because we can manipulate things. But more often than not this manipulation simply creates major problems …“We can think of magnificent things; just look at the incredible architecture we’ve created.” So? Spiders spin incredible webs. Birds weave beautiful nests. Bugs create towers that they live in. Complexity does not necessarily mean superiority. | rhetorical |
| 2 | 108 | …What anti-copyright fanfic enthusiasts like yourself don’t realize is that you’re just trying to have your cake and eat it too. …You can do that already with public domain material or with your own distilled characters and settings. But you don’t want to. Why not? Because you REALLY WANT the popularity of these modern franchises to hold you up and make your story make sense. You’re not asking for the ideas, you already have them. You’re asking for the popularity and depth of concept; and nobody promised you that. | rhetorical |
| 3 | 240 | …If you scrutinise the majority of Reykjavik sized “hellhole” cities, you will find that they are often an extension of a greater metropolitan area. For example, Watford is the same size as Reykjavik and Exeter is the same size as Reykjavik. Exeter has a lower crime rate than Watford — why? Because Watford is essentially a city within the greater metropolitan area of London. | rhetorical |
| 230 | 1 | My guess is you’ll have to go to or phone their office to apply …In most states the initial court appearance is considered a “critical stage of the process” where you are entitled to Counsel …If this is your first offense most places have a pre-trial diversion program …Talk to your lawyer …Whatever happens this will still likely be expensive. A “social thing.” What? you only drink to make other people more interesting? :) | rhetorical |
| 146 | 2 | …so why would they even try and be granted legal visitation when they know the issue will be looked at in court? Are they trying to somehow go around this issue? And if so, how? …They just closed the suit with his first wife with 3 kids and that one lasted since 1994–2017 …It’s not for custody, they are asking for visitation rights, which is ridiculous. | informational |
| 237 | 3 | Wait, what? Not a single higher spot in Estonia, but a few bumps in Latvia and Lithuania. Do your research, we have huge “mountains” compared to them =) | rhetorical |
Table 1: SRAQ instances ranked by inner product with the SRAQ-derived diffMean direction (top) and RQ-derived (bottom) at layer 48 of Qwen3-32B. Each row reports both ranks. Bold text marks the target question; ellipses indicate truncated text for readability. 5 5 5 Some top-ranked instances are omitted here because they contain socially sensitive content; the complete rankings are reproducible from the released code.
| Subset | SRAQ direction | RQ direction |
| Top 1% | 188.8 | 126.4 |
| Top 3% | 150.9 | 116.4 |
Table 2: Mean token length of top-ranked SRAQ examples selected by each diffMean direction under the ranking setup used in Section 7. Token counts are computed using the Llama-3.3-70B-Instruct tokenizer.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: AUROC differences (PCA minus no PCA) for diffMean linear probes under mean-token and last-token pooling.
Figure 6: Per-layer explained-variance ratio of the 64th principal component (i.e., the marginal variance explained by the 64th principal component) for mean-token vs. last-token representations.
Figure 7: Subspace alignment between RQ and SRAQ across layers for multiple models. We report cosine similarity and geodesic distance between the corresponding layer-wise subspaces, using normalized layer index on the -axis. Higher distances indicate weaker cross-dataset alignment.
Figure 8: Test AUROC of training-free diffMean directions across layers under alternative pooling strategies. Results are shown for RQ ( question_with_context ) and SRAQ ( paragraph ) using Qwen3-32B and Llama-3.3-70B-Instruct. We compare mean pooling over all tokens, last-token pooling, pooling over the last 5 or 10 tokens, and mean pooling over question tokens only.
Figure 9: Test AUROC of training-free diffMean directions across normalized layers for ModernBERT-large using the [CLS] representation, shown for RQ and SRAQ.
Figure 10: Prompt template used for question generation in the steering experiments.
| Parameter | Value |
| max_new_tokens | 50 |
| do_sample | True |
| temperature | 0.7 |
| top_p | 0.9 |
| repetition_penalty | 1.1 |
Table 3: Generation parameters used in the steering experiments.
Figure 11: Alpha sweep results on Qwen3-32B. Mean rhetorical score (y-axis) as a function of the steering coefficient (x-axis) for different steering layers, evaluated separately on rhetorical and informational contexts.
Figure 12: Prompt used to score generated questions on a rhetorical–informational scale.
Figure 13: Additional last-token probe results. Both panels report within-setting performance (training and evaluation within the same dataset, model, and input formulation) as a function of layer. (a) Results for additional LLMs beyond those highlighted in the main figures. (b) Results for the full_turn input formulation. Across settings, trends are consistent with the main text: rhetorical question intent remains linearly separable across layers, and last-token representations provide stable signals.
Figure 14: Within-dataset alignment across additional LLMs. Comparison of probe alignment metrics computed within the same dataset for models not shown in the main text. Although probes achieve similar discriminative performance, their learned directions exhibit limited alignment, indicating that multiple, non-collinear directions can support linear separability even within a fixed dataset. Trends are consistent with the main results.
Figure 15: Cross-dataset alignment across additional LLMs. Comparison of probe alignment metrics when probe directions learned on one dataset are applied to another, for models not shown in the main text. Despite partial transferability in discriminative performance, alignment between probe directions remains limited, reflecting divergence in the induced rankings across datasets. These results are consistent with the trends reported in the main text.
More in this topic
Pragmatic Reasoning in Language Models
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
See all 103 papers