Identifying conceptually equivalent questions across different surveys is a challenging information retrieval task where surface-level textual similarity is insufficient. Two questions may be lexically similar yet assess entirely different sociological sub-concepts, or lexically different yet conceptually equivalent. We introduce a conditional information retrieval framework that addresses this by decomposing the task into two steps: concept extraction, which identifies the underlying sociological concept of a survey question using a fine-tuned generative model, and concept-aware retrieval, which augments question representations with the extracted concept as an explicit conditioning signal for finer-grained alignment. To reflect the realistic data scarcity conditions of specialised social science annotation, we use CLOSER as a case study. Our framework consistently outperforms baselines by 2-3 points in both MAP and Recall. We further analyse fine-tuning strategies including SFT, DPO, and dynamic DPO, finding that SFT delivers the most stable gains while dynamic DPO interacts differently with instruction-tuned versus distilled reasoning backbones. Finally, we demonstrate that a model fine-tuned on a general social science thesaurus (ELSST) transfers robustly to task-specific survey data, offering a practical solution under data scarcity.
Figures & tables
Figure 1: Two-step conditional retrieval framework for retrieving conceptually equivalent items. Given a survey question qi , (1) Concept Extraction uses an LLM to extract the underlying sociological concept t^i ; (2) Concept-Conditioned Encoding concatenates the extracted concept with the question as a conditioning signal before encoding; (3) Conceptual Comparison computes pairwise cosine similarity between query and candidate representations, returning items above threshold τ
Split
CLOSER-ID
CLOSER-OOD
Train
611
-
Test
405
3479
Table 1: Final count of instances per dataset. CLOSER-ID is deduplicated and split into train and test splits. CLOSER-OOD serves as the out-of-distribution test set only.
Table 2: Results on both CLOSER-ID and CLOSER-OOD. Highest metric within each backbone model group is highlighted in bold . The ELSST model uses Qwen2.5-7B-Instruct as the backbone model. Note that other than the baseline, as it is not applicable, all models use the conditioned representations.
Table 4: Ablation study results obtained using models trained with dynamic DPO training. Quantitative results (a) and qualitative examples (b) illustrate the impact of dynamic sampling on metrics and conceptual term granularity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Top-three retrieved items for the query “ About how often does your child do the following: How often does he: Help in the house 1|Nearly every day. ... 6|Not at all ” (ground-truth concept: Extra school activities: housework ). Bold entries show the model-predicted concept used for conditioning; italicised entries show the annotated gold concept of each retrieved item.
Table 6: Top-three retrieved items for the query ” He/She does not seem to feel sorry after he/she misbehaves. 1|Often true. 2|Sometimes true. 3|Not true “ (ground-truth concept: childdev ). Extracted conceptual terms are in bold and ground-truth conceptual terms are in italics .
Large Language Models (LLMs) offer a scalable way to simulate survey respondents using demographic profiles and observed reference responses. However, the conventional approach of predicting one question per prompt repeatedly encodes the same context, limits each target to a narrow set of reference responses, and prevents later predictions from using information in earlier answers. Predicting multiple questions in one prompt can reduce these costs, share a broader pool of references, and let later predictions build on earlier ones. This requires forming coherent batches, selecting shared references, and ordering questions and references effectively. We propose Semantically Coherent Batching and Ordering (SCBO), a training-free framework that addresses these challenges. SCBO first uses an LLM to extract compact semantic representations from survey items and filter out template noise. It then groups related questions into batches and builds a shared reference bank using target-specific retrieval and centroid-based completion. Finally, it orders target questions from easy to hard and arranges references according to their semantic alignment with those questions. Experiments on four large-scale survey datasets and four LLMs show that SCBO substantially reduces token consumption and inference time while generally improving prediction accuracy over a non-batched baseline. Code is available at https://anonymous.4open.science/r/SCBO-41D8.
Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real questionnaires, however, ask each respondent a sequence of related questions. A simulated respondent should show coherent preferences across the whole questionnaire, not merely accurate distributions for isolated items. Existing single-item methods cannot accurately reproduce how the same person answers a complete survey. We propose FullRespondent-LLM (FR-LLM), which fine-tunes two specialized LLMs: a marginal model for each item's response distribution and a respondent-level autoregressive model for dependencies across answers. Marginal-Constrained Joint Projection (MCJP) then projects the autoregressive joint distribution onto the set satisfying the item-level marginals learned by the first model. This yields complete questionnaires with realistic cross-item relationships while retaining strong item-level accuracy. On two real-world social survey datasets, FR-LLM more accurately reproduces multi-question response patterns, maintains competitive single-item accuracy, and generalizes better to unseen populations and questions. In a small commercial-survey dataset, we use simulated responses to make pricing and stocking decisions; FR-LLM achieves the highest realized profit.
Ji Huang, Mengfei Li, Shuai Shao
School of Computer Science, University of Science and Technology of China · School of Management, Fudan University
Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.
Chan-Tung Ku, Chan Hsu, Pei-Cing Huang +3
Department of Information Management National Sun Yat-sen University Kaohsiung, Taiwan · Institute of Political Science National Sun Yat-sen University Kaohsiung, Taiwan · Graduate Institute of Library and Information Science National Chung Hsing University Taichung, Taiwan