Cohort comfort models (CCMs) aim to overcome the extensive individual data requirements of personal comfort models (PCMs) by grouping occupants based on demographic and psychometric characteristics (cold start) or similar thermal preferences (warm start). We evaluated the validity of the reported CCM performance gains through (i) a methodological assessment of the original study, (ii) a replication using one of the two original datasets (DORN: 20 participants, 6 months) under a revised experimental protocol, and (iii) a replication using an independently collected dataset (PersonAI: 72 participants, 4 weeks). Classification performance was evaluated using F1-micro, as in the original study, and F1-macro to account for class imbalance. Under F1-micro, a trivial baseline that always outputs the majority class performed on par with cohort models. Under F1-macro, all cohort models achieved lower median scores than a simple baseline that makes random predictions based on the estimated distribution of thermal preferences. In our methodological assessment we identified the following key issues: Limited data per participant and per minority class, reliance on a metric insensitive to class imbalance, missing trivial baselines, inadequate cross validation strategy, and multiple occurrences of information leakage. Our findings challenge the original study's conclusion that CCMs improve thermal preference prediction. CCMs did not outperform simple class-distribution baselines. Future studies should improve experimental designs, including controlled thermal conditions, to obtain sufficient observations across thermal preference classes.
Figures & tables
Figure 1: Overview of the study methodology.
Figure 2: Pearson correlation matrix between numeric environmental/biological features and the thermal comfort target.
target_num
BFI2SOpen−Mindedness
0.23
BFI2XSOpen−Mindedness
0.21
BFI2SAestheticSensitivity
0.18
BFI2SIntellectualCuriosity
0.18
BFI2SCreativeImagination
0.16
BFI2SEnergyLevel
0.16
Table 1: Pearson correlation (r) between aggregated one-time onboarding survey scores and the thermal comfort target.
Figure 3: Per-user distribution of F1-micro (blue) and F1-macro (orange) scores for the PCM, PCM Majority Baseline, PCM ApproxDist Baseline, best warm start cohort model, best cold start cohort model, cohort baseline, and the cohort majority baseline. The dashed red line signifies the Random 3-class F1 score.
Figure 4: F1-micro performance comparison across all cohort approaches for the DORN (top) and PersonAI (bottom) datasets. Tilde-prefixed entries denote the ablation condition in which test users are purposely assigned to an incorrect cluster. The red band marks the range between the General Purpose and PCM medians. Cold and warm experiments are shown in blue and orange, respectively.
Figure 5: F1-macro performance comparison across all cohort approaches for the DORN (top) and PersonAI (bottom) datasets. Tilde-prefixed entries denote the ablation condition in which test users are purposely assigned to an incorrect cluster. The red band marks the range between the General Purpose and PCM medians.
Figure 6: Class distribution of thermal preference labels for the DORN and PersonAI datasets. Both datasets are dominated by the “no change” class.
Figure 7: Class distribution of thermal preference labels per participant.
Figure 9
Figure 9: Per-user percentage change in F1-micro from the general-purpose model to each cohort/PCM approach, for DORN and PersonAI.
Figure 10: Learning curves for the participants with the most available data for PCM models in each dataset
In many applications, human and LLM evaluators use assessments of relevant criteria to create an overall evaluation for an item or individual. For example, in admissions, committees assess candidates on attributes such as test scores, GPA, and research experience to evaluate their overall fit for the program. Another example arises in medical care where clinicians use patient reports of symptoms to consider preliminary diagnoses and assess risks. Each setting involves mapping multiple criteria to an overall evaluation -- a process that reflects the evaluator's underlying preferences. We focus on the fundamental question of learning these preferences. Many applications of this problem make specific modeling assumptions on evaluator preferences that may be substantially violated in the real world. We make the minimal assumption that the preference function is coordinate-wise non-decreasing, which is reasonable in a large number of evaluation settings. We theoretically characterize the severity of model mismatch for many common assumptions and show that it can lead to significant issues for learning evaluator preferences and other important downstream tasks. We then present an algorithm for learning evaluators' preferences that is robust to model mismatch. We prove theoretically that our algorithm can learn any preference function without sacrificing performance when the linearity assumption holds. Evaluations of our algorithm with synthetic simulations and real-world data confirm its ability to learn preferences robustly and illustrate key aspects of LLM and human preferences.
Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, all models considered in our analysis exhibit self-coherence, with the most probable outcomes stabilizing rapidly under repeated sampling. At the same time, we observe substantial discordance across both model families and scales, with little consensus even among their most probable outcomes. These patterns hold across nine open-weight models, three choice domains, and show robustness under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that outcomes are influenced more by the choice of model than by the wording of the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.
Fan Huang, Minsuk Kim, C. Tyler Diggans +1
Indiana University Bloomington Bloomington, IN, USA · Massachusetts Institute of Technology Cambridge, MA, USA · Root Dynamix LLC Sewickley, PA, USA
LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.