Cohort comfort models (CCMs) aim to overcome the extensive individual data requirements of personal comfort models (PCMs) by grouping occupants based on demographic and psychometric characteristics (cold start) or similar thermal preferences (warm start). We evaluated the validity of the reported CCM performance gains through (i) a methodological assessment of the original study, (ii) a replication using one of the two original datasets (DORN: 20 participants, 6 months) under a revised experimental protocol, and (iii) a replication using an independently collected dataset (PersonAI: 72 participants, 4 weeks). Classification performance was evaluated using F1-micro, as in the original study, and F1-macro to account for class imbalance. Under F1-micro, a trivial baseline that always outputs the majority class performed on par with cohort models. Under F1-macro, all cohort models achieved lower median scores than a simple baseline that makes random predictions based on the estimated distribution of thermal preferences. In our methodological assessment we identified the following key issues: Limited data per participant and per minority class, reliance on a metric insensitive to class imbalance, missing trivial baselines, inadequate cross validation strategy, and multiple occurrences of information leakage. Our findings challenge the original study's conclusion that CCMs improve thermal preference prediction. CCMs did not outperform simple class-distribution baselines. Future studies should improve experimental designs, including controlled thermal conditions, to obtain sufficient observations across thermal preference classes.
Figures & tables
Figure 1: Overview of the study methodology.
Figure 2: Pearson correlation matrix between numeric environmental/biological features and the thermal comfort target.
target_num
BFI2SOpen−Mindedness
0.23
BFI2XSOpen−Mindedness
0.21
BFI2SAestheticSensitivity
0.18
BFI2SIntellectualCuriosity
0.18
BFI2SCreativeImagination
0.16
BFI2SEnergyLevel
0.16
Table 1: Pearson correlation (r) between aggregated one-time onboarding survey scores and the thermal comfort target.
Figure 3: Per-user distribution of F1-micro (blue) and F1-macro (orange) scores for the PCM, PCM Majority Baseline, PCM ApproxDist Baseline, best warm start cohort model, best cold start cohort model, cohort baseline, and the cohort majority baseline. The dashed red line signifies the Random 3-class F1 score.
Figure 4: F1-micro performance comparison across all cohort approaches for the DORN (top) and PersonAI (bottom) datasets. Tilde-prefixed entries denote the ablation condition in which test users are purposely assigned to an incorrect cluster. The red band marks the range between the General Purpose and PCM medians. Cold and warm experiments are shown in blue and orange, respectively.
Figure 5: F1-macro performance comparison across all cohort approaches for the DORN (top) and PersonAI (bottom) datasets. Tilde-prefixed entries denote the ablation condition in which test users are purposely assigned to an incorrect cluster. The red band marks the range between the General Purpose and PCM medians.
Figure 6: Class distribution of thermal preference labels for the DORN and PersonAI datasets. Both datasets are dominated by the “no change” class.
Figure 7: Class distribution of thermal preference labels per participant.
Figure 9
Figure 9: Per-user percentage change in F1-micro from the general-purpose model to each cohort/PCM approach, for DORN and PersonAI.
Figure 10: Learning curves for the participants with the most available data for PCM models in each dataset