HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior

Jin Huang; Diego Ferreras Garrucho; Yutong Xie; Walter M. Yuan; Qiaozhu Mei; Chen Lian; Jonathon Hazell

cs.CLOct 6, 2026

HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior

Authors: Jin Huang, Diego Ferreras Garrucho, Yutong Xie, Walter M. Yuan, Qiaozhu Mei, Chen Lian, Jonathon Hazell

Organizations: University of Michigan, Ann Arbor · London School of Economics · MobLab Inc · University of California, Berkeley

Abstract

Large language models (LLMs) have the potential to meet a key goal in economics: a quantitative model of household decision making, across a variety of settings. Yet existing evaluations cover few surveys and outcomes, and do not study how households adjust to changing economic conditions. We introduce a new evaluation, HouseholdBench, which unites 6 U.S. household surveys and 32 prediction tasks spanning numeric, categorical and probabilistic outcomes, related to consumption, income, labor, expectations, and housing. Using past behavior, demographics and macroeconomic conditions, the tasks test whether LLMs predict behavior, including how households adjust to changes in various policies. We evaluate 13 proprietary and open-weight LLMs against a no-change baseline and a gradient-boosted tree model. Most LLMs outperform the no-change baseline, including for policy response tasks -- with the best model lowering error for numeric outcomes by 12.2%. Across most tasks, gradient-boosted trees rank first; leading proprietary LLMs approach their performance, but open-weight models lag. LLMs exhibit systematic over- and underprediction across different tasks. We identify methods that enable a 4 billion parameter open-weight model to match proprietary models' performance: fine-tuning and aggregating 16 predictions per observation. Improvements generalize to policy-response tasks, which are excluded from fine-tuning. We release our datasets, code, and leaderboard on our website: https://jn-huang.github.io/householdbench

Figures & tables

Panel of Figure 1 — Figure 1: HouseholdBench turns six household surveys into 32 prediction tasks across five topics.

Baseline tasks			Policy-response tasks
Task ID	Prediction target	Type	Task ID	Prediction target	Type
Consumption and saving
cons_cex_categories	Spending in 12 categories	N	cons_cex_rebate01	Spending after the 2001 tax rebates	N
cons_cex_total	Total, nondurable, and durable spending	N	cons_cex_sspay	Daily spending around Social Security payments	N
cons_psid_wealth	Net worth in the next wave	N	cons_cex_stimulus08	Spending after the 2008 stimulus payments	N
cons_sce_growth	Year-on-year growth in monthly spending	N	cons_psid_jobloss	Food spending the year after a job loss	N

Table 1: The 32 HouseholdBench tasks by topic. Baseline tasks condition on the household’s history and the macroeconomic environment. Policy-response tasks add a policy, shock, or scenario. Type: N numeric, C categorical, P probabilistic. Appendix H describes every task in full.

Study	Data	Years	Outcomes predicted					Design
			Spending	Income	Labor	Macro	Housing	Policy
Athey et al. (2026)	US labor panels (PSID, NLSY)	1979–2021			$\checkmark$
Cruz et al. (2024)	American Community Survey (ACS)	2018		$\checkmark$	$\checkmark$		$\checkmark$
Brynjolfsson et al. (2025)	PSID, UK valuation surveys	2017–2021		$\checkmark$
Jia et al. (2026)	Dutch household panel (LISS)	2023–2024			$\checkmark$		$\checkmark$
Wu et al. (2025)	US consumer surveys (SCE, Nielsen)	2018–2023				$\checkmark$		$\checkmark$

Table 2: Comparison between HouseholdBench and selected studies.

		Average rank within topic						Average score by target type
Rank	Model	Consumption	Housing	Income	Labor	Macroeconomic	Average rank	RelMAE $\downarrow$	Macro-F1 $\uparrow$	TV $\downarrow$
		(# = 9)	(# = 5)	(# = 5)	(# = 10)	(# = 3)		(# = 18)	(# = 9)	(# = 5)
1	XGBoost	2.3 (0.6)	5.8 (1.0)	2.6 (0.8)	4.4 (0.7)	2.0 (0.8)	3.4 (0.4)	0.811 (0.011)	0.432 (0.014)	0.141 (0.004)
[1pt/2pt] 2	Fable 5.1	6.0 (0.5)	4.4 (0.8)	2.4 (0.9)	4.2 (0.6)	2.0 (0.9)	3.8 (0.3)	0.878 (0.013)	0.413 (0.013)	0.139 (0.005)
3	GPT-5.6 sol	5.3 (0.5)	5.2 (0.7)	6.2 (1.0)	4.8 (0.5)	6.0 (0.7)	5.5 (0.3)	0.910 (0.013)	0.399 (0.011)	0.139 (0.005)
4	GPT-6 astra	4.4 (0.4)	8.6 (0.7)	4.6 (0.7)	6.4 (0.4)	5.0 (0.8)	5.8 (0.3)	0.915 (0.014)	0.369 (0.006)	0.139 (0.005)

Table 3: Average rank by topic and average score by target type, pre-cutoff test split. Best LLM in bold, second best underlined. Ranks are taken over the models shown. Brackets report bootstrap SEs.

Figure 2: Model performance relative to the no-change benchmark on baseline and policy-response tasks. Solid bars report mean scores for baseline tasks, while hatched bars report mean scores for policy-response tasks. Whiskers are pointwise 95% basic IID bootstrap intervals.

Figure 3: LLMs exhibit systematic bias in predicting household behavior. Left. LLMs consistently overestimate household behavior on some tasks and underestimate it on others. Right. LLMs overpredict the majority class on categorical tasks. Bold means that a model predicts the majority class more often than the ground truth. (B) and (P) denote baseline and policy-response tasks.

			Baseline tasks (seen in fine-tuning)			Policy-response tasks (held out)
	Model	$K$	RelMAE $\downarrow$	Macro-F1 $\uparrow$	TV $\downarrow$	RelMAE $\downarrow$	Macro-F1 $\uparrow$	TV $\downarrow$
			(# = 8)	(# = 6)	(# = 4)	(# = 10)	(# = 3)	(# = 1)
Reference	XGBoost	1	0.84	0.46	0.15	0.79	0.36	0.12
[1pt/2pt]	GPT-6 astra	1	0.89	0.39	0.15	0.94	0.32	0.11
	Claude Opus 5	1	0.90	0.47	0.15	0.91	0.28	0.13
	Fable 5.1	1	0.89	0.44	0.15	0.86	0.35	0.11

Table 4: Results of SFT. The

K

column is the number of aggregated predictions. Best among the language models in bold, second best underlined. XGBoost is refitted on each task it is scored on, so its policy-response scores are not held out.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure D1: Difference in mean total variation by target, separately for unchanged, changed and all target probability vectors. Bars show

\mathrm{TV}(\mathrm{model})-\mathrm{TV}(\mathrm{no\mbox{-}change})

on the same valid observations, without division. Negative values favor the model; zero denotes equal performance. Each row reports one target; multi-target tasks are not pooled. The figure shows twelve prediction models using pre-cutoff valid responses; the no-change row is omitted. Facet titles count evaluation rows for that target before excluding invalid model responses. Unchanged and Changed partition those rows. Green denotes a negative difference, red a positive difference and grey a tie. Black bars are 95% basic paired IID bootstrap intervals for the difference, resampling rows within task and omitting draws without a valid subgroup observation. Each target uses one scale across its three panels; scales differ across targets. Continued on the next page.

Figure D1: Difference in mean total variation by target (continued). The model set, sample definitions and interval construction are the same as on the first page. Continued on the next page.

Figure D1: Difference in mean total variation by target (continued). Solid bars denote baseline tasks and hatched bars denote the policy task. Each model and its no-change comparator use the same valid observations in both the point estimate and every bootstrap draw.

Figure D2: Mean performance before and after the cutoff. Panel (a) compares matched pre-cutoff and post-cutoff task samples for the main-leaderboard models evaluated in both samples, excluding the no-change baseline. Panel (b) compares pre-cutoff performance with performance on the 454 rows across nine tasks released on or after 1 July 2026, the first release wave after Claude Fable 5.1’s June 2026 knowledge cutoff; its pre-cutoff sample covers the same nine tasks. Models follow the main-leaderboard order. Solid bars report equal-task means of the raw task metric in each panel’s pre-cutoff sample; hatched bars report the corresponding later-sample means. Whiskers are pointwise 95% basic IID bootstrap intervals. The RelMAE and total-variation axes are reversed because lower values indicate better performance.

Figure D3: Model performance over outcome-release time for the main-leaderboard models. The legend follows the main-leaderboard order. Vertical bars are pointwise 95% basic IID bootstrap intervals.

			Baseline tasks (seen in fine-tuning)			Policy-response tasks (held out)
	Model	$K$	RelMAE $\downarrow$	Macro-F1 $\uparrow$	TV $\downarrow$	RelMAE $\downarrow$	Macro-F1 $\uparrow$	TV $\downarrow$
			(# = 8)	(# = 6)	(# = 4)	(# = 10)	(# = 3)	(# = 1)
Reference	XGBoost	1	0.84	0.46	0.15	0.79	0.36	0.12
[1pt/2pt]	GPT-6 astra	1	0.89	0.39	0.15	0.94	0.32	0.11
	Claude Opus 5	1	0.90	0.47	0.15	0.91	0.28	0.13
	Fable 5.1	1	0.89	0.44	0.15	0.86	0.35	0.11

Appendix

Table E1: Results of SFT and SFT with the number-token loss. The

K

Numeric , RelMAE $\downarrow$ (# = 18)
	Model	$K=1$	$K=2$	Improvement of $K=2$ from $K=1$ (%)	$K=16$	Improvement of $K=16$ from $K=1$ (%)
Proprietary	Fable 5.1	0.878	0.879	$-$ 0.2	–	–
	Claude Opus 5	0.905	0.902	0.3	–	–
	GPT-5.6 sol	0.910	0.910	0.0	–	–
[1pt/2pt] Open-weight	Qwen3.5-4B	1.304	1.272	2.5	1.223	6.3
	+ SFT	1.164	1.087	6.6	1.017	12.6

Appendix

Table F1: Aggregating predictions of one model, pre-cutoff test split.

K=1

is the score of one draw, and

K=2

and

K=16

are the scores of the aggregate of two and sixteen draws. Each improvement is the change from

K=1

as a percentage of the

K=1

score, positive where the metric gets better.

Hyperparameter	Value
LoRA rank	8
LoRA alpha	32
LoRA dropout	0.05
LoRA target modules	all linear layers of the language model
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , weight decay $0.1$ )
Peak learning rate	$5\times 10^{-5}$

Appendix

Table G1: Fine-tuning hyperparameters, shared by Qwen3.5-4B and Qwen3.6-27B and by both training losses.

Explore similar work

Ideological Bias in LLMs' Economic Causal Reasoning
Apr 23, 2026Donggyu Lee, Hyeok Yun, Jungwon Kim +4Large Language Model Bias Ideology
Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea +1
Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers
Sep 4, 2026Yumiao Li, Peixin Liu, Donglin Di +2Large Language Model Policy Optimization Agent Behavior Modeling
School of Economics and Management, Tsinghua University, Beijing, China · Actuarial Science and Risk Management, University of Illinois Urbana-Champaign, Urbana, IL, USA · School of Computing, Harbin Institute of Technology, Harbin, China
Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management
Sep 30, 2025Runze Zhang, Xiaowei Zhang, Mingyang ZhaoUser Simulation Large Language Model Evaluation
Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

Apr 23, 2026, cs.AIJ/K move · Enter open · S save

Ideological Bias in LLMs' Economic Causal Reasoning

Donggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min+3

Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea+1

Large Language Model Bias Ideology

Open paper AskPDF arXiv

Do large language models (LLMs) exhibit systematic ideological bias when reasoning about economic causal effects? As LLMs are increasingly used in policy analysis and economic reporting, where directionally correct causal judgments are essential, this question has direct practical stakes. We present a systematic evaluation by extending the EconCausal benchmark with ideology-contested cases - instances where intervention-oriented (pro-government) and market-oriented (pro-market) perspectives predict divergent causal signs. From 10,490 causal triplets (treatment-outcome pairs with empirically verified effect directions) derived from top-tier economics and finance journals, we identify 1,056 ideology-contested instances and evaluate 20 state-of-the-art LLMs on their ability to predict empirically supported causal directions. We find that ideology-contested items are consistently harder than non-contested ones, and that across 18 of 20 models, accuracy is systematically higher when the empirically verified causal sign aligns with intervention-oriented expectations than with market-oriented ones. Moreover, when models err, their incorrect predictions disproportionately lean intervention-oriented, and this directional skew is not eliminated by one-shot in-context prompting. These results highlight that LLMs are not only less accurate on ideologically contested economic questions, but systematically less reliable in one ideological direction than the other, underscoring the need for direction-aware evaluation in high-stakes economic and policy settings.

Abstract

Explore similar work

Ideological Bias in LLMs' Economic Causal Reasoning

Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers

Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management