BANGLAPLAN: A Benchmark for Constraint-Based Planning and Decision-Making in Bangla

Md Rayhanul Masud; S M Maksudul Alam; Mohammad Jaminur Islam

cs.CLOct 14, 2026

BANGLAPLAN: A Benchmark for Constraint-Based Planning and Decision-Making in Bangla

Authors: Md Rayhanul Masud, S M Maksudul Alam, Mohammad Jaminur Islam

Organizations: University of California, Riverside

Abstract

Large language models (LLMs) increasingly help users arrange appointments, plan journeys, and allocate limited resources. These tasks require models to satisfy constraints on time, cost, capacity, and order, or determine that no feasible solution exists. Bangla is the world's seventh most widely spoken language, and existing benchmarks cover many language and reasoning tasks. However, benchmarks for evaluating planning and decision-making in Bangla under explicit constraints remain lacking. We introduce BANGLAPLAN, a benchmark of 1,000 problems across ten task families grounded in everyday life and institutions in Bangladesh. It includes feasible and infeasible problems in four language registers: everyday, formal, colloquial, and Bangla--English mixed. Answers cover eight structured types, including schedules, routes, allocations, and ordered steps, in a common JSON format. We construct the benchmark through language-model-assisted drafting and independent human validation. We evaluate 11 open-weight and proprietary model configurations using exact match, separately reporting feasible-case accuracy, infeasibility detection, and format errors. We find a large performance gap: GPT-6.1 Sol achieves 98.7% accuracy and Qwen3.5-27B with thinking reaches 80.3%, while direct-answer configurations score at most 30.9%. Seven of eleven configurations perform below the always-``infeasible'' baseline. Our findings also suggest that harder problems primarily reduce plan-construction accuracy, while infeasibility detection remains more robust. BANGLAPLAN provides a reproducible basis for evaluating planning and decision-making in Bangla. BANGLAPLAN is available at: https://github.com/mrayhanulmasud/banglaplan.

Figures & tables

Figure 1: A BanglaPLAN problem and its near-boundary infeasible variant. Left: the Bangla request with an English translation. A student needs 10 minutes to print her admit card at a shop that opens at 8:00, followed by 25 minutes of travel to the exam centre. Right: a gate-closing time of 8:45 permits the gold plan; changing it to 8:30 makes the request infeasible, with gold answer is_feasible: false .

Family	Representative problem types	Family	Representative problem types
F01 Eligibility	scholarships, exams, leave, mobile-money rules	F06 Sequencing	recipes, construction, crop calendars
F02 Procedures	NID correction, passports, land mutation	F07 Allocation	beds, relief rice, irrigation, Qurbani meat
F03 Appointments	clinics, biometrics, banks, legal aid	F08 Procurement	haat shopping, Iftar supplies, tenders
F04 Journeys	Eid travel, ferries, hospital referrals	F09 Logistics	ports, ferry berths, waste collection
F05 Rosters	exams, invigilation, garment shifts	F10 Emergency	evacuation, triage, ambulance dispatch

Table 1: The ten task families in BanglaPLAN , each comprising ten problem types with ten instances per type.

Panel of Figure 2 — Figure 2: Construction of BanglaPLAN . Authors define the taxonomy. Annotators select and rewrite Claude-suggested scenarios in Bangla, creating feasible and near-boundary infeasible problems with unique canonical gold answers. An independent annotator solves each problem without seeing the gold answer, followed by a second check using Claude Opus 5.5. Humans resolve disagreements and revise flagged issues (red).

Frontier models (reasoning effort: low)
	All 1,000 instances			744 feasible instances				256 infeasible instances
Model	EM $\uparrow$	Incorrect	Format	Plan $\uparrow$	Wrong plan	Absten- tion	Format	Flag $\uparrow$	False plan	Format
Always “infeasible”	25.6	74.4	0.0	0.0	0.0	100.0	0.0	100.0	0.0	0.0
GPT-6 Luna	85.1	14.9	0.0	82.7	14.8	2.6	0.0	92.2	7.8	0.0
GPT-6.1 Sol	98.7	1.3	0.0	98.4	0.8	0.8	0.0	99.6	0.4	0.0
Qwen3.5, four sizes (thinking off unless marked)

Table 2: Zero-shot results on 1,000 BanglaPLAN instances (all values in %). Each feasible or infeasible block sums to 100, up to rounding. EM : exact-match accuracy over all instances. For feasible instances, Plan denotes a correct plan, Wrong plan an incorrect plan, and Abstention an incorrect “infeasible” answer. For infeasible instances, Flag denotes a correct “infeasible” answer and False plan a proposed plan. Format denotes failure to return the required JSON array. Bold and underline mark the best and second-best EM, Plan, and Flag scores. Gray marks EM below the always-“infeasible” baseline of 25.6.

Figure 3: Exact-match score decomposed into correct plans (blue) and “correct infeasible” answers (orange). The dashed line shows “always-infeasible” baseline.

Figure 4: Planning accuracy drops with difficulty, while infeasibility detection remains stable. Left: correct plans on feasible instances. Right: correct “infeasible” answers on infeasible instances.

	EM $\uparrow$		Plan $\uparrow$		Flag $\uparrow$
Model	bn	en	bn	en	bn	en
GPT-6.1 Sol	98.7	97.4	98.4	96.6	99.6	99.6
Qwen3.5-2B	3.3	5.8	4.2	6.6	0.8	3.5
Qwen3.5-4B	14.1	22.4	8.3	8.6	30.9	62.5
Qwen3.5-9B	19.6	24.3	12.8	15.2	39.5	50.8

Table 3: Bangla prompts vs. their English translations (same instances, same scoring) English improves Qwen EM but yields limited gains in plan accuracy; GPT-6.1 Sol performs better in Bangla.

		Feasible		Infeasible
	EM $\uparrow$	Plan $\uparrow$	Abstention	Flag $\uparrow$	Format
Always “infeasible”	25.3	0.0	100.0	100.0	0.0
Qwen3.5-2B, untrained	2.4	3.2	0.0	0.0	18.7
+ SFT (gold answers)	27.1	4.0	89.5	95.2	0.0

Table 4: Fine-tuning Qwen3.5-2B on gold answers (%), scored on the 166 held-out instances, 42 of them infeasible, so always answering “infeasible” scores 25.3.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: The evaluation prompt: the instance’s gold_prompt , sent verbatim.

Model	Instance	Kind	Rule in the prompt	What the model’s plan does	Verdict
Qwen3.5-2B	F04-T038-001	route	Fare at most 2,100 BDT; patient needs oxygen, which the microbus lacks; arrival by 02:00	Takes ambulance and microbus for 3,700 BDT; arrives at 02:05	V
Qwen3.5-2B	F03-T025-007	schedule	Plan for 9–11 December; 60 minutes between villages; no overlaps	Dates every action 2026-11-11, copied from the format example; allows no travel time	V
Qwen3.5-2B	F02-T015-005	ordered steps	Hospital slip before application; verification at least 24 h later, on Sun/Tue	Applies before obtaining the slip; verifies immediately on a Monday	V
Qwen3.5-2B	F01-T002-008	classification	International track requires English score $\geq 6.5$ ; applicant has 6.0	Selects the international track	V
Qwen3.5-2B	F08-T078-007	selection	One bidder per lot; bidder 1 fails the copier test; bidder 3 is blacklisted; capacity is 40 pieces	Selects all three bidders, including ineligible copiers, and buys 75 desk sets and 24 cabinets	V
Qwen3.5-2B	F10-T097-002	matching	Exactly one unit per slot; list each slot once	Assigns two engines to the same slot and lists it twice	V

Appendix

Table 5: Ten audited Wrong plan cases. V: hard-constraint violation; S: feasible but worse on the objective; N: optimal but fails a tie-break. Instance IDs omit the BP- prefix. The audit sheets include the relevant rules in Bangla.

	$n$	GPT-6 Luna	GPT-6.1 Sol	Qwen3.5-2B	Qwen3.5-4B	Qwen3.5-9B	Qwen3.5-27B	Qwen3.5-27B (think)	Gemma 3 12B	TigerLLM 12B	Gemma 2 9B	Llama 3.1 8B
F01 Eligibility	100	98.0	97.0	21.0	34.0	46.0	64.0	99.0	43.0	45.0	40.0	34.0
F02 Procedures	100	88.0	100.0	0.0	7.0	10.0	24.0	77.0	0.0	0.0	2.0	2.0
F03 Appointments	100	78.0	97.0	2.0	15.0	19.0	28.0	68.0	0.0	5.0	6.0	4.0
F04 Journeys	100	94.0	100.0	1.0	19.0	27.0	30.0	88.0	0.0	9.0	2.0	4.0
F05 Rosters	100	80.0	100.0	0.0	17.0	19.0	30.0	77.0	4.0	9.0	7.0	4.0
F06 Sequencing	100	81.0	99.0	0.0	10.0	13.0	23.0	68.0	0.0	2.0	4.0	0.0

Appendix

Table 6: Exact match (%) by task family (bn). Bold = best, underline = second best per row. Procurement (F08), sequencing (F06), and appointments (F03) are hardest, requiring models to select items within a budget, respect dependencies, or coordinate timed bookings.

	$n$	GPT-6 Luna	GPT-6.1 Sol	Qwen3.5-2B	Qwen3.5-4B	Qwen3.5-9B	Qwen3.5-27B	Qwen3.5-27B (think)	Gemma 3 12B	TigerLLM 12B	Gemma 2 9B	Llama 3.1 8B
allocation	160	88.8	99.4	1.2	10.0	13.8	28.7	84.4	8.1	10.0	10.6	1.2
classification	100	98.0	97.0	21.0	34.0	46.0	64.0	99.0	43.0	45.0	40.0	34.0
matching	60	95.0	100.0	5.0	25.0	26.7	43.3	93.3	30.0	30.0	28.3	20.0
ordered steps	140	88.6	100.0	0.0	6.4	8.6	23.6	74.3	0.0	0.0	1.4	1.4
ranking	40	80.0	95.0	10.0	22.5	32.5	40.0	72.5	27.5	20.0	12.5	10.0
route	100	94.0	100.0	1.0	19.0	27.0	30.0	88.0	0.0	9.0	2.0	4.0

Appendix

Table 7: Exact match (%) by answer kind (bn). Bold = best, underline = second best per row. GPT-6.1 Sol leads on seven of eight answer kinds, while all other models achieve their highest exact match on classification.

	$n$	GPT-6 Luna	GPT-6.1 Sol	Qwen3.5-2B	Qwen3.5-4B	Qwen3.5-9B	Qwen3.5-27B	Qwen3.5-27B (think)	Gemma 3 12B	TigerLLM 12B	Gemma 2 9B	Llama 3.1 8B
code-mixed	200	82.5	99.0	4.0	17.0	21.0	29.0	79.0	9.0	12.0	11.0	8.5
colloquial	100	87.0	100.0	1.0	18.0	25.0	29.0	81.0	7.0	8.0	9.0	6.0
everyday	500	85.0	98.8	4.0	12.4	18.8	31.0	80.8	9.8	12.4	9.8	6.2
formal	200	87.0	97.5	2.0	13.5	17.5	33.5	80.0	5.5	8.5	8.5	6.5

Appendix

Table 8: Exact match (%) by language register (bn). Bold = best, underline = second best per row. GPT-6.1 Sol leads on every register. Among the three strongest models, GPT-6 Luna varies most, scoring lowest on code-mixed prompts (82.5%)

LLM Planning

Large language model planning studies how models decompose goals, choose actions, and revise plans under constraints.

See all 137 papers

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models
May 20, 2026Ziliang Zhao, Zenan Xu, Shuting Wang +7LLM Evaluation LLM Planning
Gaoling School of Artificial Intelligence, Renmin University of China · LLM Department, Hunyuan Team, Tencent · Beijing Academy of Artificial Intelligence +1
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
Jun 4, 2026Jiayu Liu, Cheng Qian, Zhenhailong Wang +10LLM Planning LLM Agent Evaluation
University of Illinois Urbana-Champaign
Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation
Aug 25, 2026Ismail Hossain, Nafi Ullah Shafin, Mohammad Abdullah Al MuminLLM Quantization Post-Training Quantization
Institute of Information and Communication Technology (IICT) Shahjalal University of Science and Technology (SUST), Sylhet, Bangladesh

May 20, 2026J/K move · Enter open · S save

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

Ziliang Zhao, Zenan Xu, Shuting Wang, Hongjin Qian+6

Gaoling School of Artificial Intelligence, Renmin University of China · LLM Department, Hunyuan Team, Tencent · Beijing Academy of Artificial Intelligence+1

LLM Evaluation LLM Planning Synthetic Benchmark Generation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Planning is a fundamental capability for large language models (LLMs) because such complex tasks require models to coordinate goals, constraints, resources, and long-term consequences into executable and verifiable solutions. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets. This limits scenario coverage, ties difficulty to surface-level proxies rather than structural sources, and offers limited support for scalable generation, automatic verification, or planning-oriented training. We introduce PlanningBench, a framework for generating scalable, diverse, and verifiable planning data for both evaluation and training. PlanningBench starts from real planning scenarios and abstracts practical workflows into a structured taxonomy of more than 30 task types, subtasks, constraint families, and difficulty factors. Guided by this taxonomy, a constraint-driven synthesis pipeline instantiates self-contained planning problems with adaptive difficulty control, quality filtering, and instance-level verification checklists. This shifts planning data construction from fixed benchmark collection to controllable generation while preserving realistic task grounding. We use PlanningBench to evaluate open-source and closed-source frontier LLMs, and find that current models still struggle to produce complete solutions under coupled constraints. Beyond evaluation, reinforcement learning on verified PlanningBench data improves performance on unseen planning benchmarks and broader instruction-following tasks. Further analysis suggests that determinate or well-specified optimal solutions provide clearer reward signals and more stable training dynamics. Overall, PlanningBench provides a controllable source of planning data for diagnosing and improving generalizable planning abilities in LLMs.

Abstract

LLM Planning

Explore similar work

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation