ViAdminShift: A Vietnamese Benchmark for Event-Level Knowledge Updates Built from the 2025 Administrative Reform
Organizations: Faculty of Information Technology, University of Science, Vietnam National University, Ho Chi Minh City, Vietnam. · Faculty of Information Technology, Van Lang School of Technology, Van Lang University, Ho Chi Minh City, Vietnam.
Abstract
On July 1, 2025, Vietnam reduced its 63 provinces to 34, abolished its 696 districts, and replaced its 10,035 commune-level units (wards) with 3,321. ViAdminShift, a Vietnamese language resource about this event, consists of 7,995 benchmark items, 2,281 retrieval passages from pre- and post-reform Wikipedia revisions and dated templates, and old-to-new tables that a released script rebuilds from the official conversion table (not redistributed). Changed items store the pre-reform answer (except 121 address items), a design uses untouched units as controls, and 400 addresses appear without diacritics. The tables agree with an independent lineage on 10,564 of 10,565 official pairs, disagreements were settled against the resolutions, and the probing-track scorer agrees with both authors' blind labels (Cohen's and ). Data, outputs, code, and a datasheet are released. In the probing track, at least 20 of 67 models know the reform. Fourteen of them give the new province alone for at most 11% of plain questions about old addresses (old province only: 14-100%); a bare date recovers up to 100%. Post hoc (80 province-free addresses, single runs), with reasoning off, today's date in the system prompt leaves 68-100% old-province-only answers for 10 of 12 models; with reasoning, an "as of" prefix in the question leaves none for seven of eight. On the 1,262-item benchmark subset, with same-source old and new passages, small models unaware of the reform give 14-30% outdated answers and the five benchmark-track knowers (also differing in size or provider) 0-5%.
Figures & tables
| Resource | Lang. | Size | Source and construction | Ctrl. | Old | Present date |
| TimeQA | En | 20k per version | Crowd-verified Wikidata facts, template questions; year, month | No | No | None (time in question) |
| TempReason | En | 13,823 (test) | Wikidata 2022 and synthetic date arithmetic; month, year | No | Training only | None (time in question) |
| ChroKnowBench | En | 37,606 variant, 27,373 invariant | Wikidata, UMLS, US federal regulations (variant); CSKG, Math-KG (invariant); yearly | Yes | No | None (year in query) |
| EvolvingQA | En | 23,283 edited, 49,504 unchanged, 198,109 new | Monthly Wikipedia diffs (2023), QA written by GPT-3.5 | Yes | Yes | n/a |
| EvolveBench | En | 1,640 (132 subjects) | Wikidata histories, partly updated by hand; GPT-4o contexts; day | No | No | User prompt |
| evolveQA | En | 922 | AWS, Azure, and WHO update corpora; LLM-written questions | Yes | Yes | Hint before the cutoff |
| Type | Example (Vietnamese; English gloss ) | Gold / outdated answer | Items |
|---|---|---|---|
| T0 headline (MCQ, short) | Cả nước hiện có tổng cộng bao nhiêu tỉnh và thành phố trực thuộc trung ương? How many provinces and centrally run cities does the country have now? | 34 / 63 | 12 |
| T6 reverse (MCQ, short) | Địa bàn tỉnh Bình Dương thuộc đơn vị hành chính cấp tỉnh nào? Which provincial-level unit is the area of Bình Dương province in? | Thành phố Hồ Chí Minh / Tỉnh Bình Dương | 378 |
| T2 composition (MCQ) | Thành phố Hồ Chí Minh hiện nay được hình thành từ (những) đơn vị hành chính cấp tỉnh nào trước đây? Which former provincial units is Ho Chi Minh City (HCMC) now formed from? | HCMC, Bà Rịa–Vũng Tàu, and Bình Dương / HCMC only | 34 |
| T1 membership (MCQ) | Xã Hòa Hiệp, huyện Xuyên Mộc, tỉnh Bà Rịa - Vũng Tàu thuộc tỉnh/thành phố nào? Which province is Hòa Hiệp commune, Xuyên Mộc district, Bà Rịa–Vũng Tàu in? | Thành phố Hồ Chí Minh / Tỉnh Bà Rịa - Vũng Tàu | 2,781 |
| T1 new name (MCQ) | Phường Thuận Giao thuộc tỉnh/thành phố nào? Which province is (the new) Thuận Giao ward in? | Thành phố Hồ Chí Minh / Tỉnh Bình Dương | 720 |
| T3 existence (MCQ) | Hiện nay, tỉnh Bình Dương có phải là một đơn vị hành chính đang tồn tại không? Nếu có, đó là đơn vị hành chính cấp nào? Is Bình Dương province an existing administrative unit today? If so, of which level? | No / yes, provincial level | 308 |
| Route | Models | Reasoning setting |
|---|---|---|
| DeepSeek API (2) | DeepSeek-V4.1-Flash, DeepSeek-V4-Pro | thinking: {type: disabled} |
| Anthropic API, OpenAI-compatible endpoint (13) | Claude Haiku 4.5, Opus 4.5, Sonnet 4.6, Opus 4.6, Opus 4.7, Opus 4.8, Sonnet 5, Opus 5, Fable 5, Fable 5.1, Haiku 5.5, Sonnet 5.5, Opus 5.5; Sonnet 5.5 (thinking off via between_tools ), Haiku 5.5, and Fable 5.1 also run at a second setting | thinking disabled (measured: Opus 4.8 38%, Haiku 5.5 22%, Sonnet 5.5 15%, others 8%); reasoning_effort: none for Opus 5.5 (100%), Fable 5 (75%), and Fable 5.1 (82%) |
| OmniRoute reseller (20) | GPT-6, GPT-5.5, gpt-oss-120B, Kimi-K3, Kimi-K2.5, GLM-5.3, GLM-5.3-Flash, Gemini-3.1-Pro, Gemini-3.8-Flash, Gemma-4-31B , MiMo-v2.6-Flash, MiMo-v2.6-Pro, MiniMax-M2.7, MiniMax-M3, Muse-Spark-1.2 ∗ , Qwen3.6-27B, Qwen3.8-27B, Qwen3.8-Flash ∗ , Qwen3.8-Max, Qwen3.8-2.4T ∗ | thinking disabled in the request, yet reasoning on 79–100% of items for GPT-6, GPT-5.5, Kimi-K3, both GLM-5.3, Gemini-3.1-Pro, gpt-oss-120B, MiniMax-M2.7, Qwen3.6-27B, and the invalid runs ( ∗ ), and on at most 50% for the rest |
| OpenRouter, default provider routing (7) | GPT-6-Luna, GPT-5.6-Terra, Tencent HY4 (preview), Grok-4.7, Gemini-4-Argon, LongCat-2.0, Mistral-Large-4; reruns of GPT-6 and Kimi-K3, the benchmark run of Claude Haiku 5.5; not counted: Step-5 (preview), stopped for budget after 117 of the 549 base items (gate: 68 of 80 correct) | reasoning: {enabled: false} , or true for the reasoning-on runs; {effort: low} where reasoning cannot be disabled (Grok-4.7, Gemini-4-Argon) |
| Local, llama.cpp server on GPU or CPU (26) | Muse-Glimmer-30B; Qwen3-1.7B, 4B, and 8B; Qwen3.5-2B, 4B, and 9B; Qwen3.5-9B-Kimi-K3-Distilled; Qwen3.6-35B-A3B; Qwen3.8-27B; Qwen-SEA-LION-v4.5-27B; Nemotron-SEA-LION-v4.8-30B-A3B; NVIDIA-Nemotron-3.5-Lightning-30B-A3B ; SeaLLMs-v3-7B; MiMo-V2.6-Distill-Qwen-9B ; Gemma-3-4B; Gemma-4-E4B; Llama-3.2-3B; Ministral-3-8B; Granite-4.2-3B and 8B; LFM2.5-8B-A1B; Ling-3.0-tiny; MiniCPM5-2B ; gpt-oss-20B; tiny-aya-global (3.35B) | enable_thinking: false in the chat template (CPU runs also reasoning_effort: low ); Muse-Glimmer at reasoning_strength: none ; Muse-Glimmer, gpt-oss-20B, and LFM2.5-8B-A1B still reason on every cue-ladder item |
Factual Knowledge in Language Models
Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.
See all 213 papers