CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing

Zewen Jiang; Yahan Li; Chenlu Guo; Diana Inkpen; Yuan Wu; Yi Chang

cs.CLOct 14, 2026

CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing

Authors: Zewen Jiang, Yahan Li, Chenlu Guo, Diana Inkpen, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University, China · College of Software, Jilin University, China · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Bilibili Inc., China ·

Abstract

Large language models (LLMs) have demonstrated multilingual capabilities across diverse language generation and understanding tasks. Yet effective code-switching requires more than multilingual proficiency: models must decide when, where, and why to switch languages in ways that fit the surrounding context. This context-sensitive ability remains insufficiently evaluated. We introduce CSWBench, a benchmark for evaluating context-sensitive code-switching in creative writing through two complementary settings: Focused Narrative and Extended Narrative. We further propose CSWEval, a theory-informed framework with two complementary levels: event-level evaluation of Licensing, Contextual Appropriateness, and Functional Gain, and whole-text evaluation of task-specific quality---overall literary and narrative quality in Focused Narrative and quality emphasizing multilingual organization in Extended Narrative. Evaluating twelve LLMs, we find that models exhibit meaningful but uneven code-switching competence across narrative contexts. Model differences are particularly pronounced in multilingual literary integration, while multilingual-focused dimensions show stronger difficulty effects overall than the general narrative dimensions; narratively salient switches are associated with stronger contextual and functional realization; and event-level CS quality is moderately associated with whole-text performance.

Figures & tables

Benchmark	Task	Granularity	Generation	CS Choice	CS Orch.	Salience
LinCE Aguilar et al. (2020)	CS NLP	Token / Utterance	$\times$	$\times$	$\times$	$\times$
CodeMixBench Yang and Chai (2025)	CS Understanding	Token / Text	$\triangle$	$\times$	$\times$	$\times$
ChiEngMixBench Yang et al. (2026)	CS Generation	Sentence Pair / Text	$\checkmark$	$\checkmark$	$\times$	$\times$
LitBench Fein et al. (2026)	Creative Writing	Story Pair	$\checkmark$	$\times$	$\times$	$\times$
WritingBench Wu et al. (2025b)	Writing	Whole Text	$\checkmark$	$\times$	$\times$	$\times$
CSWBench	Creative Writing	Event + Whole Text	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison of CSWBench with representative code-switching and writing benchmarks. Task denotes the evaluation setting; Granularity lists representative assessment units. Generation denotes evaluation of generated text; CS Choice indicates whether switching decisions are evaluated against contextual conditions; CS Orch. indicates whether multilingual behavior is assessed across a text; and Salience indicates whether the narrative importance of individual switches is considered.

\checkmark

denotes explicit coverage,

\triangle

partial coverage, and

\times

an aspect not evaluated. NLP = natural language processing; CS = code-switching; Orch. = orchestration.

Figure 1: Illustrative distinction between genuine code-switching and surface multilingual material. The examples distinguish meaningful language switches from English expressions that do not constitute code-switching (CS). A genuine switch involves English use attributable to current speaker or discourse voice as part of an ongoing interaction or narrative. By contrast, proper names, conventionalized borrowings, and quoted or displayed English without active uptake are excluded. The distinction depends on how English participates in discourse, rather than on the mere presence of English words.

Figure 2: Overview of the CSWBench construction and CSWEval evaluation pipeline. Source records are screened and transformed into Seed-conditioned prompts; CSWEval defines event-level and whole-text assessment; and twelve LLMs generate Focused and Extended narratives for evaluation. Event-level dimensions include Licensing (D1), Contextual Appropriateness (D2), and Functional Gain (D3). The illustrated whole-text dimensions are for Extended Narrative; text fragments and scores are schematic.

Models	Focused Narrative					Extended Narrative
Models	D1	D2	D3	Whole	$\Sigma$	D1	D2	D3	Whole	$\Sigma$
Proprietary Models
GPT-5.6-Sol	4.998	3.349	3.348	3.806	15.501	4.881	4.102	4.032	4.752	17.767
GPT-5.6-Terra	4.997	3.333	3.340	3.740	15.410	4.769	4.069	4.020	4.494	17.352
GPT-5.5	4.999	3.357	3.362	3.819	15.537	4.781	4.007	3.946	4.614	17.348
Gemini-3.5-Flash	4.995	3.323	3.299	3.276	14.893	4.894	3.871	3.674	4.032	16.471

Table 2: Main Results of CSWBench. Dual-judge mean scores for twelve LLMs on Focused and Extended Narrative. D1, D2, and D3 represent Licensing, Contextual Appropriateness, and Functional Gain, respectively; Whole denotes task-specific whole-text quality. Each dimension is scored out of 5, and

\Sigma

is their sum (/20). Bold and underlined values indicate the highest and second-highest scores in each column, respectively.

Panel of Figure 3 — Figure 3: Narrative salience and position-swap sensitivity. Top panels show mean Contextual Appropriateness (D2) and Functional Gain (D3) scores by salience level (S1–S3, increasing narrative consequence), with Q1–Q3 ranges and overlaid medians. Bottom panels show whole-text score changes as increasing numbers of code-switching event pairs or Chinese-span pairs are swapped, comparing sensitivity to the two types of disruption.

Group	Original CS stronger	Tie	Monolingual counterfactual stronger	Directional score
HIGH	56.9%	20.8%	22.2%	+0.347
LOW	29.9%	48.6%	21.5%	+0.083

Table 3: Functional Gain counterfactual judgments. The 72 events are divided into HIGH and LOW groups by original D3 scores. Original code-switching realizations are compared with human-reviewed Chinese-only alternatives through cross-judge comparisons in both presentation orders. Percentages report pooled outcomes, while directional scores assign

+1

to original-CS wins,

0

to ties, and

-1

to monolingual wins.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Total	ASCEND	SwitchLingua
Source records sampled	3,500	382	3,118
Retained source records	3,009	287	2,722
Validated Seeds retained for benchmark construction	596	109	487
Final evaluation prompts	807	146	661
Focused	407	76	331
Extended	400	70	330

Appendix

Table 4: Construction and benchmark-selection statistics.

Coverage item	Focused ( $n=407$ )	Extended ( $n=400$ )	Cross-task
Scene domains	13/14	14/14	Largest gap 5.0 pp
Participant relations	6/9	8/9	Largest gap 4.8 pp
Interaction patterns	11/11	11/11	All retained
Preferred styles	5/5	5/5	All baseline styles retained
Unique source groups	350	340	88 / 93 prompts repeat a group
Stable scene/relation/style	12 / 5 / 5	14 / 8 / 5	$\geq$ two source groups

Appendix

Table 5: Descriptive coverage of the Focused and Extended prompt sets, including the largest cross-task prevalence gap for each category. A source group is an ASCEND conversation or a SwitchLingua record to which a prompt’s Seed is bound; a category is stable when covered by at least two distinct source groups.

Figure 4: Scene-domain distribution of the two prompt sets over ten reporting categories, which aggregate the 14 recorded scene domains, for prompts derived from ASCEND and SwitchLingua Lovenia et al. (2022) ; Xie et al. (2025) . The ten categories are consumer services and mobility, culture and identity, digital and technical contexts, education, health and medical, leisure and sport, other or underspecified, private social life, public information and institutions, and work and professional life.

Figure 5: Coverage of the recorded category universes. Each bar pair gives the number of categories retained by the Focused and Extended sets against the recorded universe size for that coverage item.

Candidate concept families	Preliminary construct	Count	Final role
Repertoire, access, represented resources	Participant–world resource grounding	9	D1 Licensing
Voice, recipient, sequence, situation and local fit	Situated interactional coherence	22	D2 Contextual Appropriateness
Contrastive meaning, stance, relationship, style and experience	Functional multilingual contribution	19	D3 Functional Gain
Recurrence, contrast, development and literary integration	Whole-text multilingual orchestration	4	Extended Whole-text; Focused uses general literary/narrative quality
Textual warrant, mediation and inferential boundaries	Evidentially constrained literary mediation	9	Cross-cutting evidence constraint, not a fifth quality score
Eventfulness, consequence and narrative importance	Narrative event importance	7	Salience S1–S3 for D2/D3 weighting, not quality

Appendix

Table 6: Condensed concept-to-construct mapping. Counts sum to 70 primary assignments.

Item	1	3	4	5
D2.1 Voice attribution	Wrong or indeterminate voice	Dialogue, thought, or narration clearly attributed	Distinguishes potentially confusable layered voices or quotation responsibility	Attribution remains precise across layered voices or quotation boundaries
D2.2 Carrier fit	Expression incompatible with its carrier	Natural fit to speech, thought, narration, message, or reuse	Carrier and expression are specifically matched in form and use	Wording and form are precisely matched to the carrier and its constraints
D2.3 Addressee trigger	Known addressee conditions conflict with the choice	Recipient’s ability, identity, or response motivates it	Precisely responds to recipient’s resources, wording, identity, or immediate uptake	The realization precisely fits the recipient’s resources, wording, identity, and immediate interactional state
D2.4 Topic trigger	Unrelated to current concept, object, or material	Topic directly supports English	Conceptual boundaries, source wording, or reference shape the local interpretation	Wording and placement are tightly grounded in the current concept, source wording, referent, or topic structure
D2.5 Emotion trigger	Applicable emotional mechanism cannot be located	Switch accompanies emotional change	Occurs at a specific emotional threshold, defensive move, or intimacy shift	Wording and placement precisely align with the established emotional state or transition
D2.6 Power, stance, identity	Applicable negotiation mechanism cannot be located	Clearly marks stance or identity	Changes the interaction at an assertion, concession, boundary-setting, alliance, or repositioning move	Wording and form precisely fit the current stance, power, or identity configuration

Appendix

Table 7: Focused D2 rubric. NA is allowed for D2.3–D2.7; D2.1, D2.2, and D2.8 always require numeric scores. For D2.7, unchanged adjacent register is NA, whereas a required but unrealized transition is scored.

Item	1	3	4	5
D3.1 Semantic precision	No added distinction, source reference, ambiguity control, or precision	Recognizable terminological/conceptual precision	Specific meaning, original wording, ambiguity, or wordplay affects understanding	The semantic contribution is exceptionally precise and tightly integrated into the event’s local meaning
D3.2 Relationship/stance gain	Language choice adds no relational or attitudinal signal	Clear relational or attitudinal signal carried by the language choice	Substantially strengthens a relational act or subtext	Carries a highly specific and strongly realized relational or attitudinal function within the interaction
D3.3 Identity/cultural specificity	Merely labels an identity	Adds a plausible, specific identity connection	Closely connects experience and current self-positioning	Makes an exceptionally specific and well-supported contribution to identity or cultural positioning
D3.4 Stylistic tension	Only typographic novelty or display	Clear local voice, rhythm, double-voicing, or restraint	Works with syntax, rhythm, imagery, or subtext to organize the passage	Is tightly integrated with local form to produce a highly realized stylistic effect
D3.5 Experiential texture	Merely labels a background	Connects to lived experience and adds texture	Interacts closely with action, medium, institution, body, or memory	Makes the represented experience exceptionally specific, credible, and texturally realized within the event

Appendix

Table 8: Focused D3 rubric. NA applies to D3.1, D3.2, D3.3, and D3.5 when the corresponding function is not applicable; D3.4 always requires a numeric score.

Dimension	1	3	4	5
L1 Completeness/coherence	Main line, causality, or ending not intelligible	Complete story with adequate causality and ending	Opening pressure, change, decisions, and ending form a sustained causal chain	Several indispensable nodes form a causal system
L2 Voice, relationships, characterization	Indistinct characters, unsupported motives, or announced change	Distinct stances, credible motives, relationship advanced through interaction	Voice, action, and subtext differentiate characters over stages	Layered motives, avoidance, behavior, and decisions form a complex coherent system
L3 Style, rhythm, imagery	Reading obstructed or basic unity absent	Fluent narration with local rhythmic/imagistic organization	A formal mechanism persists across locations	Voice, rhythm, imagery, and perspective form a distinctive system
L4 Emotional/thematic depth	Implausible emotion or abstract conclusions replace narration	Plot, action, and dialogue support emotion/theme	Two mutually constraining values develop persistently	Layered conflicts revise each other and sustain interpretive tension
L5 Originality/nonformulaic realization	Predominantly generic template	Some effective choices within a familiar whole	Mutually supporting non-obvious choices on at least two levels	Distinctive unified narrative principle constrained by earlier development

Appendix

Table 9: Focused whole-text rubric. Score 2 denotes partial success between anchors 1 and 3; all five dimensions require numeric scores, with no NA/IE.

ID	Dimension	Evidence
M-F1	Multilingual narrative-system stability	Voice ownership; speech/thought/quotation boundaries; carriers; focalization; preservation/translation policy; language facts and rules maintained or coherently developed across scenes.
M-F2A	Character A bilingual profile and language arc	Repertoire, function/relationship/affect mappings, scene responsiveness, motivated transitions and non-switching; the same multilingual person developing on textual evidence.
M-F2B	Character B bilingual profile and language arc	The same checks independently for B, without requiring symmetry or rewarding event count alone.
M-F3.1	Structure, causality, pacing, and closure	Scene progression, causal continuity, escalation, turning points, proportional pacing, and an ending reflecting actions and consequences.
M-F3.2	Character and relationship development	Goals, knowledge, emotions, actions, and transitions supporting character development; relationship change earned through prior interaction.
M-F3.3	Scene, dialogue, prose, and emotional execution	Scene realization, distinct dialogue, clarity/control, sensory and atmospheric specificity, credible emotion, and engagement without ornament as a proxy.

Appendix

Table 10: Extended whole-text rubric.

Figure 6: Per-dimension model profiles for the two settings. The rightmost extended dimension group separates multilingual-integration dimensions from general narrative dimensions. Values are dual-judge means on the 1–5 dimension scale.

Model	M-F1	M-F2A	M-F2B	M-F3.1	M-F3.2	M-F3.3	M-F3.4
GPT-5.6-Sol	4.737	4.902	4.679	4.819	4.886	4.391	4.851
GPT-5.6-Terra	4.330	4.584	4.514	4.544	4.684	4.348	4.455
GPT-5.5	4.510	4.777	4.621	4.595	4.786	4.315	4.691
Gemini-3.5-Flash	3.953	4.201	4.058	4.055	4.110	3.954	3.893
Qwen3.7-Plus	3.751	4.074	3.915	3.809	3.876	3.595	3.636
Qwen3.7-Max	3.896	4.091	4.018	3.834	3.894	3.697	3.808

Appendix

Table 13: Extended whole-text dimension means (/5). IDs and definitions are in Appendix H .

Task / comparison	$\rho$	Mean $\|\Delta r\|$	Max $\|\Delta r\|$
Focused: GPT vs. Dual	1.000	0.000	0
Focused: DeepSeek vs. Dual	0.993	0.167	1
Focused: GPT vs. DeepSeek	0.993	0.167	1
Extended: GPT vs. Dual	0.986	0.333	1
Extended: DeepSeek vs. Dual	1.000	0.000	0
Extended: GPT vs. DeepSeek	0.986	0.333	1

Appendix

Table 14: Rank robustness on the same twelve models and current /20 task totals. Average ranks resolve ties.

Figure 8: Judge-side ranking shift for the twelve models under the benchmark aggregation rules. The markers give the rank assigned by the GPT-side judge, the DeepSeek-side judge, and their article-level mean.

Prompt constructor	GPT annotator	Qwen annotator
GPT	F .951 E .965	F .888 E .979
Qwen	F .958 E .916	F .937 E .972

Appendix

Table 15: Rank agreement with the original main-experiment ranking under the

2\times 2

Prompt-constructor

\times

event-annotator sensitivity design. Each cell reports Spearman’s

\rho

across the twelve generation models for Focused (F) and Extended (E) Narrative.

	D2			D3
Quantity	Sol $\to$ DS	DS $\to$ Sol	Quantity	Sol $\to$ DS	DS $\to$ Sol
Whole	0.269	0.239	Whole	0.274	0.237
M-F1	0.221	0.214	M-F1	0.212	0.199
M-F2A	0.222	0.155	M-F2A	0.235	0.157
M-F2B	0.185	0.184	M-F2B	0.183	0.184
M-F3.1	0.195	0.151	M-F3.1	0.193	0.147

Appendix

Table 16: Cross-judge associations in Extended Narrative: one judge’s event-level score against the other judge’s whole-text score, in both directions. Sol

\to

DS reads from the event-level judge to the whole-text judge, and DS

\to

Sol reverses the two roles; the arrows mark the direction of the association, not a causal path. Section 5.4 reports their mean. Generation-model and prompt-card fixed effects are removed.

Setting	Ref.	Grouping F1 [95% CI]	Boundary F1	Exact
Focused	A1	.919 [.820, 1.000]	.980	.875
Focused	A2	.968 [.917, 1.000]	.990	.917
Extended	A1	.998 [.994, 1.000]	.988	.958
Extended	A2	.999 [.997, 1.000]	.990	.917

Appendix

Table 17: Agreement between GPT-5.6-Sol’s recorded event grouping and the two human annotators over candidates identified as CS by both sides. Grouping F1 compares same-event membership; Boundary F1 compares grouping-induced event boundaries; Exact is the proportion of articles with identical projected event partitions.

Annotation class	Factual acc.	Temporal acc.
Character profile	98.93%	96.98%
Character state	97.77%	99.44%
Narrative arc	99.50%	99.61%
Relationship state	96.59%	99.24%
World state	99.19%	99.18%

Appendix

Table 18: Adjudicated fidelity of the evidence records used in Extended Narrative. Temporal accuracy is computed over assertions for which temporal placement is applicable.

Setting	Evaluator	Self $n$	Non-self $n$	Self residual	Non-self residual	Self $-$ non-self [95% CI]
Focused	GPT-5.6-Sol	15	165	$-$ 0.548	$-$ 0.193	$-$ 0.355 [0.100, 0.595]
Focused	DeepSeek-V4-Flash	15	165	$-$ 0.351	$-$ 0.238	$-$ 0.112 [ $-$ 0.409, 0.248]
Extended	GPT-5.6-Sol	12	132	$-$ 1.236	$-$ 1.337	$-$ 0.101 [ $-$ 1.370, 1.482]
Extended	DeepSeek-V4-Flash	12	132	$-$ 0.632	$-$ 1.096	$-$ 0.463 [ $-$ 1.333, 0.524]

Appendix

Table 19: Human-anchored evaluator–generator overlap diagnostic. Residuals are evaluator article totals minus the three-rater human consensus. The final contrast compares exact-self outputs with all non-self outputs. Confidence intervals use 2,000 article-level bootstrap resamples.

Contrast	Estimate	95% CI
Exact self $-$ other family	$-$ 0.450	[0.170, 0.682]
Same family $-$ other family	$-$ 0.522	[0.378, 0.667]
Exact self $-$ same family	$-$ 0.072	[ $-$ 0.345, 0.161]

Appendix

Table 20: Decomposition of the Focused Narrative GPT evaluator signal. Contrasts are differences in evaluator-minus-human article-level residuals.

Setting	Aggregation	$\boldsymbol{\rho}$	Max $\|\Delta r\|$	Top model
Focused	Drop D1	1.000	0	GPT-5.5
Focused	D1 half weight	1.000	0	GPT-5.5
Focused	Rank average	1.000	0.5	GPT-5.5
Extended	Drop D1	1.000	0	GPT-5.6-Sol
Extended	D1 half weight	1.000	0	GPT-5.6-Sol
Extended	Rank average	.977	1.5	GPT-5.6-Sol

Appendix

Table 22: Sensitivity of within-setting model ordering to alternative aggregation schemes. Spearman’s

\rho

is computed against the equal-weight

\Sigma

ordering. Average ranks are used for ties. For the rank-average rows, Max

|\Delta r|

denotes the change in the underlying average-rank value rather than displacement in final leaderboard position.

Setting	D1	D2	D3	Whole
Focused	.061	.282	.380	1.190
Extended	.487	.942	1.120	1.775

Appendix

Table 23: Across-model range of the reported dimensions in each narrative setting. Each value is the maximum model mean minus the minimum model mean.

§	Quantity	Estimate	Original 95% CI	Seed-clustered 95% CI
5.3	Focused salience–D2	0.642	0.625, 0.658	0.635, 0.650
5.3	Focused salience–D3	0.496	0.475, 0.517	0.484, 0.508
5.3	Extended salience–D2	0.768	0.756, 0.779	0.763, 0.773
5.3	Extended salience–D3	0.789	0.778, 0.799	0.785, 0.793
5.4	D2–M-F1	0.217	0.190, 0.244	0.186, 0.255
5.4	D2–M-F2A	0.188	0.161, 0.215	0.120, 0.255

Appendix

Table 24: Cross-judge associations. Rows for Section 5.3 report within-article event-level associations between narrative salience and local quality; rows for Section 5.4 report article-level associations between local and whole-text quality. Estimates are the means of the two reciprocal cross-judge correlations. The original intervals are computed at the article level; Seed-clustered intervals resample Seed clusters within each setting with replacement (10,000 replicates).

Model	Focused /20	Extended /20
GPT-5.6-Sol	15.501 [15.476, 15.527]	17.767 [17.710, 17.823]
GPT-5.6-Terra	15.410 [15.382, 15.438]	17.352 [17.271, 17.433]
GPT-5.5	15.537 [15.513, 15.561]	17.348 [17.283, 17.410]
Gemini-3.5-Flash	14.893 [14.862, 14.925]	16.471 [16.385, 16.552]
Qwen3.7-Plus	14.376 [14.339, 14.413]	16.065 [15.969, 16.160]
Qwen3.7-Max	14.540 [14.504, 14.576]	16.220 [16.139, 16.298]

Appendix

Table 25: Seed-clustered 95% percentile intervals for the Focused and Extended within-setting composites. Intervals resample the 546 Seed clusters with replacement (10000 replicates), drawing both settings jointly.

Model	Events/article	Median	CS density /1k	CS span (%)	$\Sigma$ (/20)
Extended Narrative
GPT-5.6-Sol	20.36	20	4.78	9.1	17.767
GPT-5.6-Terra	11.39	11	2.94	5.5	17.352
GPT-5.5	20.52	20	4.51	6.6	17.348
DeepSeek-V4-Flash	17.32	17	4.54	11.2	16.962
DeepSeek-V4-Pro	20.25	21	4.16	9.0	16.567

Appendix

Table 26: Switching behavior by generation model. CS density is the number of discovered CS events per 1,000 non-whitespace code points; CS span is the mean article-level proportion covered by identified CS-event spans.

Dimension / split	Hard	Easy	Easy $-$ hard	Standardized gap	95% CI
M-F1	3.645	3.943	0.2983	1.182	0.229, 0.368
M-F2A	4.159	4.268	0.1092	0.601	0.057, 0.162
M-F2B	3.812	4.245	0.4333	1.216	0.330, 0.536
M-F3.1	3.937	4.039	0.1025	0.546	0.051, 0.154
M-F3.2	4.112	4.178	0.0667	0.427	0.022, 0.112
M-F3.3	3.753	3.848	0.0942	0.591	0.050, 0.139

Appendix

Table 27: Prompt-difficulty gaps. The first seven rows use the joint prompt-level bootstrap of the difficulty analysis (2,000 replicates). Standardized gaps divide the easy–hard difference by

\sqrt{(s_{\mathrm{hard}}^{2}+s_{\mathrm{easy}}^{2})/2}

, where the two SDs are computed within the hard and easy prompt groups from per-prompt means of the six validation models. Standardized gaps are reported only for the seven dimension rows. The remaining rows repeat the CS-versus-general contrast, defined as the mean easy–hard gap across M-F1 and M-F3.4 minus the mean gap across M-F3.1–M-F3.3, under alternative six/six splits. In the main split prompt difficulty is estimated from GPT-5.6-Sol, DeepSeek-V4-Flash, Gemini-3.5-Flash, MiniMax-M2.7, Qwen3.7-Max, Qwen3.5-35B-A3B and validated on the other six; each swap row moves one family between the two halves (GPT swap: GPT-5.6-Terra for GPT-5.6-Sol; DeepSeek swap: DeepSeek-V4-Pro for DeepSeek-V4-Flash; MiniMax swap: MiniMax-M2.5 for MiniMax-M2.7; Qwen swap: Qwen3.7-Plus and Qwen3.5-397B-A17B for Qwen3.7-Max and Qwen3.5-35B-A3B).

Measure	Basis	D2	D3	Whole	$\Sigma$
Extended Narrative
Events	pooled	-.026	+.014	+.163	+.032
Events	within	-.105	-.123	+.030	-.076
Density	pooled	-.149	-.145	-.063	-.123
Density	within	-.049	-.077	+.026	-.027
Focused Narrative

Appendix

Table 28: Article-level associations between switching propensity and benchmark scores. Within-model correlations remove generation-model means before computing the correlation.

Setting	Model pair	$\boldsymbol{\Delta\Sigma}$	95% CI	Holm $p$
Focused	GPT-5.6-Sol – GPT-5.5	$-0.0355$	$[-0.0670,-0.0040]$	.0536
Focused	Qwen3.7-Plus – Qwen3.5-397B-A17B	$+0.0193$	$[-0.0343,0.0717]$	.4648
Focused	MiniMax-M2.5 – MiniMax-M2.7	$-0.0865$	$[-0.1578,-0.0143]$	.0504
Extended	GPT-5.6-Terra – GPT-5.5	$+0.0046$	$[-0.0899,0.0975]$	.9064
Extended	Gemini-3.5-Flash – DeepSeek-V4-Pro	$-0.0967$	$[-0.2176,0.0257]$	.2368

Appendix

Table 29: Within-setting model pairs that are not significant after Holm correction over the 66 pairwise comparisons. Differences are row-first minus row-second in the reported

\Sigma

(/20).

Setting	Comparison	Articles A / B	$\boldsymbol{\rho}$	95% CI	Max $\|\Delta r\|$
Focused	ASCEND vs. SwitchLingua	912 / 3,972	.993	[.979, 1.000]	1
Focused	Shared vs. task-specific Seeds	3,132 / 1,752	.993	[.979, 1.000]	1
Extended	ASCEND vs. SwitchLingua	840 / 3,960	.986	[.944, 1.000]	1
Extended	Shared vs. task-specific Seeds	3,132 / 1,668	.986	[.979, 1.000]	1

Appendix

Table 30: Within-setting robustness of model ordering to source and Seed composition. Spearman’s

\rho

compares the twelve-model rankings under the two conditions in each row. Articles A and B are the scored articles belonging to each side of the comparison, twelve per prompt; the prompt-level counts behind them are reported in the text. Confidence intervals use prompt-clustered bootstrap resampling.

Setting	Clusters	Widest rank	$P(\text{rank-1}$	$P(\text{top-3}$
	(Seed/group)	interval	$\text{holds})$	$\text{set holds})$
Focused	407 / 350	1 / 1	.986 / .987	1.000 / 1.000
Extended	400 / 340	1 / 1	1.000 / 1.000	1.000 / 1.000

Appendix

Table 31: Source-group clustered bootstrap. Every quantity is reported for Seed clustering (current intervals) and, after the slash, for source-group clustering. Both settings are drawn jointly. The widest rank interval is the largest

97.5\%

-minus-

2.5\%

rank range across the twelve models.

Code-Switching

Code-switching studies how people combine languages in speech and text and how AI systems can recognize, understand, and generate these mixtures.

See all 22 papers

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Code Generation Code Generation Evaluation
Yandex School of Data Analysis, Applied AI Institute · GigaCode
Latent Mechanisms of Language Control in Multilingual Language Models
Aug 31, 2026Ryo Mitsuhashi, Sabri Boughorbel, Majd HawaslyMultilingual Language Models Multilingual Language Model Evaluation
Princeton University New Jersey, USA · Prince Sattam bin Abdulaziz University Riyadh, Saudi Arabia · QCRI, Hamad Bin Khalifa University, Doha, Qatar
Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning
May 28, 2026Shunta Asano, Jeonghun Baek, Toshihiko YamasakiMultilingual Language Models Code-Switching
The University of Tokyo

Jun 18, 2026J/K move · Enter open · S save

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov+4

Yandex School of Data Analysis, Applied AI Institute · GigaCode

Code Generation Code Generation Evaluation Code Generation Benchmarks Multilingual Code Generation

Open paper AskPDF arXiv

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

Abstract

Code-Switching

Explore similar work

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Latent Mechanisms of Language Control in Multilingual Language Models

Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning