Far Out: Evaluating Language Models on Slang in Australian and Indian English

Deniz Kaya Dilsiz; Dipankar Srirag; Aditya Joshi

cs.CLFeb 17, 2026

Far Out: Evaluating Language Models on Slang in Australian and Indian English

Authors: Deniz Kaya Dilsiz, Dipankar Srirag, Aditya Joshi

Organizations: University of New South Wales, Sydney, Australia

Abstract

Language models exhibit systematic performance gaps when processing text in non-standard language varieties, yet their ability to comprehend variety-specific slang remains underexplored for several languages. We present a comprehensive evaluation of slang awareness in Indian English (en-IN) and Australian English (en-AU) across seven state-of-the-art language models. We construct two complementary datasets: WEB, containing 377 web-sourced usage examples from Urban Dictionary, and GEN, featuring 1,492 synthetically generated usages of these slang terms, across diverse scenarios. We assess language models on three tasks: target word prediction (TWP), guided target word prediction (TWP $^*$ ) and target word selection (TWS). Our results reveal four key findings: (1) Higher average model performance TWS versus TWP and TWP $^*$ , with average accuracy score increasing from 0.03 to 0.49 respectively (2) Stronger average model performance on WEB versus GEN datasets, with average similarity score increasing by 0.03 and 0.05 across TWP and TWP $^*$ tasks respectively (3) en-IN tasks outperform en-AU when averaged across all models and datasets, with TWS demonstrating the largest disparity, increasing average accuracy from 0.44 to 0.54. These findings underscore fundamental asymmetries between generative and discriminative competencies for variety-specific language, particularly in the context of slang expressions despite being in a technologically rich language such as English.

Figures & tables

Figure 1: Methodological Overview of Our Approach.

Variety	Count		$\bar{\text{p}}$	$\bar{\text{d}}$	$\bar{\text{u}}$		Perp.		Latest Entry
Variety	web	gen	$\bar{\text{p}}$	$\bar{\text{d}}$	web	gen	web	gen	Latest Entry
en-AU	313	1244	7.7	138	90.4	443.0	701.5	48.9	Sept 10 2025
en-IN	64	248	8.0	197.7	114.1	468.2	406.5	47.1	Oct 15 2025

Table 1: Constructional statistics of web and gen .

\bar{\text{p}}

\bar{\text{d}}

and

\bar{\text{u}}

is the average character length of a slang phrase, corresponding definition and usage/scenario respectively. Perp. represents average perplexity computed using GPT-2 Radford et al. (2019) .

Variety	R 1	R 2	R ${}_{\text{L}}$	Sim.
en-AU	0.25	0.03	0.16	0.39
en-IN	0.24	0.03	0.15	0.37

Table 2: Diversity analysis of generated scenarios in gen . R

{}_{\text{(1, 2, L)}}

are average ROUGE scores and Sim. is the average cosine similarity between scenario embeddings, extracted using allMiniLM-L6-v2 Reimers and Gurevych (2019) .

Figure 3: Performance comparison of various models on the target word prediction (TWP) task across en-AU and en-IN.

Figure 4: Performance comparison of various models on the guided target word prediction (TWP ∗ ) task across en-AU and en-IN.

Figure 5: Performance comparison of various models on the target word selection (TWS) task across en-AU and en-IN.

Task	Acc.	Sim.
TWP	0.02	0.27
TWP ∗	0.04	0.28
TWS	0.49	0.61

Table 3: Average performance of language models across tasks, averaged over all models, varieties, and datasets.

Domain	TWP		TWP ∗		TWS
Domain	Acc.	Sim.	Acc.	Sim.	Acc.	Sim.
web	0.04	0.29	0.07	0.32	0.50	0.61
gen	0.01 (0.03)	0.26 (0.03)	0.02 (0.05)	0.24 (0.08)	0.49 (0.01)	0.61 (0.00)

Table 4: Effect of domain on task performance, averaged across models and varieties. The decrease in performance is shown in (red) .

Variety	TWP		TWP ∗		TWS
Variety	Acc.	Sim.	Acc.	Sim.	Acc.	Sim.
en-AU	0.02	0.26	0.04	0.27	0.44	0.57
en-IN	0.03 (0.01)	0.28 (0.02)	0.05 (0.01)	0.29 (0.02)	0.54 (0.10)	0.65 (0.08)

Table 5: Effect of language variety on task performance, averaged across models and domains. The increase in performance is shown in (blue) .

Category	Target phrase	Description	Prediction
Literalisation	maggot bag	Australian slang for a meat pie	pie
Generic Substitution	absolute unit	Someone very heavy or large	behemoth
Semantic Drift	bogan	An unsophisticated person from a working-class background	heavy metal band
Contextual Misinterpretation	a kangaroo loose in the top paddock	Intellectually inadequate	more to this picture
True Failure	amber fluid	Australian slang for beer	34

Table 6: Representative examples of error categories observed in olmo predictions on the gen dataset.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Variety	en-AU	en-IN
phrase $p$	smoko	prepone
definition $d$	a slang term used on building sites in Australia, meaning a morning-tea break, or a smoke break.	Function: transitive verb Inflected forms: pre•poned; pre•pon•ing Etymology: Latin preponere to place before, prepone, from pre- + ponere to place – more at POSITION Date: Has been in use in urban English spoken in India since at least the 1950s To advance an event or activity to an earlier time. The closest American usage is “to advance” the timing of something. The word came into vogue in urban India as the opposite of “postponing” something.
usage example $u$	“we’ll knock off at 11 for smoko”	“To make sure we get to enjoy the fireworks display that starts promptly at 9 PM, let us prepone the dinner engagement to 7 rather than 8 tomorrow evening”
usage scenario $u$	The fluorescent lights of the London advertising agency hummed, a stark contrast to the quiet focus on Anika’s face as she stared at the mock-up on her screen. Her colleague, Ben, who had recently transferred from their Melbourne office, spun around in his chair. "You’ve been at that for hours, I’m grabbing a coffee and a biscuit, you keen for a quick smoko?"	Anjali tapped her stylus against the glass wall of the conference room in their bustling Bangalore office, catching Rohan’s attention as he walked by. The quarterly progress charts were displayed on the large monitor, but her focus was clearly elsewhere. "Rohan, I just got off the phone with the clients from Singapore; they’re flying in two days earlier than planned, so we need to prepone the final project presentation to Tuesday morning."

Appendix

Table 8: Examples from web and gen datasets.

Language Model	Parameters	Release Date
bert-base-uncased	110M	October 2018
roberta-large	355M	July 2019
xlm-roberta-large	550M	November 2019
granite-4.0-1b	1B	October 2025
llama-3.2-3b-instruct	3B	September 2024
olmo-2-1124-7b-instruct	7B	November 2024

Appendix

Table 9: Language Models used in Experimental Setup, their approximate Parameter Counts and Public Release Date

Language Model	Variety	Total	EM	Acc.	Sim (G)	Sim (M)	Corr
Target Word Prediction
bert-base-uncased	en-AU	1238	2	0.002	0.726	0.254	0.341
bert-base-uncased	en-IN	243	3	0.012	0.729	0.291	0.670
roberta-large	en-AU	1238	5	0.004	0.730	0.270	0.464
roberta-large	en-IN	243	6	0.025	0.738	0.312	0.742
xlm-roberta-large	en-AU	1238	3	0.002	0.726	0.260	0.370

Appendix

Table 10: Model performance metrics with gen across Target Word Prediction, Guided Target Word Prediction and Target Word Selection. We provide results on Exact Matches (EM), Accuracy (Acc), Sim(G) and Sim(M) refer to average cosine similarity scores using embedding models Granite-embedding-125m-english embeddings and Awasthy et al. (2025) Sentence-BERT embeddings Reimers and Gurevych (2019) respectively. We compute Pearson Correlation (Corr.) between the resulting similarity scores to measure the two models agreement.

Language Model	Variety	Total	EM	Acc.	Sim (G)	Sim (M)	Corr
Target Word Prediction
bert-base-uncased	en-AU	375	11	0.029	0.736	0.273	0.777
bert-base-uncased	en-IN	77	3	0.039	0.737	0.287	0.796
roberta-large	en-AU	375	22	0.059	0.743	0.304	0.824
roberta-large	en-IN	77	5	0.065	0.747	0.307	0.796
xlm-roberta-large	en-AU	375	17	0.045	0.737	0.288	0.831

Appendix

Table 11: Model performance metrics with web across Target Word Prediction, Guided Target Word Prediction and Target Word Selection. We provide results on Exact Matches (EM), Accuracy (Acc), Sim(G) and Sim(M) refer to average cossine similarity scores using embedding models Granite-embedding-125m-english embeddings and Awasthy et al. (2025) Sentence-BERT embeddings Reimers and Gurevych (2019) respectively. We compute Pearson Correlation (Corr) between the resulting similarity scores to measure the two models agreement.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content
Aug 5, 2026Arianna Denitto, Beatrice Savoldi
University of Torino, Department of Humanities, Via S. Ottavio 20, 10124 Torino, Italy · Fondazione Bruno Kessler, MT Unit, Via Sommarive 18, 38123, Povo (TN), Italy
Individual and Combined Effects of English as a Second Language and Typos on LLM Performance
Apr 6, 2026Serena Liu, Yutong Yang, Prisha Sheth +9Language Model Robustness LLM Evaluation
Harvard University, Boston, MA 02138, US
Side-by-side Comparison Amplifies Dialect Bias in Language Models
May 23, 2026Kritee Kondapally, Claire J. Smerdon, Pooja C. Patel +5Social Bias in Language Models Algorithmic Fairness
University of Southern California, USA

Aug 5, 2026J/K move · Enter open · S save

Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content

Arianna Denitto, Beatrice Savoldi

University of Torino, Department of Humanities, Via S. Ottavio 20, 10124 Torino, Italy · Fondazione Bruno Kessler, MT Unit, Via Sommarive 18, 38123, Povo (TN), Italy

Open paper AskPDF arXiv

Despite its cultural relevance and diffusion, queer slang remains underrepresented in Natural Language Processing research. Towards addressing this gap, we introduce Slang-Q, a manually curated dataset of naturally user-generated English sentences paired with queer slang terms and reference definitions, built upon a newly constructed taxonomy of 118 queer terms. We use this resource to conduct a first exploratory evaluation of language models on their ability to understand and define queer slang under varying prompting conditions. Slang-Q is intended as a basis for studying how current models handle sensitive, community-specific language and whether they can provide accurate and reliable information about such forms of identity and linguistic expression.

Abstract

LLM Evaluation

Explore similar work

Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Side-by-side Comparison Amplifies Dialect Bias in Language Models