cs.CLOct 19, 2026
SaveLoWAgent: A Low-Resource Multilingual Benchmark for Web Agents
Organizations: University of South Florida, USA · Instituto Politécnico Nacional, Mexico · Independent Researcher · Saarland University, Germany
Abstract
Web agents are typically evaluated on English-language interfaces, leaving their ability to operate across the world's multilingual websites largely unexplored. This gap is especially significant for low-resource languages, where standardized resources for evaluating web agents remain limited. To address this gap, we introduce Low-Resource Web Agent Benchmark (LoWAgent), a benchmark for evaluating web agents across different low-resource languages (LRLs) in comparison to the default, English. LoWAgent evaluates web agents across English and four LRLs: Hindi, Bengali, Urdu, and Amharic. The benchmark comprises of six fully localized websites, instantiated across five languages to yield 3,000 task instances. We evaluate three closed-source and five open-source models across all five languages on all six websites. Our findings show that both closed-source and open-source models experience performance degradation for LRLs compared to English, with substantially greater performance drops observed among the open-source model agents. We further evaluate the robustness of the models on realistic interface disruptions, such as cookie notices and advertisement pop-ups, and find that these further degrade agentic performance. Finally, we show that post-training on a small subset of LoWAgent traces can significantly improve low-resource language agentic task performance, demonstrating the usefulness of our benchmark.
Figures & tables
Figure 1: Example trajectories for equivalent shopping tasks in (a) English and (b) Bengali . The agent successfully completes the English task by applying the required price filter and selecting a valid item. In contrast, for the Bengali task, the agent correctly navigates and searches for the product but skips the filtering step, resulting in the selection of an item that violates the specified price constraint and leading to task failure.
| Action | Description |
| goto <path> | Opens a path on the current site. |
| click <n> | Follows link . |
| fill <n> k=v; | Sets and submits the named fields of form . When no fields are present, presses the form’s button. |
| stop <answer> | Ends the episode with an answer. |
Table 1: Action space: List of all actions available to the web agent in LoWAgent .
Figure 2: Example execution trajectory of the three different types of tasks in LoWAgent : (1) Information-seeking, (2) State-changing, and (3) Unachievable tasks.
| Type | Model | EN | BN | HI | UR | AM |
| Closed Source | Gemini-3.1-Flash-Lite | 86.40% | -3.70% | -1.50% | -4.00% | -7.30% |
| GPT-5.6 Luna | 85.30% | -7.30% | -6.60% | -9.30% | -16.60% | |
| GPT-6 Luna | 83.60% | -11.20% | -8.90% | -15.60% | -14.70% | |
| Open Source | Llama-3.1-8B | 40.90% | -16.00% | -11.60% | -12.70% | -34.50% |
| Gemma3-4B | 12.70% | -1.80% | 0.00% | -4.70% | -6.70% | |
| Gemma3-12B | 45.10% | -13.10% | -8.70% | -11.80% | -15.80% |
Table 2: Web agent performance relative to English. The English column reports the accuracy , while Language ( ) denotes the percentage-point difference in accuracy from English for each language . Open-source models exhibit substantially larger performance drops in non-English languages, while Gemini-3.1-Flash-lite has the smallest cross-lingual degradation.
Figure 3: Results on the 150 visual tasks across 3 websites and all 5 languages. Performance of open-source LLMs is significantly lower than the closed-source models (Gemini, GPT-5.6, and GPT-6).
Figure 4: Performance degradation across LLMs with and without interface disruptions (cookie notices and advertisement pop-ups). The cookie notice appears at the beginning of each task and must be dismissed before the agent can proceed. The advertisement pop-up appears after each click on the website, prompting the agent to install an app. Accepting the offer redirects the agent away from the website and results in task failure, so the agent must correctly dismiss the pop-up to continue. All models see substantial performance drops when interface disruptions are introduced, with Gemini-3.1 Flash-Lite showing the largest accuracy decline.
Figure 5: Results on the test subset for Llama-3.1-8B and Gemma3-12B after GRPO training on LoWAgent . Both models show substantial improvement across all languages, reiterating the effectiveness of LoWAgent .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Screenshot of the Ministore website in English, showing its interface and layout as presented to the agent during task execution.
Figure 7: Screenshot of the Redora website in English, showing its interface and layout as presented to the agent during task execution.
Figure 8: Screenshot of the GariBazaar website in Urdu, showing its interface and layout as presented to the agent during task execution.
Figure 9: Screenshot of the RailSetu website in Hindi, showing its interface and layout as presented to the agent during task execution.
Figure 10: Screenshot of the Haritia National Portal website in Bengali, showing its interface and layout as presented to the agent during task execution.
Figure 11: Screenshot of the Selam Airways website in Amharic, showing its interface and layout as presented to the agent during task execution.
Figure 12: Screenshot of the cookie notice visible to the agent during task execution.
Figure 13: Screenshot of the advertisement pop-up visible to the agent during task execution.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 93.3% | 90.7% | 88.0% | 92.0% | 86.7% | 90.1% |
| Redora | 85.3% | 85.3% | 84.0% | 81.3% | 82.7% | 83.7% |
| GariBazaar | 85.3% | 85.3% | 85.3% | 84.0% | 72.0% | 82.4% |
| Haritia Portal | 76.0% | 69.3% | 80.0% | 70.7% | 73.3% | 73.9% |
| RailSetu | 82.7% | 76.0% | 81.3% | 78.7% | 76.0% | 78.9% |
| Selam Airways | 96.0% | 89.3% | 90.7% | 88.0% | 84.0% | 89.6% |
Table 3: Gemini-3.1-Flash-Lite success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 93.3% | 72.0% | 73.3% | 61.3% | 57.3% | 71.5% |
| Redora | 80.0% | 77.3% | 78.7% | 78.7% | 77.3% | 78.4% |
| GariBazaar | 86.7% | 84.0% | 78.7% | 80.0% | 68.0% | 79.5% |
| Haritia Portal | 77.3% | 70.7% | 74.7% | 77.3% | 64.0% | 72.8% |
| RailSetu | 81.3% | 76.0% | 80.0% | 77.3% | 64.0% | 75.7% |
| Selam Airways | 93.3% | 88.0% | 86.7% | 81.3% | 81.3% | 86.1% |
Table 4: GPT-5.6 Luna success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 90.7% | 64.0% | 61.3% | 44.0% | 48.0% | 61.6% |
| Redora | 86.7% | 76.0% | 78.7% | 86.7% | 78.7% | 81.3% |
| GariBazaar | 85.3% | 82.7% | 78.7% | 80.0% | 73.3% | 80.0% |
| Haritia Portal | 76.0% | 76.0% | 74.7% | 70.7% | 68.0% | 73.1% |
| RailSetu | 73.3% | 62.7% | 72.0% | 65.3% | 64.0% | 67.5% |
| Selam Airways | 89.3% | 73.3% | 82.7% | 61.3% | 81.3% | 77.6% |
Table 5: GPT-6 Luna success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 5.3% | 0.0% | 6.7% | 4.0% | 0.0% | 3.2% |
| Redora | 2.7% | 1.3% | 1.3% | 0.0% | 0.0% | 1.1% |
| GariBazaar | 2.7% | 1.3% | 0.0% | 1.3% | 0.0% | 1.1% |
| Haritia Portal | 5.3% | 6.7% | 2.7% | 0.0% | 0.0% | 2.9% |
| RailSetu | 9.3% | 5.3% | 8.0% | 4.0% | 0.0% | 5.3% |
| Selam Airways | 4.0% | 0.0% | 2.7% | 1.3% | 0.0% | 1.6% |
Table 6: Llama-3.2-3B success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 28.0% | 22.7% | 25.3% | 26.7% | 6.7% | 21.9% |
| Redora | 37.3% | 29.3% | 29.3% | 30.7% | 1.3% | 25.6% |
| GariBazaar | 36.0% | 17.3% | 26.7% | 21.3% | 4.0% | 21.1% |
| Haritia Portal | 42.7% | 28.0% | 28.0% | 37.3% | 16.0% | 30.4% |
| RailSetu | 46.7% | 28.0% | 29.3% | 28.0% | 5.3% | 27.5% |
| Selam Airways | 54.7% | 24.0% | 37.3% | 25.3% | 5.3% | 29.3% |
Table 7: Llama-3.1-8B success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 9.3% | 5.3% | 8.0% | 8.0% | 9.3% | 8.0% |
| Redora | 8.0% | 8.0% | 9.3% | 9.3% | 5.3% | 8.0% |
| GariBazaar | 14.7% | 14.7% | 12.0% | 10.7% | 8.0% | 12.0% |
| Haritia Portal | 6.7% | 8.0% | 12.0% | 2.7% | 4.0% | 6.7% |
| RailSetu | 25.3% | 17.3% | 21.3% | 14.7% | 6.7% | 17.1% |
| Selam Airways | 12.0% | 12.0% | 13.3% | 2.7% | 2.7% | 8.5% |
Table 8: Gemma3-4B success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 37.3% | 25.3% | 33.3% | 22.7% | 24.0% | 28.5% |
| Redora | 38.7% | 29.3% | 37.3% | 38.7% | 34.7% | 35.7% |
| GariBazaar | 48.0% | 29.3% | 30.7% | 26.7% | 22.7% | 31.5% |
| Haritia Portal | 38.7% | 38.7% | 34.7% | 36.0% | 24.0% | 34.4% |
| RailSetu | 53.3% | 34.7% | 41.3% | 40.0% | 37.3% | 41.3% |
| Selam Airways | 54.7% | 34.7% | 41.3% | 36.0% | 33.3% | 40.0% |
Table 9: Gemma3-12B success rates across the six websites and five languages.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 46.7% | 40.0% | 30.7% | 28.0% | 26.7% | 34.4% |
| Redora | 46.7% | 34.7% | 28.0% | 34.7% | 28.0% | 34.4% |
| GariBazaar | 50.7% | 45.3% | 40.0% | 44.0% | 30.7% | 42.1% |
| Haritia Portal | 46.7% | 36.0% | 34.7% | 37.3% | 28.0% | 36.5% |
| RailSetu | 62.7% | 50.7% | 52.0% | 52.0% | 44.0% | 52.3% |
| Selam Airways | 64.0% | 52.0% | 58.7% | 54.7% | 48.0% | 55.5% |
Table 10: Gemma3-27B success rates for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 77.3% | 38.7% | 45.3% | 37.3% | 26.7% | 45.1% |
| Redora | 56.0% | 53.3% | 54.7% | 53.3% | 36.0% | 50.7% |
| GariBazaar | 70.7% | 49.3% | 53.3% | 48.0% | 24.0% | 49.1% |
| Haritia Portal | 41.3% | 38.7% | 45.3% | 46.7% | 26.7% | 39.7% |
| RailSetu | 69.3% | 56.0% | 57.3% | 54.7% | 38.7% | 55.2% |
| Selam Airways | 70.7% | 62.7% | 66.7% | 54.7% | 33.3% | 57.6% |
Table 11: GPT-OSS-20B success rates for each website and language.
| Access | Model | EN | BN | HI | UR | AM |
| Closed Source | Gemini-3.1-Flash-lite | 86.4% | 82.7% | 84.9% | 82.4% | 79.1% |
| GPT-5.6 Luna | 85.3% | 78.0% | 78.7% | 76.0% | 68.7% | |
| GPT-6 Luna | 83.6% | 72.4% | 74.7% | 68.0% | 68.9% | |
| Open Source | Llama-3.2-3B | 4.9% | 2.4% | 3.6% | 1.8% | 0.0% |
| Llama-3.1-8B | 40.9% | 24.9% | 29.3% | 28.2% | 6.4% | |
| Gemma3-4B | 12.7% | 10.9% | 12.7% | 8.0% | 6.0% |
Table 12: Web agent success rates aggregated across all six websites for each language, for closed-source and open-source models.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 12.0% | 17.3% | 13.3% | 16.0% | 12.0% | 14.1% |
| Redora | 10.7% | 14.7% | 8.0% | 18.7% | 6.7% | 11.7% |
| GariBazaar | 9.3% | 26.7% | 20.0% | 22.7% | 10.7% | 17.9% |
| Haritia Portal | 5.3% | 13.3% | 12.0% | 8.0% | 8.0% | 9.3% |
| RailSetu | 2.7% | 13.3% | 12.0% | 13.3% | 5.3% | 9.3% |
| Selam Airways | 28.0% | 40.0% | 40.0% | 33.3% | 34.7% | 35.2% |
Table 13: Gemini-3.1-Flash-Lite success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 50.7% | 40.0% | 49.3% | 30.7% | 26.7% | 39.5% |
| Redora | 40.0% | 42.7% | 38.7% | 34.7% | 41.3% | 39.5% |
| GariBazaar | 60.0% | 53.3% | 57.3% | 42.7% | 42.7% | 51.2% |
| Haritia Portal | 49.3% | 38.7% | 48.0% | 30.7% | 48.0% | 42.9% |
| RailSetu | 53.3% | 42.7% | 60.0% | 54.7% | 49.3% | 52.0% |
| Selam Airways | 42.7% | 37.3% | 44.0% | 40.0% | 50.7% | 42.9% |
Table 14: GPT-5.6-Luna success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 56.0% | 56.0% | 58.7% | 38.7% | 45.3% | 50.9% |
| Redora | 66.7% | 72.0% | 72.0% | 68.0% | 72.0% | 70.1% |
| GariBazaar | 81.3% | 73.3% | 73.3% | 69.3% | 72.0% | 73.9% |
| Haritia Portal | 38.7% | 68.0% | 68.0% | 60.0% | 56.0% | 58.1% |
| RailSetu | 70.7% | 61.3% | 65.3% | 60.0% | 62.7% | 64.0% |
| Selam Airways | 72.0% | 74.7% | 74.7% | 54.7% | 80.0% | 71.2% |
Table 15: GPT-6-Luna success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 18.7% | 22.7% | 22.7% | 14.7% | 0.0% | 15.7% |
| Redora | 33.3% | 22.7% | 24.0% | 16.0% | 0.0% | 19.2% |
| GariBazaar | 32.0% | 17.3% | 21.3% | 14.7% | 0.0% | 17.1% |
| Haritia Portal | 46.7% | 25.3% | 29.3% | 22.7% | 1.3% | 25.1% |
| RailSetu | 34.7% | 17.3% | 25.3% | 21.3% | 1.3% | 20.0% |
| Selam Airways | 36.0% | 14.7% | 28.0% | 18.7% | 0.0% | 19.5% |
Table 16: Llama-3.1-8B success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 13.3% | 16.0% | 14.7% | 13.3% | 14.7% | 14.4% |
| Redora | 2.7% | 6.7% | 5.3% | 6.7% | 2.7% | 4.8% |
| GariBazaar | 12.0% | 8.0% | 8.0% | 12.0% | 6.7% | 9.3% |
| Haritia Portal | 16.0% | 13.3% | 17.3% | 12.0% | 8.0% | 13.3% |
| RailSetu | 12.0% | 12.0% | 12.0% | 12.0% | 10.7% | 11.7% |
| Selam Airways | 10.7% | 10.7% | 9.3% | 6.7% | 8.0% | 9.1% |
Table 17: Gemma-3-4B success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 29.3% | 13.3% | 16.0% | 6.7% | 30.7% | 19.2% |
| Redora | 5.3% | 14.7% | 9.3% | 9.3% | 16.0% | 10.9% |
| GariBazaar | 12.0% | 10.7% | 9.3% | 5.3% | 16.0% | 10.7% |
| Haritia Portal | 13.3% | 26.7% | 25.3% | 17.3% | 20.0% | 20.5% |
| RailSetu | 14.7% | 13.3% | 9.3% | 12.0% | 22.7% | 14.4% |
| Selam Airways | 14.7% | 22.7% | 17.3% | 14.7% | 30.7% | 20.0% |
Table 18: Gemma-3-12B success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 14.7% | 24.0% | 20.0% | 20.0% | 21.3% | 20.0% |
| Redora | 12.0% | 9.3% | 8.0% | 9.3% | 10.7% | 9.9% |
| GariBazaar | 12.0% | 13.3% | 13.3% | 17.3% | 10.7% | 13.3% |
| Haritia Portal | 16.0% | 14.7% | 14.7% | 12.0% | 13.3% | 14.1% |
| RailSetu | 13.3% | 17.3% | 14.7% | 14.7% | 13.3% | 14.7% |
| Selam Airways | 17.3% | 26.7% | 24.0% | 20.0% | 26.7% | 22.9% |
Table 19: Gemma-3-27B success rates with interface disruptions for each website and language.
| Website | EN | BN | HI | UR | AM | Overall |
| Ministore | 25.3% | 21.3% | 32.0% | 24.0% | 14.7% | 23.5% |
| Redora | 24.0% | 16.0% | 20.0% | 12.0% | 9.3% | 16.3% |
| GariBazaar | 37.3% | 28.0% | 20.0% | 26.7% | 8.0% | 24.0% |
| Haritia Portal | 28.0% | 28.0% | 22.7% | 29.3% | 8.0% | 23.2% |
| RailSetu | 44.0% | 36.0% | 26.7% | 28.0% | 12.0% | 29.3% |
| Selam Airways | 38.7% | 38.7% | 32.0% | 32.0% | 10.7% | 30.4% |
Table 20: GPT-OSS-20B success rates with interface disruptions for each website and language.
More in this topic
Web Agent Benchmarks
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.
See all 63 papers