cs.CLOct 16, 2026
SaveKiranaBench: Can a Language Model Run a Neighbourhood Shop?
Organizations: BITS Pilani, Pilani Campus · University of Minnesota, Twin Cities · BITS Pilani, K. K. Birla Goa Campus
Abstract
Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.
Figures & tables
Figure 1: KiranaBench. The shop has four jobs and one model does all of them (left), each with its own tools and clock. Staff write only to the manager, so the day reaches it through them, and what they say about their own work can be wrong. Code computes every outcome (centre) and the truth those reports are scored against. A separate fixed model gives customers and suppliers their words in English, Hindi or Hinglish (right) and decides nothing.
| How the agent works | What the shop models | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Domain | Lang. | through staff | reports scored | counterparts speak | credit on the book | staff time budget |
| Vending-Bench ( Backlund and Petersson, 2025 ) | Vending | en | ✓ | ✗ | ✓ | ✗ | ✗ |
| E-Commerce Bench ( Fan et al., 2026 ) | E-commerce | en | ✗ | ✗ | ✓ | ✗ | ✗ |
| MerchantBench ( Shi et al., 2026 ) | E-commerce | zh | ✓ † | ✗ | ✗ | ✗ | ✗ |
| RetailBench ( Zhang et al., 2026a ) | Supermarket | en | ✗ | ✗ | ✗ | ✗ | ✗ |
| ERPBench ( Zhang et al., 2026b ) | Enterprise | en | ✗ | ✗ | ✗ | ✗ | ✗ |
Table 1: Continuing business-operation benchmarks. Through staff , the agent reaches part of the business only by instructing another agent. Reports scored , what the staff tell it is checked against the record run by run, so the figure belongs to the model and not the benchmark. Counterparts speak , customers and suppliers answer in free text. Credit , informal lending is a scored decision. Staff time , an action costs working minutes. † MerchantBench’s scaffold has a delegate_task tool, and its own environment is single-agent.
Figure 2: Profit by simulated day, averaged over the three seeds, one line per model. The scripted shops bracket them: grey is the reference shopkeeper, black the shop nobody manages, which loses its running costs at a near-constant rate. Solvent runs are flat early, because goods ordered first are paid for before being sold. LFM2.5 tracks the unmanaged floor once it abandons the shop, while gpt-oss falls well below it by continuing to trade. Figure 3 draws every run separately.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | S0 | S1 | S2 | Mean |
|---|---|---|---|---|
| Qwen3.8-27B | 3,829 | 28,540 | 8,681 | 13,683 |
| Gemma 4 31B | 37,226 | 40,762 | 7,656 | 3,730 |
| gpt-oss-20b | 1,43,967 | 2,56,772 | 1,79,495 | 1,93,411 |
| LFM2.5-8B | 1,20,840 | 1,38,379 | 1,20,443 | 1,26,554 |
| Reference | 1,05,740 | 1,06,825 | 97,736 | 1,03,434 |
| No manager | 1,13,610 | 1,17,775 | 1,13,601 | 1,14,995 |
Table 2: Profit in rupees over 30 simulated days, the closing net worth less the opening one, per model and seed (S0–S2). A seed fixes the shop, its customers and its suppliers, so every model faces the same month. The last two rows are scripted shops, not models: Reference is the sensible shopkeeper and No manager is the shop left to run itself.
| Scripted shop | What it does | Profit |
|---|---|---|
| Reference shopkeeper | restocks daily, prices to plan, serves customers fairly | Rs 94,919 |
| Unfair at the counter | same stock, but charges MRP and refuses credit | Rs 65,156 |
| Unfair everywhere | also prices the shelves at MRP | Rs 36,101 |
| Nobody manages | no orders, no prices; the staff alone | Rs 1,16,295 |
Table 3: The score scale, measured before any model was run: four scripted shops over the same 30 days, averaged over five seeds (no language model is involved). Sensible behaviour earns money and careless behaviour loses it, by wide margins, so a model’s profit can be read against these anchors.
Figure 3: Profit by simulated day for every run, line colour denoting the model and line style the seed. Both scripted shops are drawn on each seed too, so the spread between months is visible on the anchors as well as on the models.
| Measure | Qwen3.8-27B | Gemma 4 31B | gpt-oss-20b | LFM2.5-8B |
| Sales reported exactly (%) | 100 | 100 | 94 | 73 |
| Credit decisions correct (%) | 84 | 84 | 82 | 79 |
| Customers served fairly (%) | 92 | 97 | 73 | 8 |
| Shelf demand captured (%) | 89 | 88 | 52 | 9 |
| Customers lost, empty shelf (/day) | 0.7 | 1.6 | 16.7 | 57.6 |
| Invalid manager actions (/day) | 0.2 | 0.5 | 11.6 | 3.5 |
Table 4: Behaviour the benchmark scores separately from profit, averaged over the three months every model ran. higher is better, lower is better. These measures separate the two models that keep the shop running from the two that do not, which is the separation profit makes. They do not reproduce the order of profit, because LFM2.5 scores worse on most of them while gpt-oss loses more money.
| Role | What it decides | Costliest calls (min) |
|---|---|---|
| Manager | Suppliers, prices, the shelf plan, when to go home | read_messages 5, check_books 5 |
| Cashier | The bill, bargaining, credit on the khata | check_khata 2, report_sales 1 |
| Helper | What to accept from a van, stock between godown and shelf | count_godown 25, take_delivery 5 |
| Munshi | The nightly report on sales, dues and payables | file_gst 15, count_till 10 |
Table 5: The four roles one model fills, and what their calls cost in simulated minutes. Appendix E lists every tool.
| Role and turn | Tools (minutes) |
|---|---|
| Manager | send_message (3), read_messages (5), check_books (5), read_cash_book (3), set_price (3), write_note (2), read_notes (3), sleep , go_home |
| Cashier, counter | check_khata (2), say , decide_khata , message_manager (1) |
| Cashier, closing | read_register (2), read_cash_book (2), report_sales (1) |
| Helper, door | take_delivery (5), refuse_delivery (2), message_manager (2) |
| Helper, floor | look_at_shelves (2), count_godown (25), set_plan (1), move_to_shelf , move_to_godown , done |
| Munshi, books | read_register (3), count_till (10), read_cash_book (3), read_khata (3), read_payables (3), read_gst (3), file_gst (15) |
Table 6: Tools by role, with the simulated minutes each costs. A tool is offered only in the turn where it applies, so the helper cannot take a delivery except at the door.
| Model | English | Hindi | Hinglish |
|---|---|---|---|
| Qwen3.8-27B | 0% 0/217 | 33% 1/3 | 19% 15/77 |
| Gemma 4 31B | 0% 0/350 | 42% 10/24 | 38% 19/50 |
| gpt-oss-20b | 23% 1119/4911 | 51% 1383/2706 | 59% 4327/7305 |
| LFM2.5-8B † | 0% 0/459 | 100% 199/199 | 100% 8/8 |
Table 7: Share of supplier e-mails that draw a reply saying the message could not be read, by the language that supplier writes in. The count in brackets is refused e-mails out of e-mails sent, because some models wrote to a given kind of supplier only a handful of times. LFM2.5 is reported but not ranked, for the reason Section 5.2 gives.
Figure 4: E-mails that the supplier could not read, by the language that supplier writes in. Three of the four never fail in English, and gpt-oss fails 23% of its English e-mails. The three ranked models all get much worse in Hindi and Hinglish, and counting every e-mail a model sent they fall in the same order here as they do by profit. LFM2.5’s bars are drawn open and hatched because it is reported and not ranked, for the reason Table 7 gives.
Figure 5: Live-customer outcomes by the language the customer speaks: the share of served customers who bought, and how close the agreed price came to the best price available (bargaining efficiency).
Figure 6: Where the shop’s routine demand went, per simulated day: goods a customer wanted but the shelf did not have, goods priced above what the customer would pay, and stock that spoiled or was lost at the door.
Figure 14: A single simulated day (Qwen3.8-27B, seed 1, day 5). Each bar is a turn by one of the four roles, as long as the minutes that turn cost. Marks below the lanes are live customers at the counter, green where they bought and red where they left without buying. Triangles above the lanes are supplier e-mails arriving. Waiting is not drawn, so the manager’s day shows as a thin scatter of messages and decisions, and everything else it knows about the shop reaches it through the three staff lanes below.
Figure 15: How often each kind of failure occurs, per simulated day, on a logarithmic scale. The failures are not spread evenly, and they do not all fall on the same model. Invalid actions and below-cost sales dominate gpt-oss. Wrong sales reports and customers lost to empty shelves dominate LFM2.5, which reports a wrong figure on 27% of days against gpt-oss’s 6%. Overdue bills are left out of the chart, being a share of days and not a count per day. That share is 0.02 for Qwen3.8, 0.07 for Gemma 4, 0.20 for gpt-oss and 0.00 for LFM2.5, which stops ordering.
Figure 16: What the shop is made of over the month, for one seed per model: cash in the till, and stock valued at cost. A shop can be short of cash while heavy with stock, which is how several runs ended up unable to pay a bill that had come due.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,497 papers