cs.CLOct 16, 2026

KiranaBench: Can a Language Model Run a Neighbourhood Shop?

Authors: Bhuvan Arora, Ankita Karmakar, Chaitanya Jain, Dilesh Churiwal, Harshit Jain, Shivansh Manish Dubey, Manish Prasad, Raja Piyush, 

Organizations: BITS Pilani, Pilani Campus · University of Minnesota, Twin Cities · BITS Pilani, K. K. Birla Goa Campus

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.

Figures & tables

Figure 1: KiranaBench. The shop has four jobs and one model does all of them (left), each with its own tools and clock. Staff write only to the manager, so the day reaches it through them, and what they say about their own work can be wrong. Code computes every outcome (centre) and the truth those reports are scored against. A separate fixed model gives customers and suppliers their words in English, Hindi or Hinglish (right) and decides nothing.
   How the agent worksWhat the shop models
BenchmarkDomainLang.through staffreports scoredcounterparts speakcredit on the bookstaff time budget
Vending-Bench ( Backlund and Petersson, 2025 )Vendingen✓✗✓✗✗
E-Commerce Bench ( Fan et al., 2026 )E-commerceen✗✗✓✗✗
MerchantBench ( Shi et al., 2026 )E-commercezh✓ †✗✗✗✗
RetailBench ( Zhang et al., 2026a )Supermarketen✗✗✗✗✗
ERPBench ( Zhang et al., 2026b )Enterpriseen✗✗✗✗✗
Table 1: Continuing business-operation benchmarks. Through staff , the agent reaches part of the business only by instructing another agent. Reports scored , what the staff tell it is checked against the record run by run, so the figure belongs to the model and not the benchmark. Counterparts speak , customers and suppliers answer in free text. Credit , informal lending is a scored decision. Staff time , an action costs working minutes. † MerchantBench’s scaffold has a delegate_task tool, and its own environment is single-agent.
Figure 2: Profit by simulated day, averaged over the three seeds, one line per model. The scripted shops bracket them: grey is the reference shopkeeper, black the shop nobody manages, which loses its running costs at a near-constant rate. Solvent runs are flat early, because goods ordered first are paid for before being sold. LFM2.5 tracks the unmanaged floor once it abandons the shop, while gpt-oss falls well below it by continuing to trade. Figure 3 draws every run separately.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

ModelS0S1S2Mean
Qwen3.8-27B++ 3,829++ 28,540++ 8,681++ 13,683
Gemma 4 31B++ 37,226−- 40,762−- 7,656−- 3,730
gpt-oss-20b−- 1,43,967−- 2,56,772−- 1,79,495−- 1,93,411
LFM2.5-8B−- 1,20,840−- 1,38,379−- 1,20,443−- 1,26,554
Reference++ 1,05,740++ 1,06,825++ 97,736++ 1,03,434
No manager−- 1,13,610−- 1,17,775−- 1,13,601−- 1,14,995
Appendix
Table 2: Profit in rupees over 30 simulated days, the closing net worth less the opening one, per model and seed (S0–S2). A seed fixes the shop, its customers and its suppliers, so every model faces the same month. The last two rows are scripted shops, not models: Reference is the sensible shopkeeper and No manager is the shop left to run itself.
Scripted shopWhat it doesProfit
Reference shopkeeperrestocks daily, prices to plan, serves customers fairly++ Rs 94,919
Unfair at the countersame stock, but charges MRP and refuses credit++ Rs 65,156
Unfair everywherealso prices the shelves at MRP−- Rs 36,101
Nobody managesno orders, no prices; the staff alone−- Rs 1,16,295
Appendix
Table 3: The score scale, measured before any model was run: four scripted shops over the same 30 days, averaged over five seeds (no language model is involved). Sensible behaviour earns money and careless behaviour loses it, by wide margins, so a model’s profit can be read against these anchors.
Figure 3: Profit by simulated day for every run, line colour denoting the model and line style the seed. Both scripted shops are drawn on each seed too, so the spread between months is visible on the anchors as well as on the models.
MeasureQwen3.8-27BGemma 4 31Bgpt-oss-20bLFM2.5-8B
Sales reported exactly (%) ↑\uparrow1001009473
Credit decisions correct (%) ↑\uparrow84848279
Customers served fairly (%) ↑\uparrow9297738
Shelf demand captured (%) ↑\uparrow8988529
Customers lost, empty shelf (/day) ↓\downarrow0.71.616.757.6
Invalid manager actions (/day) ↓\downarrow0.20.511.63.5
Appendix
Table 4: Behaviour the benchmark scores separately from profit, averaged over the three months every model ran. ↑\uparrow higher is better, ↓\downarrow lower is better. These measures separate the two models that keep the shop running from the two that do not, which is the separation profit makes. They do not reproduce the order of profit, because LFM2.5 scores worse on most of them while gpt-oss loses more money.
RoleWhat it decidesCostliest calls (min)
ManagerSuppliers, prices, the shelf plan, when to go homeread_messages 5, check_books 5
CashierThe bill, bargaining, credit on the khatacheck_khata 2, report_sales 1
HelperWhat to accept from a van, stock between godown and shelfcount_godown 25, take_delivery 5
MunshiThe nightly report on sales, dues and payablesfile_gst 15, count_till 10
Appendix
Table 5: The four roles one model fills, and what their calls cost in simulated minutes. Appendix E lists every tool.
Role and turnTools (minutes)
Managersend_message (3), read_messages (5), check_books (5), read_cash_book (3), set_price (3), write_note (2), read_notes (3), sleep , go_home
Cashier, countercheck_khata (2), say , decide_khata , message_manager (1)
Cashier, closingread_register (2), read_cash_book (2), report_sales (1)
Helper, doortake_delivery (5), refuse_delivery (2), message_manager (2)
Helper, floorlook_at_shelves (2), count_godown (25), set_plan (1), move_to_shelf , move_to_godown , done
Munshi, booksread_register (3), count_till (10), read_cash_book (3), read_khata (3), read_payables (3), read_gst (3), file_gst (15)
Appendix
Table 6: Tools by role, with the simulated minutes each costs. A tool is offered only in the turn where it applies, so the helper cannot take a delivery except at the door.
ModelEnglishHindiHinglish
Qwen3.8-27B0% 0/21733% 1/319% 15/77
Gemma 4 31B0% 0/35042% 10/2438% 19/50
gpt-oss-20b23% 1119/491151% 1383/270659% 4327/7305
LFM2.5-8B †0% 0/459100% 199/199100% 8/8
Appendix
Table 7: Share of supplier e-mails that draw a reply saying the message could not be read, by the language that supplier writes in. The count in brackets is refused e-mails out of e-mails sent, because some models wrote to a given kind of supplier only a handful of times. †\dagger LFM2.5 is reported but not ranked, for the reason Section 5.2 gives.
Figure 4: E-mails that the supplier could not read, by the language that supplier writes in. Three of the four never fail in English, and gpt-oss fails 23% of its English e-mails. The three ranked models all get much worse in Hindi and Hinglish, and counting every e-mail a model sent they fall in the same order here as they do by profit. †\dagger LFM2.5’s bars are drawn open and hatched because it is reported and not ranked, for the reason Table 7 gives.
Figure 5: Live-customer outcomes by the language the customer speaks: the share of served customers who bought, and how close the agreed price came to the best price available (bargaining efficiency).
Figure 6: Where the shop’s routine demand went, per simulated day: goods a customer wanted but the shelf did not have, goods priced above what the customer would pay, and stock that spoiled or was lost at the door.
Figure 14: A single simulated day (Qwen3.8-27B, seed 1, day 5). Each bar is a turn by one of the four roles, as long as the minutes that turn cost. Marks below the lanes are live customers at the counter, green where they bought and red where they left without buying. Triangles above the lanes are supplier e-mails arriving. Waiting is not drawn, so the manager’s day shows as a thin scatter of messages and decisions, and everything else it knows about the shop reaches it through the three staff lanes below.
Figure 15: How often each kind of failure occurs, per simulated day, on a logarithmic scale. The failures are not spread evenly, and they do not all fall on the same model. Invalid actions and below-cost sales dominate gpt-oss. Wrong sales reports and customers lost to empty shelves dominate LFM2.5, which reports a wrong figure on 27% of days against gpt-oss’s 6%. Overdue bills are left out of the chart, being a share of days and not a count per day. That share is 0.02 for Qwen3.8, 0.07 for Gemma 4, 0.20 for gpt-oss and 0.00 for LFM2.5, which stops ordering.
Figure 16: What the shop is made of over the month, for one seed per model: cash in the till, and stock valued at cost. A shop can be short of cash while heavy with stock, which is how several runs ended up unable to pay a bill that had come due.
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

Explore similar work

CardsList
Aug 9, 2026

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce Business Arena, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.

Yijun Pan, Yukun Lian, Kunyu Shi +5

1Accio Team, Alibaba Group · 2Yale University

LLM Agent EvaluationAI Agent BenchmarksE-CommerceLong-Horizon LLM Agents
AskPDF
Jun 14, 2026

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observable decision process and is designed to support thousand-day-scale simulations. In this environment, agents must manage pricing, replenishment, supplier selection, shelf assortment, inventory aging, customer feedback, external events, and cash-flow constraints. We evaluate seven contemporary LLMs under representative agent frameworks over a 180-day evaluation horizon and compare them with a privileged oracle policy. Results show substantial variation across models: only a small subset survives the full evaluation horizon, and even the strongest LLM runs remain substantially behind the oracle policy in final net worth and sales outcomes. Behavioral analysis attributes these gaps to incomplete evidence acquisition, surface-level decision making, and the lack of a consistent long-horizon policy. RetailBench provides a controlled testbed for studying reliable autonomy in economically grounded long-horizon decision-making.

Linghua Zhang, Jun Wang, Jingtong Wu +1

Ant Group · City University of Hong Kong

Long-Horizon Agent EvaluationAI Agent BenchmarksLLM Decision-Making
AskPDF
Aug 31, 2026

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. Our code is available at https://github.com/QwenLM/E-CommerceBench.

Wei Fan, Xinjie Shen, Xudong Guo +8

1Qwen Team, Alibaba Group · Department of Computer Science and Engineering, HKUST · 2Taobao & Tmall Group, Alibaba Group

Long-Horizon Agent EvaluationAI Agent BenchmarksLong-Horizon Agent TasksAgentic Commerce
AskPDF