How reliably can AI agents do India's work?

Global agent benchmarks test clean, English-only sandboxes. Indian-language leaderboards test whether a model understands Hindi or Tamil. Neither tells you whether an agent can file, reconcile, order or refund correctly on the systems India runs on.

GST and invoicing

Reconcile purchase registers with GSTR-2B, check e-invoices and catch input tax credit errors.Example: find the three invoices where the supplier's GSTIN doesn't match the return.

ONDC commerce

Search, select and order across sellers on the open network, including substitutions and cancellations.Example: a customer asks for "do kilo aloo aur ek Amul butter, kal subah".

UPI payment operations

Triage failed transactions, raise and track disputes and explain refunds to customers.Example: money left the account but the merchant shows unpaid.

Health records

Handle consent requests and record linking on ABDM-style flows without overstepping consent.Example: share only the last two lab reports, for 30 days.

Indian-language voice

Turn spoken requests in Hindi, Hinglish, Tamil, Marathi and more into correct, structured actions.Example: a noisy voice note asking to reschedule a delivery.

• The result, not the reply

  The grader checks what the agent actually did: the order placed, the mismatches flagged, the dispute raised. What it says about it doesn't count.

• Every run, not the best run

  Each task runs several times. Only tasks solved correctly on every run count towards the score.

• Cost per task

  Tokens and tool calls are measured, so you can see what reliability costs with each model.

• Things going wrong

  Timeouts, out-of-stock items and malformed records are injected on purpose. Real systems fail; agents have to cope.

• Hidden test set

  A public set lets anyone try the tasks. A private set decides the scores, so results can't be tuned to known answers.

• Simulated, not live

  Tasks run on simulators built from public specifications with synthetic data. No real payment, filing or health record is touched.

• AI labs and model builders who want to show their models handle real tool use in India, not just benchmarks in English.
• Companies choosing AI agents for finance, commerce, payments or support, and needing evidence before a pilot.
• Researchers studying agent reliability in multilingual, messy, real-world conditions.

What does AIBench measure?

Whether AI models and agents can complete real Indian business tasks correctly, every time, and at what cost. Tasks cover GST reconciliation, ONDC commerce, UPI payment operations, ABDM health-record consent and requests spoken in Indian languages.

Does AIBench use real UPI, GST or health systems?

No. Tasks run against simulators built from public specifications, with synthetic data. No real payments, tax filings or health records are touched.

How is this different from Indian-language LLM leaderboards?

Indian-language leaderboards test whether a model understands a language. AIBench tests whether an agent can act: call the right systems, follow the rules and finish the task.

Is the benchmark open?

The methodology and a public task set will be open. A hidden test set stays private so scores reflect real ability rather than tuning to known answers.

When are the first results?

The first task tracks are being built now. Leave your email below and you'll get the first report as soon as it's published.

Can my organisation contribute tasks?

Yes. If your team runs a workflow that AI agents should be tested on, choose "Company choosing AI agents" or "Researcher" below and mention your organisation. We'll be in touch.