TradesBench: how good are LLMs at running a simulated home services business?

TL;DR: TradesBench is an attempt at benchmarking models on long horizon decision making. We ran five models through two weeks of a simulated home services business. All five made a profit, but none matched the oracle, a simple script that follows good practices; the best model finished about $6,400 behind it. What stood out:

• Models note down their mistakes, not all learn from them. After a technician arrived late once, Opus started checking travel time and job duration before accepting any booking. It never missed another appointment. Argon and Muse made similar rules but still scheduled jobs too close together. Argon missed seven appointments, and Muse missed five.
• Models fail to understand long-term effects. None of the models realized that doing a good job leads to more customer calls over time, while missing appointments leads to fewer calls.
• Models make decisions based on too little information. Opus decided that late installation times were a bad idea after just three customers declined. Argon stopped offering weekend work after only two customers declined. This decision hurt Argon's performance and made it much worse than the oracle (the ideal model).

Task

The LLM has to operate Sunridge Services, a small HVAC, plumbing, and electrical home services company with four technicians. Over two weeks, a well-run company gets around 250 calls for new work. For each call, the agent must decide whether to take the job and, if so, choose the technician, the day, and a two-hour arrival window. The customer gets one offer and either accepts or declines it.

Each technician's drive is simulated through the real Phoenix road network and weather conditions to estimate drive time. If work has not started by the end of a job's arrival window, the appointment is missed.

What makes this challenging?

• Taking on work carries risk. Each extra job makes a technician's schedule tighter. Job lengths vary. Hence a packed day may lead to missed appointments, which lose those customers and reduce future demand.
• Turning work away costs money too. Wages, on-call pay, and overhead are deducted whether the technicians work or not.
• Good judgement matters. An HVAC installation, for example, invoices $4,500 to $15,000 and takes 2 to 8 hours on site, but only one technician, Diana Chen, can do it, and only on weekdays. Filling her day with small service calls can leave no room for it.
• Demand is unpredictable. Weather affects the demand and technicians' travel time. Customers decline offers they don't like and may cancel jobs they booked. How many calls come in depends on how well the company served its earlier customers.

The agent operates the business through an HTTP API and SQL queries. Its workspace includes documentation for both, plus a short company handbook.

Details about how the simulation works are in Appendix A. Details about the agent harness, system prompt, and handbook are in Appendix B. Source code is available on GitHub.

Repeatability

TradesBench is repeatable. Every uncertain event, such as whether a customer accepts an offered slot or later cancels, is determined by hashing the run's seed with the identity of that event (A.1). Therefore:

• For the same seed, the same actions lead to the same world state. Any run can be replayed, audited, or resumed from a checkpoint (the entire world state is one SQLite file).
• The same actions cannot get different random draws. When two models make the same offer to the same customer on the same seed, the customer gives the same answer. How long a job takes is fixed by the job. Differences in the final score are differences in the model's judgement.

Scoring

Every model starts with $25,000. At the end of the 14-day simulation, profit is final cash minus that starting cash. Bankruptcy ends the run.

Why $25,000? It is just enough to kickstart the business, but it does not cover two weeks of fixed costs on its own, so the company has to earn its way. Those costs are $2,916 on a weekday ($1,520 in base pay for the four technicians, $146 in evening on-call pay, and $1,250 in operating overhead) and $2,126 on a weekend day ($876 in on-call pay for two technicians over 12 hours, plus overhead). Over the two weeks that comes to $37,664, so a model has to earn more than that to show any profit.

The figure is small enough that doing nothing bankrupts the business by day 10, and large enough that a good operator can grow the business.

Analysis

We tested five models using the pi agent harness described in Appendix B: Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, and Muse Wonder each at xhigh thinking effort, and Gemini 4 Argon at high. We ran each model multiple times on seed 5, and the published results are the best run of each model.

Setup

The simulated window is October 1 to 14, 2026. Sunridge Services has four technicians. Marcus Vega does HVAC service. Diana Chen does HVAC service and installation. Roberto Salas is the plumber and Katherine Osei the electrician. All four work 7 AM to 5 PM on weekdays. Marcus and Roberto are also on call for off-hours work: the 5 to 7 PM evening window on weekdays and 7 AM to 7 PM on weekends.

At the start the agent's workspace folder contains two things: a docs/ folder with the reference for every API operation and SQL view, and HANDBOOK.md, a short company handbook (B.3), and an empty MEMORY.md. The Pi harness gives the model four tools: read, bash, edit, and write.

Each simulated day starts a fresh conversation with the same system prompt (B.2). At 6:45 AM the model gets an opening prompt: "The office opens at 07:00. No call is active." Each new-job call arrives as its own prompt with the caller's name, phone, address, customer type, requested service, and whether it is urgent. The model has to resolve each call by booking, rejecting, or, if it is a cancellation call, acknowledging it. Once the last job of the day finishes, a closing prompt lets the model review the day and update MEMORY.md.

Results

Each row is the model's best run on seed 5. Profit is final cash minus starting cash. Hours is the sum of model turn times in the run. Tokens is the total processed by the API for the run, including cache reads and writes. Cost is the API bill for the run.

The oracle is a script that follows good practices of running a typical home service business, with the same information as the LLMs (A.8 has the details). It is a reference, not the most optimal way to run the business. The accept-all baseline books every call into the first open window, including work the company doesn't do. The reject-heavy baseline rejects 80% of calls at random and books the rest like accept-all. The do-nothing baseline never answers a call while payroll and overhead get deducted every day.

We have published the transcripts of each model's best run on Hugging Face.

The notes below come from analysing the published transcripts.

Missed appointments

A missed appointment loses the customer and reduces future demand, so the best runs were careful about dispatching: they booked a job only when the technician could reach it in time, allowing for the drive and for how long the job before it might take. Opus missed one appointment, on day two: it booked a plumbing job behind a job across town that ran long. It made that mistake a rule and then wrote a small script to check each booking. The script replays the technician's day many times with different job lengths and drive times, and counts how often a later job would start after its window closes. Opus then booked only when that risk was low. It missed nothing in the remaining 12 days.

Astra missed three, each time after booking one more job onto a schedule that was already tight. Sol missed two. Argon missed seven: it wrote itself a rule not to book back-to-back jobs more than 18 miles apart, then broke it, and five of its misses came in the last three days. Muse missed five by planning each chain of jobs on typical lengths, and kept doing it after writing the lesson down. The oracle missed five.

Muse finished last mainly because of its misses. Two of them came in the first two days and kept its calls low for the rest of the run: it received 170 new-job calls, against 270 for Opus.

Installations

Installations are the most valuable jobs, and most of the gap to the oracle came from them. The models fell behind on days when Diana (the HVAC installation technician) did no installation or a customer declined one. Customers are half as likely to accept an installation window that starts at 3 PM or later, because that is too late in the day for a long installation job. Opus inferred it from three declines and from then on mostly offered installation windows before 1 PM. Astra never noticed it: three of its four declined installations were 3 PM offers.

Astra, Argon, and Opus finished within $820 of each other, in different ways. Astra completed 11 installations to Opus's 9, while Opus completed more service calls. Argon completed the most installations, 13, and the fewest service calls of the three.

Weekend work

After two customers turned down Saturday windows, Argon decided that routine callers refuse weekend work and mostly stopped offering it. It completed 6 weekend jobs, against 23 for Opus. Most of its gap to the oracle came from the two weekends; on weekdays it was close to the oracle.

Workspace and MEMORY.md

Besides writing simple scripts to call the API, each model organised its workspace differently:

• Opus built the most helper scripts. Besides scripts that show everything about a call and the day's schedule, it wrote tools that learn from its own run: one counts how often customers accepted offers by how far ahead they were, one measures how long each type of job actually takes, and one replays a technician's day to estimate the risk of a miss.
• Astra and Argon turned a whole booking into one command: register the customer and the location, then book. Argon's tool also prints an overview of the company at the start of each call.
• Opus, Astra, and Sol logged every change they made to a file, so they could check what they had already done.
• Sol and Muse kept to a small API helper and planned each booking by hand.

Each day starts a fresh conversation, so MEMORY.md is how the models remember the run. Opus kept it as a short guide: a morning checklist, its current strategy, and a table of daily results. At the end of each day it moved the day's detailed log into a separate archive file. Astra and Sol did the same, saving each day's details into dated files and keeping MEMORY.md as a summary. Both also started writing their notes with the spaces removed, apparently to keep them short. Argon kept one short file that reads like a company handbook, with the margin on each job type and notes on each technician. Muse added to the same file all run without cleaning it up, and by the end it was 411 lines long.

Conclusion

Current models are good at making decisions that have immediate consequences. They struggle at connecting their current choices to results that show up days later and at sticking to what they have learned. We expect long-horizon agents to improve as models get better at planning and learning from their past decisions.

As is the case with every benchmark, TradesBench may also have bugs we may have missed. We will update the benchmark and this report as we find and fix them.

A.1 One hash per event

To decide whether an event with probability p happens, the engine hashes the seed and event identity, reads the first 64 bits as a whole number u, and checks whether u is below p x 2⁶⁴. This design has two useful properties:

• Isolation. Each event's number depends only on the seed and that event's identity. Booking one extra job cannot shuffle the luck of every event after it.
• Only whole numbers are compared. The engine converts each probability to a whole number threshold once. This avoids small rounding differences, so the same run produces the same results everywhere.

A.2 Will the customer say yes?

Each call gets one offer. The customer accepts or declines. The rules differ by urgency:

• Urgent callers accept any window on the same calendar day. They accept a next-day window 15% of the time and anything later 5% of the time.
• Routine callers start at 90% and become less likely to accept the longer they wait, down to a floor of 25%.
• Installation customers are routine callers. A window that starts at 3 PM or later halves their chance of accepting. Installations are never booked in the evening or at weekends.

Here lead_days is the time from the call to the start of the offered window, in days. A routine caller offered a window three days out accepts 57% of the time. A week out, 31%.

The handbook does not give these numbers. The agent has to learn them from what happens to its offers.

A.3 Will they cancel?

Once booked, an appointment may still fall through. The chance of cancellation starts at a different rate for each customer type and rises by the same amount for each day of waiting. Urgent jobs and storm days then adjust that rate. In the formula, lead_days is the number of days between booking and the appointment:

Residential customers cancel most often, with an 8% starting rate. Apartment and property-management customers start at 6%, commercial customers at 5%. A residential job booked a week ahead has roughly a 14% chance of cancellation. If a storm hits on the service day, that rises to nearly 20%.

A positive draw is not the end of it. The customer still has to phone in during calling hours, 7 AM to 7 PM, and at least an hour before the appointment, and the agent acknowledges the cancellation.

A later window is less likely to be accepted and, if accepted, more likely to be cancelled. Booking too far ahead can therefore reduce revenue.

A.4 Demand

The company has one hidden demand level: the number of calls it gets in a weekday. It starts at 18 and can never go above 30. It changes each day based on the previous day's appointments:

Every job done on time adds a little. Every missed or failed visit cuts it sharply, and that customer never calls again. A bad first day can push the level to zero. Rejecting a caller costs only that job; it has no effect on future demand. Cancellations also have no effect.

Weekends get 70% of the level, with a minimum of 10 calls while the level is at least 10. Each day's calls include 2 to 5 calls for work the company doesn't do and 0 to 2 installation requests. These counts depend only on the seed and the date, so every agent on the same seed gets the same ones. The rest are service calls for HVAC, plumbing, and electrical. About a quarter of service calls come from existing customers who have not been lost.

A.5 Weather

Phoenix weather is simulated rather than fixed in advance, for each of ten service zones. Daily highs follow the seasons, averaging 87°F across the year and peaking in summer. Day-to-day variation tends to move back toward the seasonal average. Storms are rare outside the summer monsoon. Weather affects the business in three ways:

• Call mix. Heat brings more HVAC calls. Storms bring more electrical and plumbing calls. Weather also changes how many calls are urgent. It does not change the total number of calls; that is set by the demand level.
• Travel time stretches in bad weather (rain 1.15x, storms 1.4x), so a tightly packed route may lead to missed appointments.
• Cancellations rise 1.4x on storm days, as A.3 showed.

The agent can ask for the weather today and a forecast up to seven days ahead. Forecasts get less accurate the further out they are.

A.6 Roads and drive time

The map is built from a pinned OpenStreetMapOpenStreetMap. Map data from OpenStreetMap contributors, available under the Open Database License. extract of Arizona supplied by GeofabrikGeofabrik publishes daily regional extracts of OpenStreetMap data.. The shipped artifact contains 1,136,222 address records across 10 Phoenix-area service zones. A road-routing engine called OSRMOpen Source Routing Machine (OSRM), a routing engine for OpenStreetMap road networks. connects addresses to the driving network. To keep the live simulation fast, the build reduces that network to 2,116 anchor points and precomputes road distances between them. Each usable address stores its road distance to a nearby anchor.

The engine calculates travel time during the run. It turns the approximate road distance between two addresses into minutes using the speed for that time of day and whether it is a weekday or weekend. It then applies the weather in the destination zone and a small, repeatable variation for the trip. Clear and cool weather leave drive time unchanged; hot weather adds 2%, extreme heat 8%, wind 5%, rain 15%, and storms 40%.

A.7 Arrival windows and routes

There are six two-hour arrival windows a day: 7-9, 9-11, 11-1, 1-3, and 3-5 are regular weekday windows, and 5-7 is the weekday evening window. The evening window and every window on Saturday and Sunday are off-hours, and only the two on-call technicians, Marcus and Roberto, can take them. Every job, including an installation, is booked into one window, and there is no limit on how many jobs a technician can hold in a window.

Each technician's day is worked as a route. They leave home for the first job, then drive straight from each finished job to the next. They take the earliest window first and, within a window, the nearest job first. Work never starts before the window opens. If work has not started by the time the window closes, the appointment is missed. The booking API does not reject a booking because the route is too tight; it is up to the agent to leave enough time. Rain or a storm can turn a route that would fit in clear weather into a missed appointment.

Job length is drawn when work starts, from a range for each job type (shortest / typical / longest):

The draw depends only on the job. The same job takes the same time whichever day, window, or technician it is booked with. A failed visit (work the company doesn't do, or a technician without the right qualification) takes about 30 minutes on site plus the drive, and earns nothing.

A.8 The oracle

The oracle is a script with no LLM involved. Despite the name, it has no hidden information: it uses only the public API, the SQL views, and the numbers in the handbook. For each call it does the following:

1. If the caller is cancelling, acknowledge it. If the requested work is not one of the four job types the company does, reject the call.
2. Otherwise, for every qualified technician who is on shift, and every window in the next 14 days, plan that technician's day with the new job added. The plan follows the same route rules as the simulation (A.7): start from home (or from a job in progress), then each window in order, nearest job first. Drive time is estimated from straight-line distance, times 1.3, at 40 km/h. Each job's length is planned at the 0.65 quantile of its handbook range, treated as a triangle from shortest through typical to longest. In other words, it assumes jobs run somewhat longer than average: about 110 minutes for HVAC service or plumbing, 127 for electrical, and 330 for an installation.
3. Keep a window only if every job on that route would start at least 40 minutes before its window closes. Never book installations in the evening or at weekends.
4. Take the earliest window that fits, then the one that adds the least driving. Routine and urgent calls are treated the same way.
5. If no window fits with the margin, try again with no margin. If still nothing fits, reject the call.

It makes one offer per call and never moves jobs that are already booked.

On seed 5 it finished at +$30,289.56 and its cash never fell below the starting $25,000. It completed 147 jobs, missed 5 appointments, had no failed visits, and had 11 jobs cancelled. Across 23 seeds it never went bankrupt.

This is ordinary dispatching practice. It does not try to learn the acceptance curves, hold room for installations, or reason about future demand. It is not omniscient and we do not claim it is optimal. A model could beat it.

A.9 Money

When a job finishes successfully, the engine adds the invoice amount to company cash and subtracts the job's material cost. Cancelled, missed, and failed jobs produce no revenue.

To price one job, the engine looks up three amounts for that service and customer type: low, usual, and high. It picks a value between low and high, weighted toward the usual amount. Residential HVAC service, for example, uses $150, $300, and $500. It then picks the material percentage for that service; HVAC service uses 15% to 25%. The same job always gets the same invoice and material percentage when replayed.

Costs:

• Base pay. Every technician is paid 10 hours at their hourly wage on every weekday, whether they work or not. Wages are $36 (Marcus, HVAC service), $37 (Roberto, plumbing), $39 (Katherine, electrical), and $40 (Diana, HVAC service and installation) an hour. That is $1,520 a weekday for the team.
• On-call pay. Marcus and Roberto are paid their hourly wage for every on-call hour, 5-7 PM on weekdays and 7 AM to 7 PM at weekends. Together that is $146 each weekday and $876 each weekend day.
• Off-hours work. An evening or weekend job bills the customer a 25% premium on top of the invoice. When it is completed and paid, the technician also gets 25% of the invoice (before the premium), so the premium covers the technician's share.
• Overtime. Work and driving outside the hours a technician is already paid for is paid at 1x their hourly wage. That means after 5 PM on a weekday, or after 7 PM for an on-call technician. For a paid job it is capped at 20% of the invoice.
• Overhead. $1,250 per day.

If cash goes below zero, the company is bankrupt and the run stops immediately.

B.1 Harness

Each model is run through pipi, an open-source coding-agent CLI with read, bash, edit, and write tools, and session management., an open-source agent harness that gives it four tools: read, bash, edit, and write. Context files, extensions, skills, and prompt templates are turned off. The agent's workspace contains the API documentation, the company handbook, and two environment variables with the address and credentials for a local HTTP endpoint.

Rules:

• Every simulated day starts a new conversation with no memory of the previous day. The workspace remains available, so the agent must save anything worth remembering in a file. The system prompt names MEMORY.md as its notes file, but the agent must maintain and open the notes itself.
• The simulation controls time. The agent cannot advance the clock, end the day, or create events. It receives a prompt when the office opens, when a call comes in, and when the day closes.
• Every call needs an answer. An unresolved call gets a reminder. If it is still unresolved after the final reminder, it counts as unanswered, which is treated the same as a rejection.

Each run happens in its own Docker container. The simulation engine and the run database belong to one user. The agent's tools run as a separate, unprivileged user who can read and write only its workspace. Before the run starts, the harness checks that the agent user cannot read the run database, the source code, or other private files, and refuses to start if it can.

B.2 System prompt

You are the owner-operator of Sunridge Services, a home-services company serving the Phoenix area with HVAC, plumbing, and electrical work. You answer the phone, decide what the company takes on, and run the business day to day.

Your score is the company's final cash position at the end of the run. The company goes bankrupt if cash drops below zero, and the run ends there.

You operate the company through the Sunridge Services HTTP API at http://127.0.0.1:$SUNRIDGE_API_PORT, using the bearer token in the environment variable SUNRIDGE_API_TOKEN. Every operation, argument, response shape, and error is documented in the reference under the docs/ directory of your workspace. docs/HANDBOOK.md describes the company's services, scheduling, pay rules, and running costs.

You read the state of the business with read-only SQL over the public views, through the query operation of that API.

Your workspace directory persists for the entire run. Your conversation context does not: each morning begins a fresh conversation that remembers nothing of the previous day's exchanges. Anything you want persisted must be written to a file in the workspace.

MEMORY.md in your workspace is your own notes file, persisted across days.

B.3 Company handbook

The agent's docs/HANDBOOK.md, verbatim, as rendered from the default config:

# Sunridge Services company handbook

## What we do

Sunridge Services serves the Phoenix area with 4 kinds of work: HVAC service, HVAC installation, Plumbing, Electrical.

## Arrival windows

Customers are offered two-hour arrival windows: 07:00-09:00, 09:00-11:00, 11:00-13:00, 13:00-15:00, 15:00-17:00, 17:00-19:00. Every job, installations included, is booked into one window. The window is the promise of when the technician arrives and starts work; typical work durations are listed below, and the work may run past the window.

## How long jobs take

Time on site once work starts, excluding the drive:

- HVAC service: 45 min to 3 h, typically 75 min.
- Plumbing: 45 min to 3 h, typically 75 min.
- Electrical: 45 min to 3½ h, typically 90 min.
- HVAC installation: 2 h to 8 h, typically 5 h.

## Office hours and off-hours

Office hours are the weekday windows 07:00-09:00, 09:00-11:00, 11:00-13:00, 13:00-15:00, 15:00-17:00. The weekday evening window (17:00-19:00) and every weekend window are off-hours.

Two technicians are on call for off-hours work: the HVAC service technician and the plumbing technician.

Every employed technician is paid 10 hours at their hourly wage for every weekday.
On-call technicians are also paid their hourly wage for every on-call hour (17:00-19:00 on weekdays and 07:00-19:00 at weekends).

Overtime (capped at 20% of the job's invoice) is paid at 1x the technician's hourly wage for work and driving outside the hours they are already paid for: after 17:00 on a weekday, or after 19:00 for an on-call technician.

## Running costs and cash

- Operating overhead: $1,250 per day.
- Starting cash: $25,000.

B.4 Simulation engine

The simulator stores the world state in a SQLite file. Every change follows the same path through the engine:

Money is kept in whole cents in an append-only ledger; the cash balance is always derived from it. Bankruptcy is recorded in the same commit as the payment that caused it.

B.5 Example call

Each morning starts with the same prompt: "The office opens at 07:00. No call is active." To book a call, the agent registers the customer and the location, then calls jobs.book with the call, the job type, a service date, a window start time, and a technician. The response says whether the customer accepted. Each call gets one offer: if the customer declines, the call ends.