Benchmarks
Measuring AI progress.
TradesBench
TradesBench is an attempt at benchmarking models on long horizon decision making. Five frontier models each ran a simulated home services business for two weeks. All five turned a profit, but none matched a simple scripted oracle.
Video Arena
Nearly 300 blind votes and over 600 agent builds comparing Remotion, HyperFrames and Motion. On the same model, film length flips the winner: Remotion takes 70% of votes past a minute, while HyperFrames wins between 30 and 60 seconds.
ITSMBench
We worked with the Atomicwork team to build a public benchmark and open-source a suite of environments with 89 tasks that evaluate whether frontier AI models can autonomously resolve real-world IT Service Management tickets.
Looking to publish a public benchmark for your model, harness, or industry? Reach out.
Book a call