Custom evals, built with your team

Our forward-deployed engineers embed with your team and build your evals on our platform, then add new tests every week as your product changes.

Your eval01 / 05

How we work

Five layers,
each built from your workload

A Newmeasure eval is a working copy of the job your agent does, built by our engineers on our own platform.

  1. We learn the workload your agent automates. We read real requests and traces with your team and map the jobs your customers pay for.

    Endpoints per app

    • CRM58
    • Billing44
    • Email38
    • Chat26
    • Docs22
  2. The initial state: accounts, records, history and the mess real companies carry.

    Months of history, thousands of records, real-looking people

    OldestActivityToday

    Stale accounts and duplicates, left in on purpose

  3. What customers actually ask for, written from real requests and traces.

    TaskOpen

    “Refund the duplicate charge and let the customer know in chat.”

    Each task written from a real request

  4. Checks that the agent did the right thing, read from the end state of the software.

    • refund_issued (order)Pass
    • no_duplicate_refund ()Pass
    • customer_notified (chat)Fail

    Read from the end state of the software

  5. The model and harness under test. Swap either one and run again.

    Model

    claude-opus-5.5

    or gpt-6-astra

    Harness

    claude-agent-sdk

    or acme-harness

Ongoing evals

An eval set that grows with your product

Every week, what went wrong in production becomes new tests and every release has to pass them.

W1W6W12

+40 every Monday

Tasks in total

 
140
 
 
 
 
 
 
 
 
 
571

Before you sleep

Release score

61.0%
61.2%
69.4%
v2.4v2.9 rc blockedv3.2
The suite grows from 140 tasks in week 2 to 571 in week 12. Release scores rise from 61.0% (v2.4) to 69.4% (v3.2); the v2.9 release candidate scored 61.2% and was blocked.

Platformized services

Want us to build it? We embed with your team

Our engineers build your evals on Benchmark Router and run them with you.

book a call
Week 024681012
  • Private evals

    A test suite only your team sees.

    BuildLive new task every week
  • Domain benchmarks

    Public benchmark with your name on it.

    BuildLaunch
  • Forward-deployed engineers

    Eval engineers in your team, on our platform.

    Inside your team for a quarter