Evals for AI-native Services Companies

Two attorneys sit down with the same 20 NDAs, and each writes their ideal redlines. Between them they make 500 edits. When they compare notes, they agree on 45%.

That is a real experiment, run by Crosby, an AI company with its own law firm, and described in their recent post on how they do evals. Most teams would read 45% as a dead end: if the experts can't even agree, how could a machine ever be graded? Crosby read it differently. If two experts mostly disagree, there is no answer key to grade against, so they built an internal labeling workbench that converts subject matter expertise from their lawyers into how their product is shaped and how their evals are outlined.

Our team came across the post and it made the rounds internally the same day. We were deeply inspired by it, because it is the approach we believe in and the one we build with our customers. This post is our version of it, written for any AI company.

Two ways to build evals

Once a company decides to get serious about evals, it hits the same fork Crosby did, and there are two ways to go.

The first is to give all the tools to the engineers: the frameworks, agents and harnesses, and have them figure out ways to convert what they've learned about the domain into good-quality evals. Picture what that actually looks like. An engineer spends weeks absorbing a domain the experts spent years learning, writes the tasks and rubrics they think are right, and puts them in front of an expert who reads them and says: that's not what I meant. Something got lost in translation. In evals, translation is the whole game.

So the first way is not an effective approach. Or at least, it is not one that works for companies of all types. Especially not for very domain-specific companies like Crosby, Solstice Health or Dodge AI, where the distance between what an engineer can absorb in a quarter and what an expert already knows is the entire problem.

The second way turns it around. Don't move the expertise to the engineers; move the work to the experts. Give them an environment where they just work, reviewing outputs the way they already do, and have an agent convert that work, their definition of what good output looks like, into a good eval set. The expertise stays where it lives. The system does the translating.

We believe the second way is right. It is what we have been building with our early customers, and the rest of this post is how it actually works.

Who this works for

We find this to be a very effective approach to co-building evals with AI-native services companies (or really anywhere there are subject-matter experts internally). AINS usually understands the workflow immediately: building evals becomes a joint effort between agents, engineers and subject matter experts, not a job for engineers alone.

AI-native services companies are the clearest case. They sell finished work made by AI, and their experts already review everything before it goes out. The review is happening every day. The problem is that it evaporates. Every correction an expert makes teaches the AI nothing, because nobody wrote it down in a form the AI can be graded against.

This is also what we have been seeing great early success with while working with Solstice Health, an AI-native pharma marketing services company.

The machine, end to end

Tasks come from three places. Humans author them. We take inspiration from the company's logs, which carry the requests people actually make, edge cases and all, rather than the requests a whiteboard imagines. And we use semi-synthetic pipelines to create domain-relevant tasks at scale, with a person checking what comes out, because purely synthetic tasks drift away from real work. Most eval sets mix all three.

Those tasks run through the harness the company has built, and we generate what their agent's understanding of the output is. This step matters more than it looks. The eval has to grade the system the company actually ships: the prompts, the tools, the orchestration. A model in isolation tells you about the model. The harness tells you about the product.

Then a subject matter expert reviews the outputs in the portal and gives their judgment as input, in whatever form fits the work: approving, editing, marking up, explaining what is wrong. The format bends to the domain. The judgment is the input that counts.

That judgment gets converted into rubrics: concrete checks that can be applied to every future output without a person in the loop.

id     = "headline_length"
check  = "fail if a headline runs
          8 words or longer;
          no headline = met"
source = "reviewer annotation"

On a recurring basis, engineers take the accumulated tasks and rubrics and approve them into the golden dataset. This gate is deliberate. The golden dataset is the company's written-down definition of good, and it should change on purpose, not by accident.

And it never sits still. Production logs keep surfacing interesting cases, experts keep adding their expertise to the platform, and the dataset grows. An eval set that froze on the day it was approved would fall behind the product it measures. The loop is the product.

A hypothetical walk-through, to make it concrete: a task lands from last week's logs. Draft the follow-up email for a launch, given the brief and the brand rules. The agent produces its version. A reviewer marks the parts that work and the parts that don't. Those marks become checks. Next week's drafts get graded against those checks before anyone on the team sees them, and the checks themselves keep getting tested against what the reviewer actually picked. Six months in, the company is no longer asking "is our AI good?" It is asking "how much closer to the best version are we this week?"

Not every annotation becomes a rubric

There is a subtlety here that decides whether the whole thing works. An intelligent system has to understand which annotations need to be converted into rubrics because they are always true, versus something that is just a one-off case or a customer-specific case and possibly does not have to be in the golden dataset.

Picture a reviewer who marks the same kind of mistake every single day, and then, once, marks something odd that only applies to one unusual client. Those two marks look identical in the portal. They are not. The first is a rubric. The second is a note. Get this wrong in one direction and the eval set fills up with noise until nobody trusts it. Get it wrong in the other and you throw away the expertise the whole system exists to capture. The judgment about judgment is part of the system, not a cleanup step at the end.

What evals should aim at

It is also important to think about the story you are achieving with the evals you write. There is no point focusing on things that are solved by the SOTA models, or on something that is about to be solved; the frontier will eat those evals within a release or two. Your evals should be looking at the absolute best version of your product and agents, and aiming to get you there. If you know what the absolute SOTA looks like, you understand what you need to hill-climb on.

An eval set that only measures yesterday's failures can only tell you to stop. One that measures the gap to the best version of your product tells you where to go next.

No two review portals look the same

Take two reviewers. One is giving feedback on how AI is generating emails. The other is giving feedback on how SAP's code should be configured. They are obviously not looking at the same screen, and they should not be. The review surface has to match the work, and it changes from company to company.

So it is important to build a system of record around the different ways, domains and industries that exist, and each of their individual ways of getting review and feedback from their subject matter experts. This is where doing the work across many companies compounds: having seen this workflow across industries and domains, we inherently have an understanding of what the right dimensions of grading and review should be when a new one comes in.

Crosby ended their post by saying these lessons extend beyond legal. We think that is right, and it is the bet we made: any company shipping expert work through AI has its own version of the 45% problem, and the fix has the same shape. Don't ask experts to become engineers. Let them work, and build the system that turns their work into evals.

If you are building an AI company with real expertise in-house and want evals built this way, email us at founders@newmeasure.ai.