Verification, testing and evaluation: building trust at scale

7.1. Two different things need to be evaluated

The software produced needs to meet requirements and preserve important properties. The agent-based process needs to solve tasks consistently, with acceptable cost and intervention.

One run might produce a correct change after many attempts and hours of supervision. That validates the result but leaves a problem in the process. Another run might be fast and elegant and produce a wrong change. That is why software tests and evaluations of the agent system complement each other.

Anthropic's guide to evals distinguishes task, trial, trajectory, and final state, and recommends evaluating the combination of model and harness. It also stresses the need for multiple runs and for calibrating model-based graders. F20

7.2. Defining the correctness oracle

An oracle is the criterion used to decide whether a result is correct. It can be a mathematical rule, a contract, a data constraint, a reference behavior, or a structured human evaluation.

The engineer should identify where that criterion comes from. If implementation and tests are derived from the same incomplete interpretation, both can agree and still fail the real requirement. Asking another agent for a second opinion helps with exploration, but the strongest independence comes from evidence with different origins.

Example: an export endpoint should only include data from the authenticated customer. A test that checks “returned a CSV” covers a very small property. A scenario with two customers, similar data, and attempts to cross identifiers tests the authorization boundary that really matters.

Operational question: what relevant error would go unnoticed if this test were the only quality signal?

7.3. Choosing the form of evidence

The table below is a decision proposal, to be adapted to the risk and the domain.

Properties and metamorphic relations can be particularly useful. In a lossless transformation, transforming and then reversing should preserve the data. In a search, adding a conjunctive filter to the same set, before pagination, should not enlarge the result set. The value comes from the relation tested and its conditions of validity.

7.4. The economics of testing need to be reconsidered

The manifesto observes that the lower cost of writing and maintaining tests may change earlier decisions about coverage. The competent application is to reassess the total cost of each strategy.

If generating tests has become cheap, you still have to pay for running them, investigating flaky failures, and maintaining environments. The team should expand evidence where that reduces risk or human work, while preserving fast feedback. It may be worth producing many more contract tests, adversarial cases, and critical journeys, as long as the result remains interpretable.

In the compiler experiment with parallel agents, Nicholas Carlini reports that his work concentrated on the environment, the verifiers, and the feedback; external references helped isolate problems for independent execution. It is a capability experiment, with limitations acknowledged by the author, and it illustrates the value of good verification conditions. F22

7.5. Human review changes its distribution

There is room to automate style checks, repetitive policies, local consistency, and many routine reviews. Human investment should migrate toward requirements, trust boundaries, changes without good evidence, and decisions with greater consequences.

This makes it possible to set policies by class of change. Updating documentation or performing a mechanically verifiable refactoring may warrant a different flow than modifying authorization or deleting data. Approval should correspond to the risk and to the quality of the controls available.

To reduce manual review sustainably, the team needs to know which defects its mechanisms detect, track the failures that slip through, and have recovery capability. The practical goal is to eliminate the need for repetitive inspection in well-controlled classes and concentrate judgment on the exceptions.

7.6. Building a small evaluation suite for the AI workflow

An initial proposal is to select representative real tasks: a localized fix, a change that crosses modules, an investigation, an interface change, a migration, and a case with an ambiguous requirement. There is no need to start with a large benchmark.

For each task, record: initial state, expected result, evaluation criteria, code version, agent configuration, time, cost, and interventions. Each attempt should start from a known, restored state, to avoid contamination by data, caches, or effects of a previous run. Include tasks that failed previously. Hold out some cases to evaluate changes without tuning the process directly to them.

Compare configurations under sufficiently similar conditions and repeat runs when variability could change the decision. Measure correct outcomes, cost, human time, regressions, and out-of-scope actions. A single favorable demonstration is a starting point for investigation.

The success narrated by the agent should be confirmed in the relevant artifact or state: file, tests, database, application, or deployment. The final record needs to distinguish checks that were run, checks that were impossible, and hypotheses still open.