Skip to navigation

Evaluating AI Coworkers

View as Markdown

Evals is pre-production testing for an AI Coworker such as Atom, AccessOps, CloudOps, or DeviceOps. Evals test the coworker’s latest draft, so you do not need to publish the coworker first. You add scenarios that represent work you care about, run them, and review whether the coworker passed or failed each behavior on the scenario.

Open a coworker from AI Workforce > AI Coworkers, then select the Evals tab.

The Evals tab of an AccessOps Manager coworker, with a success rate summary and a list of scenarios and results.

What’s in Evals

The Evals header shows when the coworker Last ran. On the right are Add scenarios, Run all, and a clock control that opens Eval history.

A summary row shows:

  • Success rate
  • Passed
  • Failed
  • Total spent

The scenario table has the columns Scenario, Last tested, Behaviors, and Result. Behaviors is a compact bar for the checks on that scenario. Result states captured in the product include Passed, Failed, and Edited, needs rerun. While a run is in progress, a scenario can also show Running or Queued.

Eval history lists previous evals (for example, Eval 1, Eval 2) with the time they ran and how many scenarios passed, so you can switch between them.

Eval history open on the Evals tab, listing Latest, Eval 2, and Eval 1 with pass counts.

Add scenarios

To add scenarios for the coworker, select Add scenarios. You can also select existing scenarios and use Edit with Assist in the selection bar at the bottom of the list.

Run an evaluation

You can run every scenario, or only the ones you select.

  1. To run everything, select Run all.
  2. To run a subset, select the scenarios you want. A bar at the bottom of the list shows N selected, with Run and Edit with Assist. Select Run.

Four scenarios selected on the Evals tab, with a selection bar offering Run and Edit with Assist.

While a run is in progress, the header shows the eval name and a status such as Eval 2 · Testing 4 scenarios, plus progress (0 of 4 done) and Stop. In the table, scenarios that have not finished yet show Running or Queued.

An eval run in progress, with Eval 2 testing four scenarios and results updating to Running or Queued.

When the run finishes, the summary updates — Success rate, Passed, Failed, and Total spent — and Last ran is refreshed. Open a scenario to review the detail.

The Evals summary after a run, showing a 40% success rate, 4 of 10 passed, 6 of 10 failed, and total spent.

Review results

Select a scenario to open its detail. The header shows the scenario title and an overall Passed or Failed badge.

The detail is organised as:

  • Comments and Activity — the conversation and activity for that run, when they were recorded.
  • What this tests — the intent of the scenario.
  • Behaviors — each check for how the coworker should act, marked passed or failed. There is no numeric score. Failed behaviors can include a short explanation. Edit lets you change the behaviors for this scenario.
  • Efficiency — Duration and Cost.

A scenario detail panel with Comments and Activity, What this tests, a Behaviors list with pass marks, and Efficiency.

Behaviors

A behavior is a check on an eval scenario: a plain-language statement of how the AI Coworker should act in that situation. Behaviors are not a separate tab on the coworker page — you view and edit them from a scenario in the Evals tab.

Each scenario can have multiple behaviors. When you run the scenario, each behavior is marked passed or failed. There is no numeric score. Failed behaviors can include a short explanation of what the coworker did instead.

The Behaviors card on an eval scenario, with four passed checks, an Edit control, and Efficiency showing duration and cost.

A Behaviors card where two checks failed, with an explanation under the first failure, and two checks passed. Efficiency is listed below.

Defining and editing behaviors

Open a scenario and select Edit on the Behaviors card to change the behaviors for that scenario. Write each behavior as a clear statement of what the coworker should do — for example, “Posts one public comment that asks only for the business reason, then waits for the reply” or “Reassigns the ticket to an available person in the ticket’s workspace, other than the requester.”

Good behaviors are specific and testable. Avoid vague checks like “handles the request well” — instead, name the exact action or decision you expect.

How behavior drifts power Performance suggestions

When a coworker stops passing a behavior it used to pass, that drift can surface as a suggestion in the Performance tab. This is how the two features connect: behaviors define what “good” looks like, evals test against them, and drifts feed the improvement loop in Performance.

FAQs

  1. When do I use Evals? Evals is pre-production testing. Use it to run scenarios and see how an AI Coworker handles the work you care about, including a pass or fail for each behavior, before you rely on a change in production.

  2. Do I need to publish a coworker before I can run evals? No. Evals test the coworker’s latest draft, so you do not need to publish the coworker first.

  3. How are results judged? Each scenario is judged against its Behaviors. After a run, every behavior is marked passed or failed. Failed behaviors can include a short explanation. The scenario also shows an overall Passed or Failed result, plus Duration and Cost.

  4. Where do I change the behaviors for a scenario? Open the scenario and select Edit on the Behaviors card.

  5. What happens when a behavior starts failing? A behavior that the coworker used to pass but now fails is a drift. That drift can surface as a suggestion in the Performance tab, where you can apply a fix to the coworker’s draft.