Learning Tactus

Evaluations

BDD specs answer “can the workflow do the right thing?”

Evaluations answer the harder question:

How often does it do the right thing across real inputs?

This is where agent engineering starts to look like ML engineering: you need datasets, metrics, thresholds, and regression tracking.

Specs vs Evals (Use Both)

Think of the split like this:

You want both because they fail differently:

The Evaluation Loop

An evaluation is just a loop with structure:

  1. Define a dataset of realistic inputs (messy notes, edge cases, adversarial-ish phrasing).
  2. Run the procedure on each case (often multiple times).
  3. Compute metrics and compare to thresholds.
  4. Track results over time.

If you’re serious about deploying agent workflows, this becomes non-optional.

Running Evaluations in Tactus

Tactus supports evaluations via an Evaluations({ ... }) block in a .tac file and the tactus eval command:

tactus eval your-procedure.tac

To measure consistency, run each case multiple times:

tactus eval your-procedure.tac --runs 10

Evaluations are complementary to tactus test --runs 10:

Metrics You Can Trust

Start with deterministic, non-model-graded metrics whenever possible:

Model-graded metrics can be useful (readability, tone, faithfulness), but treat them like any other model:

How It Connects to the Running Example

For the recap workflow, a small but effective evaluation dataset would include:

The metrics you care about early are boring—but they prevent incidents:

Once those are stable, you can add higher-level quality metrics (readability, tone, summarization quality).