Measure and improve a factory

Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it.

Feature	What it tells you
Dashboard metrics	How much work the factory produced, and what it cost.
Scorers	Whether completed conversations meet criteria you define.
Benchmarks	How different configurations perform on the same tasks.
Self-improvement	Which repeated failures get investigated and turned into follow-up work.

Read dashboard metrics

The control room dashboard shows activity, cost, autonomy, and evaluation results:

Metric	What it shows
Total runs	All agent runs, with breakdowns by agent type, status, source, model, and more.
PRs opened	Pull requests created from factory work.
PRs merged	Pull requests that merged.
Autonomy %	The share of merged work that shipped without human edits.
Time saved	An estimate of engineer-hours saved by merged factory PRs.
PR latency	How long pull requests spend in each stage.
Cost per PR	An estimate of credits spent per pull request.
Most expensive PRs	The highest-cost pull requests.
Scorer cards	Results from your Scorers.
Self-improvement PRs	The three newest Self-improvement pull requests, regardless of the selected date range.

Time saved and Cost per PR are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR counts recorded credits and can undercount actual usage.

Caution

PRs merged, PR latency, and the detail in Most expensive PRs require the GitHub App and only cover activity from after you install it.

Use the dashboard to pick which runs to investigate, not to conclude what caused a change. Total runs includes evaluation, benchmark, and Self-improvement runs, so a higher run count with a flat PR count could mean harder tasks, retries, or measurement activity.

Configure Scorers

A Scorer uses an LLM judge to classify completed conversations against criteria you write, such as “did the agent run the tests before opening a PR?” Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix.

Configure these fields:

• Judge instructions - The criteria the judge checks for.
• Classifications - The labels the judge can assign, each with a score.
• Pass threshold - The score a conversation needs to pass.
• Sample rate - The portion of completed conversations to evaluate.
• Judge model - The model that acts as the judge.

You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after a conversation completes, the judge evaluates it and records a classification, a score, and its reasoning.

Mode	Use it when	You get
Manual	You want to score one conversation or test new judge instructions.	An evaluation of the selected conversation.
Periodic	You want an ongoing sample to track over time.	A baseline to compare against after a change.

Changing Pass threshold updates how past scores display as pass or fail; the recorded results don’t change.

Compare configurations with benchmarks

A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes:

• Agent - The agent whose configurations you compare.
• Tasks - Fixed prompts with success criteria.
• Configurations - The harness, model, and runner combinations to test.
• Scorers - Your classification Scorers, applied to every trial.
• Repetitions - The number of trials per task and configuration.

You can create a benchmark task from a completed run’s detail pane, and Warp copies the run’s input into the task. Add success criteria before you launch.

Every benchmark also runs Correctness, a built-in scorer that marks each trial as pass or fail against the task’s success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn’t combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don’t include model usage, so the true cost is higher.

Configure and review Self-improvement

Turn on Self-improvement for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as definitions as code, its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review.

The factory-level Analysis model setting chooses the model used for this analysis. Factories managed from files in a GitHub repo can’t change this setting in the control room.

Each Self-improvement pull request includes a Regressions addressed section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence.

Run a practical improvement loop

Change one measurable thing at a time:

flowchart LR
  Define[Define a Scorer] --> Baseline[Collect a baseline]
  Baseline --> Inspect[Inspect failures]
  Inspect --> Benchmark[Benchmark a candidate]
  Benchmark --> Adopt[Review and adopt]
  Adopt --> Monitor[Keep monitoring]
  Monitor --> Inspect
  Inspect -.->|Repeated failures| Improve[Self-improvement]
  Improve -.-> Adopt

1. Define a Scorer. Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge’s results against a few conversations you review yourself.
2. Collect a baseline. Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
3. Inspect failures. Read the judge’s reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
4. Benchmark a candidate. Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
5. Review and adopt. If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
6. Keep monitoring. Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs.

Next step

Record an adopted change in factory definitions as code so your team can review the factory configuration.