Runs

​

How runs work

A run config defines which agent, test cases, and eval functions to use when executing an eval. When you start a run, IMP schedules one job per test case in the eval_runs queue. Each job replays the test case conversation through the agent, collects the response, and scores it with the configured eval functions. Results appear in the matrix as jobs complete.

​

The results matrix

Open any eval set and select the Results tab. The matrix is a grid where:

• Rows are the test cases that belong to the set.
• Columns are run configs (sorted oldest to newest).
• The Average row pinned at the top of the data area shows the mean score across completed results for that run, colored against the pass threshold.

By default the last three runs are shown. A Show older runs button (left-arrow chevron) in a narrow rail column reveals five more at a time.

​

Score cells

Each cell shows the result for one test case in one run: Click a completed (green/orange/red) cell to open the result detail sheet for that job.

​

Result detail sheet

The sheet has four tabs:

• Overview — score, duration, status badge, job ID, error details (if any), and token usage (input / output / total).
• Messages — the full conversation as rendered messages, read-only.
• Functions — per-eval-function score and configuration.
• Raw — the raw metadata JSON for debugging.

​

Run a new eval

Click Run eval in the page header (or New run config in the overflow menu). The New run config dialog opens: Essentials:

• Name (required) — a label for this run, e.g. “GPT-4o baseline”.
• Agent (required) — which agent to run the cases through.
• Test cases (required) — which cases to include (multi-select from the set’s cases).
• Eval functions (required) — the functions that score each result (multi-select).

Advanced (collapsible; summary shown as e.g. “average · pass ≥ 70 · 300s”):

• Scoring method — Average, Median, or Sum.
• Pass threshold (0–100) — scores at or above this value are colored green.
• Timeout (seconds) — per-job timeout before the worker marks the job failed.

Click Create run config. The new column appears in the matrix. To execute it, click Start run from the column’s overflow menu (⋯).

​

Column overflow menu

Each run column has a ⋯ menu (write access only for mutating actions): Deleting a run shows a warning: “Already-scheduled queue jobs are not removed. Check the queue panel to manage them.”

​

Queue management

A Queue chip appears above the matrix (right-aligned) when there are jobs in any state. The chip shows “Queue: N active · M failed”. Click it to open the Queue sheet. The queue sheet shows:

• Status — Active / Paused / Maxed.
• Concurrency — max queue and worker concurrency, job timeout, rate limit.
• Status tabs — Active, Waiting, Failed, Completed. Each tab shows a count badge.
• Jobs table — name, ID, attempts, created/processed/finished timestamps, inputs, outputs, per-row actions.

Write-access actions in the queue panel: The retry dialog includes an optional “Delete the original job(s) after retrying” checkbox.

​

Next steps