---
title: Review and backtest a change
description: See what a proposed pack change would do to real recent decisions, and which agents it would reach, before anyone approves it.
sidebar:
  order: 4
---

An approver needs to see consequences, not diffs. A **backtest** re-decides your agents' recent real cases under a proposal and under the current release, and tells you which decisions would change.

## Run a backtest

In ChatGPT or Claude with the Felspar plugin, after `propose_pack`:

> What would this proposal change? Backtest it.

The `reviewing-pack-changes` skill calls `backtest` with the proposal ID, then writes a short summary. Through REST, call `POST /v0/proposals/{proposalId}/backtest`.

| Input | Meaning |
| --- | --- |
| `proposal_id` | The proposal from `propose_pack` |
| `limit` | How many saved cases, newest first. 1 to 50, default 25. |
| `execution_ids` | Only these saved cases (up to 50) |
| `reinterpret` | `true` to re-classify the excerpts whose definitions changed. Run without it first to see the estimate. |

## What it reports

Each saved case comes back as one of:

| Result | Meaning |
| --- | --- |
| `unchanged` | Same outcome under the proposal |
| `changed` | Different outcome: current to proposed |
| `unresolved` | Still undecided under the proposal, with the questions still open and why |
| `needs_reinterpretation` | A definition this case's evidence was classified against changed, so its text must be read again. Comes with an estimate of the work. |
| `unavailable` | This case can't be re-decided (for example, an older record format) |

Each case also shows both sides' escalation, and the report counts `escalationChanged`: cases whose hand-off would differ, even when the outcome doesn't. A scored pack shows each case's total on both sides too.

It also lists the agents that ran those cases, and the agents following the job: on **Latest** (they will get the release once it's approved and released) or **pinned** to one of its releases (they won't).

## Breaking changes

Code that acts on Felspar's results may map escalation IDs to queues, or read subject types and relationship names. When a proposal removes one of those, or changes the job's subject, the report carries `breaking`:

```json
"breaking": {
  "changes": ["removes escalation compliance-review", "removes relationship contract.matter"],
  "agents": 1
}
```

`agents` is how many agents run or follow the job. `check_pack` and `propose_pack` return the same list, so the author sees it before the backtest. A breaking change isn't refused; it is something the approver and the owners of those agents must know before release.

## How it re-decides

- Each case is decided at its own original time, so evidence freshness is judged as it was.
- Numbers and dates are re-read exactly from the saved text, at no cost.
- A classified answer is reused only when the classifier would be asked exactly the same thing under the proposal: same excerpt, same question definition, same answers, same instructions.
- Anything else needs a new answer. By default no model is called; the case is reported as `needs_reinterpretation` with an estimate of how many excerpts and requests it would take.
- With `reinterpret: true`, only those excerpts are classified, within a job's usual limits per case. Backtests draw on their own daily allowance, separate from the one your agents' decisions use, so a backtest can't slow production decisions.

## What it doesn't do

A backtest writes nothing. It creates no release, receipt or decision, changes no past decision, and its report is returned, not stored. It covers only past cases: it doesn't predict volume or every future case.

Cases come from the last 30 days of real decisions on the job (decisions are kept for 30 days). Cases whose evidence can't be re-read are reported as such, never guessed.

## Who can backtest

Members, approvers and owners, with their own current access to every connected source behind the proposal. Agent keys can't. A backtest counts as 10 decisions toward [rate limits](/reference/limits).

## Writing the review summary

The `reviewing-pack-changes` skill produces a summary for the approver:

1. **The change:** which proposal, against which release, and which passages of the Doc changed.
2. **Counts:** changed, unchanged, unresolved.
3. **Why decisions flip,** grouped by cause ("all 14 flipped because the size floor moved from 200 to 100"), with two or three examples.
4. **Meaning changes:** if a definition's wording changed, how many cases need their text re-read, and the choice to run that.
5. **Escalations and breaking changes:** how many cases would hand off differently, and every `breaking` change, with the agents that depend on the job.
6. **Who is affected:** agents on Latest get the release when it's released; pinned agents don't.

Then hand it to an approver. See [Approve and release](/guides/approve-and-release).

## Related

- [Source changes](/guides/source-changes)
- [MCP tools: backtest](/reference/mcp-tools#backtest)