Review and backtest a change
See what a proposed pack change would do to real recent decisions, and which agents it would reach, before anyone approves it.
An approver needs to see consequences, not diffs. A backtest re-decides your agents’ recent real cases under a proposal and under the current release, and tells you which decisions would change.
Run a backtest
In ChatGPT or Claude with the Felspar plugin, after propose_pack:
What would this proposal change? Backtest it.
The reviewing-pack-changes skill calls backtest with the proposal ID, then writes a short summary. Through REST, call POST /v0/proposals/{proposalId}/backtest.
| Input | Meaning |
|---|---|
proposal_id |
The proposal from propose_pack |
limit |
How many saved cases, newest first. 1 to 50, default 25. |
execution_ids |
Only these saved cases (up to 50) |
reinterpret |
true to re-classify the excerpts whose definitions changed. Run without it first to see the estimate. |
What it reports
Each saved case comes back as one of:
| Result | Meaning |
|---|---|
unchanged |
Same outcome under the proposal |
changed |
Different outcome: current to proposed |
unresolved |
Still undecided under the proposal, with the questions still open and why |
needs_reinterpretation |
A definition this case’s evidence was classified against changed, so its text must be read again. Comes with an estimate of the work. |
unavailable |
This case can’t be re-decided (for example, an older record format) |
Each case also shows both sides’ escalation, and the report counts escalationChanged: cases whose hand-off would differ, even when the outcome doesn’t. A scored pack shows each case’s total on both sides too.
It also lists the agents that ran those cases, and the agents following the job: on Latest (they will get the release once it’s approved and released) or pinned to one of its releases (they won’t).
Breaking changes
Code that acts on Felspar’s results may map escalation IDs to queues, or read subject types and relationship names. When a proposal removes one of those, or changes the job’s subject, the report carries breaking:
"breaking": {
"changes": ["removes escalation compliance-review", "removes relationship contract.matter"],
"agents": 1
}agents is how many agents run or follow the job. check_pack and propose_pack return the same list, so the author sees it before the backtest. A breaking change isn’t refused; it is something the approver and the owners of those agents must know before release.
How it re-decides
- Each case is decided at its own original time, so evidence freshness is judged as it was.
- Numbers and dates are re-read exactly from the saved text, at no cost.
- A classified answer is reused only when the classifier would be asked exactly the same thing under the proposal: same excerpt, same question definition, same answers, same instructions.
- Anything else needs a new answer. By default no model is called; the case is reported as
needs_reinterpretationwith an estimate of how many excerpts and requests it would take. - With
reinterpret: true, only those excerpts are classified, within a job’s usual limits per case. Backtests draw on their own daily allowance, separate from the one your agents’ decisions use, so a backtest can’t slow production decisions.
What it doesn’t do
A backtest writes nothing. It creates no release, receipt or decision, changes no past decision, and its report is returned, not stored. It covers only past cases: it doesn’t predict volume or every future case.
Cases come from the last 30 days of real decisions on the job (decisions are kept for 30 days). Cases whose evidence can’t be re-read are reported as such, never guessed.
Who can backtest
Members, approvers and owners, with their own current access to every connected source behind the proposal. Agent keys can’t. A backtest counts as 10 decisions toward rate limits.
Writing the review summary
The reviewing-pack-changes skill produces a summary for the approver:
- The change: which proposal, against which release, and which passages of the Doc changed.
- Counts: changed, unchanged, unresolved.
- Why decisions flip, grouped by cause (“all 14 flipped because the size floor moved from 200 to 100”), with two or three examples.
- Meaning changes: if a definition’s wording changed, how many cases need their text re-read, and the choice to run that.
- Escalations and breaking changes: how many cases would hand off differently, and every
breakingchange, with the agents that depend on the job. - Who is affected: agents on Latest get the release when it’s released; pinned agents don’t.
Then hand it to an approver. See Approve and release.