Skip to content
Felspar
Esc
↑↓navigate↵open⌘Jpreview
On this page

Write test cases

Write test cases that make a pack trustworthy - one per outcome, both sides of every boundary, every exception and missing evidence.

A test case is the answers for one case plus the outcome the policy says it should get, and the rule that should decide it. Tests are how a reviewer trusts a pack, and how the next change is checked against today’s behavior. Write each one so it would catch a real mistake.

Test cases live in the pack draft’s cases list. check_pack runs all of them every time.

{
  "id": "namesake",
  "why": "Rule 1: the address shows a different company",
  "facts": { "page_kind": "different_company" },
  "expect": { "outcome": "review_wrong_site", "rule": "wrong-site" }
}

A case can also say which record it is about and which hand-off the policy expects. From a fictional support-triage pack:

{
  "id": "outage",
  "why": "Send service failures to the on-call engineer.",
  "subject": { "id": "48213" },
  "facts": { "topic": "failure", "premium": "yes" },
  "expect": { "outcome": "on-call", "rule": "failure", "escalation": "incident-review" }
}

subject.id is written as an agent would send it. expect.escalation is the escalation the document says this case hands off to; check_pack fails the test if the outcome names a different one.

Coverage checklist

  • Every outcome has at least one case.
  • Every boundary has a case just inside and just outside it.
  • Every exception has a case where it applies and one where it almost does.
  • Every rule is exercised by at least one case.
  • Every escalation the policy names has a case that reaches it.
  • Real past decisions are included, each with what actually happened.

Ask for real past cases first; they find gaps synthetic cases miss. Write synthetic cases only for coverage the real ones lack.

Rules for good tests

  • State the expected outcome from the policy, independently of what the draft returns. Copying the draft’s output into the expectation makes the test pass by definition.
  • When a test fails, decide with the policy owner whether the rule or the expectation is wrong. Fix that one. Never weaken a test to make it pass.
  • A past exception stays an exception. If a real case was approved against the written policy, note it and tell the owner; don’t rewrite the rule to match it.
  • Tests may contain customer data. Use only what reviewers in your workspace may see.

When the tests pass, propose_pack sends the new cases to review with the pack. Reviewers see each case’s expected and actual result on the review page.

Tests and backtests

Tests check the pack against cases you wrote. A backtest checks a proposed change against cases your agents actually decided in the last 30 days. Use both: tests for what the policy says should happen, backtests for what would really change.

Coming

Adding test cases as their own tool, and test cases with evidence text that exercise the classifier, are coming. Today, cases travel inside the pack draft and give answers directly.

Was this page helpful?