v3 holds quality and uses 6.2% more prompt tokens.

Exchange Triage extraction prompts v1, v2 and v3 over the 27-case golden set, through the production pipeline (intake, extraction, grounding, appetite; writer off) on claude sonnet via the CLI. Baseline v2 (what shipped 2026-07-27), candidate v3 (production), v2-repeat as the noise floor, v1 as the frozen plain prompt. Reference: the golden labels. Judges: two source-attribution classifiers, validated against human labels in suites/exchange-triage.labels.jsonl. 27 cases, two variants, 7 binary checks. Tokens and cost are metered from the provider's usage report.

finished108 of 108 pairsstarted 2026-10-20 04:58 · ended 2026-10-20 04:59

Gate passed. Pass-all did not drop and decision_correct, fields_correct did not fall.

Noise floor: the candidate's change is within it. The baseline run twice (v2 against v2-repeat) moved pass-all +0.0% against the candidate's +22.2%; decision_correct +0.0% against the candidate's +0.0%, fields_correct +0.0% against the candidate's +3.7%. A drop no larger than the baseline's own drift is not evidence of a regression.

26%v2

pass-all rate, baseline (7 of 27)

48%v3

pass-all rate, candidate (13 of 27)

+6.2%

prompt tokens per case, 4,778 to 5,076 (+6.0% to +6.5% on the mean)

-17.3%

cost per case, $0.0232 to $0.0192 (API-equivalent)

+6.4%

latency per case, 89.7 s to 95.4 s

Checks

Every check is pass or fail per case. Rates carry Wilson 95% intervals. A check both variants pass everywhere is saturated: it guards against regressions but no longer discriminates.

check	kind	v2	v3	delta	95% interval (candidate)
extraction_schema Extraction JSON matched the schema within the pipeline's two attempts.	code	27/27 100.0%	27/27 100.0%	+0.0%	88% to 100%	saturated
fields_correct Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null).	code	26/27 96.3%	27/27 100.0%	+3.7%	88% to 100%
construction_canonical constructionType is the canonical ISO class the appetite rules consume, not the broker's wording.	code	18/18 100.0%	18/18 100.0%	+0.0%	82% to 100%	saturated
nulls_stay_null Fields the source never states come back null (no guessed TIV, year built, state or subclass).	code	5/5 100.0%	5/5 100.0%	+0.0%	57% to 100%	saturated
decision_correct The routing decision matches the golden label.	code	23/27 85.2%	23/27 85.2%	+0.0%	68% to 94%
attribution_jev Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above.	judge	19/27 70.4%	27/27 100.0%	+29.6%	88% to 100%
attribution_claude Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful.	judge	11/27 40.7%	16/27 59.3%	+18.5%	41% to 75%

1 per-case regression (baseline passed, candidate failed): GOLD-09 attribution_claude. Read these first; a single run of a nondeterministic model also produces them on noise, so the gate is on rates.

Prompt tokens, paired

One row per case. Grey is v2; the colored dot is v3. Mean change +6.2%, median change 314 tokens, bootstrap 95% interval on the mean difference 287 to 310 tokens.

v2v3 used fewerv3 used more

Per-case ledger

Dots are checks in the order of the table above; hollow means the check did not apply. Tokens are prompt tokens the model read (fresh input plus cache read plus cache creation).

case	tokens v2	tokens v3	change	decision	fitScore
GOLD-01	4,975	5,289	+6.3%	IN APPETITE	90
GOLD-02	5,028	5,342	+6.2%	IN APPETITE	88
GOLD-03	5,285	5,493	+3.9%	IN APPETITE	84
GOLD-04	4,811	5,125	+6.5%	IN APPETITE	84
GOLD-05	5,049	5,291	+4.8%	IN APPETITE	88
GOLD-06	5,054	5,332	+5.5%	DECLINED	12
GOLD-07	4,881	5,165	+5.8%	DECLINED	12
GOLD-08	4,816	5,100	+5.9%	INCOMPLETE	72
GOLD-09	4,657	5,043	+8.3%	INCOMPLETE	74
GOLD-10	4,939	5,187	+5.0%	REFERRED	32
V-01	4,718	5,032	+6.7%	IN APPETITE	90
V-02	4,601	4,915	+6.8%	INCOMPLETE	80
V-03	4,522	4,840	+7.0%	INCOMPLETE	74
V-04	4,671	4,955	+6.1%	IN APPETITE	88
V-05	4,670	4,984	+6.7%	IN APPETITE	88
V-06	4,685	4,999	+6.7%	IN APPETITE	86
V-07	4,737	5,031	+6.2%	IN APPETITE	88
V-08	4,595	4,909	+6.8%	INCOMPLETE	88
V-09	4,677	4,951	+5.9%	IN APPETITE	82
V-10	4,742	5,026	+6.0%	DECLINED	28
V-11	4,761	5,075	+6.6%	IN APPETITE	88
V-12	4,720	5,034	+6.7%	IN APPETITE	86
V-13	4,682	4,996	+6.7%	INCOMPLETE	84
V-14	4,579	4,893	+6.9%	INCOMPLETE	72
V-15	4,809	5,093	+5.9%	IN APPETITE	88
V-16	4,599	4,913	+6.8%	INCOMPLETE	86
GOLD-27	4,739	5,051	+6.6%	IN APPETITE	88

Judge validation

Each judge check is a classifier. Its verdicts count only as far as they agree with a person's labels on a split it never saw. Positive means a failure is present. TPR is the share of human-labelled failures the judge caught; TNR the share of human-labelled passes it left alone. Two judges on the same property can be compared row for row.

attribution_claude

split	n	human fails	human passes	TPR	TNR
all	54	14	40	100.0% 78% to 100%	65.0% 50% to 78%
train	12	1	11	100.0% 21% to 100%	54.5% 28% to 79%
dev	22	3	19	100.0% 44% to 100%	73.7% 51% to 88%
test	20	10	10	100.0% 72% to 100%	60.0% 31% to 83%

54 labels matched a judge verdict; 0 did not.

attribution_jev

split	n	human fails	human passes	TPR	TNR
all	54	14	40	50.0% 27% to 73%	100.0% 91% to 100%
train	12	1	11	100.0% 21% to 100%	100.0% 74% to 100%
dev	22	3	19	100.0% 44% to 100%	100.0% 83% to 100%
test	20	10	10	30.0% 11% to 60%	100.0% 72% to 100%

54 labels matched a judge verdict; 0 did not.

Where runs fail

Rows are the last stage that succeeded, columns the first that failed. The matrix says where to look before any judge is built.

v2

last success \ first failure	grounding	extraction	appetite
extraction	5
intake		1
grounding			3

9 of 27 traces failed a stage.

v3

last success \ first failure	grounding	appetite
extraction	5
grounding		3

8 of 27 traces failed a stage.

Method

Variants. v2: Extraction prompt v2: canonical construction classes and the broker-working-value rule. Shipped 2026-07-27. v2-repeat: v2 run a second time on identical inputs: the noise floor. v3: Extraction prompt v3, production: null-first subclass rule and source tied to a citable span, after a live over-extraction Arize caught. v1: Extraction prompt v1, frozen: plain instructions, no construction mapping, no few-shot examples.

Checks. extraction_schema (code): Extraction JSON matched the schema within the pipeline's two attempts. fields_correct (code): Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). construction_canonical (code): constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. nulls_stay_null (code): Fields the source never states come back null (no guessed TIV, year built, state or subclass). decision_correct (code): The routing decision matches the golden label. attribution_jev (judge): Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. attribution_claude (judge): Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful.

Gate. Candidate pass-all may not drop by more than 0%, and the rate of decision_correct and fields_correct may not be lower on the candidate. Regressions per case are listed, not gated.

Numbers. Token counts and cost come from the provider's usage report on every call (metered). Cost is the API-equivalent figure; these calls ran on a subscription. Latency is wall time per case including every call the variant made. Intervals: Wilson for rates, seeded percentile bootstrap (2,000 resamples) for paired means. No p-values.

Pairing. Only cases both variants completed are compared, in suite order, so every paired number has the same n (27). Errors: 0 on v2, 0 on v3.

Reproduce.

bun src/cli.ts run exchange-triage.ts --run-id 2026-10-19-prompts
bun src/cli.ts report exchange-triage 2026-10-19-prompts

Static report · Summary (Markdown) · Trace lines