Exchange Triage extraction prompts v1, v2 and v3 over the 27-case golden set, through the production pipeline (intake, extraction, grounding, appetite; writer off) on claude sonnet via the CLI. Baseline v2 (what shipped 2026-07-27), candidate v3 (production), v2-repeat as the noise floor, v1 as the frozen plain prompt. Reference: the golden labels. Judges: two source-attribution classifiers, validated against human labels in suites/exchange-triage.labels.jsonl. 27 cases, two variants, 7 binary checks. Tokens and cost are metered from the provider's usage report.
Every check is pass or fail per case. Rates carry Wilson 95% intervals. A check both variants pass everywhere is saturated: it guards against regressions but no longer discriminates.
| check | kind | v2 | v3 | delta | 95% interval (candidate) | |
|---|---|---|---|---|---|---|
| extraction_schema Extraction JSON matched the schema within the pipeline's two attempts. | code | 27/27 100.0% | 27/27 100.0% | +0.0% | 88% to 100% | saturated |
| fields_correct Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). | code | 26/27 96.3% | 27/27 100.0% | +3.7% | 88% to 100% | |
| construction_canonical constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. | code | 18/18 100.0% | 18/18 100.0% | +0.0% | 82% to 100% | saturated |
| nulls_stay_null Fields the source never states come back null (no guessed TIV, year built, state or subclass). | code | 5/5 100.0% | 5/5 100.0% | +0.0% | 57% to 100% | saturated |
| decision_correct The routing decision matches the golden label. | code | 23/27 85.2% | 23/27 85.2% | +0.0% | 68% to 94% | |
| attribution_jev Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. | judge | 19/27 70.4% | 27/27 100.0% | +29.6% | 88% to 100% | |
| attribution_claude Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful. | judge | 11/27 40.7% | 16/27 59.3% | +18.5% | 41% to 75% |
1 per-case regression (baseline passed, candidate failed): GOLD-09 attribution_claude. Read these first; a single run of a nondeterministic model also produces them on noise, so the gate is on rates.
One row per case. Grey is v2; the colored dot is v3. Mean change +6.2%, median change 314 tokens, bootstrap 95% interval on the mean difference 287 to 310 tokens.
Dots are checks in the order of the table above; hollow means the check did not apply. Tokens are prompt tokens the model read (fresh input plus cache read plus cache creation).
| case | v2 | v3 | tokens v2 | tokens v3 | change | decision | fitScore | retries | fieldsWrong | modelErrors |
|---|---|---|---|---|---|---|---|---|---|---|
| GOLD-01 | 4,975 | 5,289 | +6.3% | IN APPETITE | 90 | 0 | 0 | |||
| GOLD-02 | 5,028 | 5,342 | +6.2% | IN APPETITE | 88 | 0 | 0 | |||
| GOLD-03 | 5,285 | 5,493 | +3.9% | IN APPETITE | 84 | 0 | 0 | |||
| GOLD-04 | 4,811 | 5,125 | +6.5% | IN APPETITE | 84 | 0 | 0 | |||
| GOLD-05 | 5,049 | 5,291 | +4.8% | IN APPETITE | 88 | 0 | 0 | |||
| GOLD-06 | 5,054 | 5,332 | +5.5% | DECLINED | 12 | 0 | 0 | |||
| GOLD-07 | 4,881 | 5,165 | +5.8% | DECLINED | 12 | 0 | 0 | |||
| GOLD-08 | 4,816 | 5,100 | +5.9% | INCOMPLETE | 72 | 0 | 0 | |||
| GOLD-09 | 4,657 | 5,043 | +8.3% | INCOMPLETE | 74 | 0 | 0 | |||
| GOLD-10 | 4,939 | 5,187 | +5.0% | REFERRED | 32 | 0 | 0 | |||
| V-01 | 4,718 | 5,032 | +6.7% | IN APPETITE | 90 | 0 | 0 | |||
| V-02 | 4,601 | 4,915 | +6.8% | INCOMPLETE | 80 | 0 | 0 | |||
| V-03 | 4,522 | 4,840 | +7.0% | INCOMPLETE | 74 | 0 | 0 | |||
| V-04 | 4,671 | 4,955 | +6.1% | IN APPETITE | 88 | 0 | 0 | |||
| V-05 | 4,670 | 4,984 | +6.7% | IN APPETITE | 88 | 0 | 0 | |||
| V-06 | 4,685 | 4,999 | +6.7% | IN APPETITE | 86 | 0 | 0 | |||
| V-07 | 4,737 | 5,031 | +6.2% | IN APPETITE | 88 | 0 | 0 | |||
| V-08 | 4,595 | 4,909 | +6.8% | INCOMPLETE | 88 | 0 | 0 | |||
| V-09 | 4,677 | 4,951 | +5.9% | IN APPETITE | 82 | 0 | 0 | |||
| V-10 | 4,742 | 5,026 | +6.0% | DECLINED | 28 | 0 | 0 | |||
| V-11 | 4,761 | 5,075 | +6.6% | IN APPETITE | 88 | 0 | 0 | |||
| V-12 | 4,720 | 5,034 | +6.7% | IN APPETITE | 86 | 0 | 0 | |||
| V-13 | 4,682 | 4,996 | +6.7% | INCOMPLETE | 84 | 0 | 0 | |||
| V-14 | 4,579 | 4,893 | +6.9% | INCOMPLETE | 72 | 0 | 0 | |||
| V-15 | 4,809 | 5,093 | +5.9% | IN APPETITE | 88 | 0 | 0 | |||
| V-16 | 4,599 | 4,913 | +6.8% | INCOMPLETE | 86 | 0 | 0 | |||
| GOLD-27 | 4,739 | 5,051 | +6.6% | IN APPETITE | 88 | 0 | 0 |
Each judge check is a classifier. Its verdicts count only as far as they agree with a person's labels on a split it never saw. Positive means a failure is present. TPR is the share of human-labelled failures the judge caught; TNR the share of human-labelled passes it left alone. Two judges on the same property can be compared row for row.
| split | n | human fails | human passes | TPR | TNR |
|---|---|---|---|---|---|
| all | 54 | 14 | 40 | 100.0% 78% to 100% | 65.0% 50% to 78% |
| train | 12 | 1 | 11 | 100.0% 21% to 100% | 54.5% 28% to 79% |
| dev | 22 | 3 | 19 | 100.0% 44% to 100% | 73.7% 51% to 88% |
| test | 20 | 10 | 10 | 100.0% 72% to 100% | 60.0% 31% to 83% |
54 labels matched a judge verdict; 0 did not.
| split | n | human fails | human passes | TPR | TNR |
|---|---|---|---|---|---|
| all | 54 | 14 | 40 | 50.0% 27% to 73% | 100.0% 91% to 100% |
| train | 12 | 1 | 11 | 100.0% 21% to 100% | 100.0% 74% to 100% |
| dev | 22 | 3 | 19 | 100.0% 44% to 100% | 100.0% 83% to 100% |
| test | 20 | 10 | 10 | 30.0% 11% to 60% | 100.0% 72% to 100% |
54 labels matched a judge verdict; 0 did not.
Rows are the last stage that succeeded, columns the first that failed. The matrix says where to look before any judge is built.
| last success \ first failure | grounding | extraction | appetite |
|---|---|---|---|
| extraction | 5 | ||
| intake | 1 | ||
| grounding | 3 |
9 of 27 traces failed a stage.
| last success \ first failure | grounding | appetite |
|---|---|---|
| extraction | 5 | |
| grounding | 3 |
8 of 27 traces failed a stage.
Variants. v2: Extraction prompt v2: canonical construction classes and the broker-working-value rule. Shipped 2026-07-27. v2-repeat: v2 run a second time on identical inputs: the noise floor. v3: Extraction prompt v3, production: null-first subclass rule and source tied to a citable span, after a live over-extraction Arize caught. v1: Extraction prompt v1, frozen: plain instructions, no construction mapping, no few-shot examples.
Checks. extraction_schema (code): Extraction JSON matched the schema within the pipeline's two attempts. fields_correct (code): Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). construction_canonical (code): constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. nulls_stay_null (code): Fields the source never states come back null (no guessed TIV, year built, state or subclass). decision_correct (code): The routing decision matches the golden label. attribution_jev (judge): Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. attribution_claude (judge): Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful.
Gate. Candidate pass-all may not drop by more than 0%, and the rate of decision_correct and fields_correct may not be lower on the candidate. Regressions per case are listed, not gated.
Numbers. Token counts and cost come from the provider's usage report on every call (metered). Cost is the API-equivalent figure; these calls ran on a subscription. Latency is wall time per case including every call the variant made. Intervals: Wilson for rates, seeded percentile bootstrap (2,000 resamples) for paired means. No p-values.
Pairing. Only cases both variants completed are compared, in suite order, so every paired number has the same n (27). Errors: 0 on v2, 0 on v3.
Reproduce.
bun src/cli.ts run exchange-triage.ts --run-id 2026-10-19-prompts bun src/cli.ts report exchange-triage 2026-10-19-prompts