Exchange Triage extraction prompts v1 to v4 over the 27-case golden set, through the production pipeline (intake, extraction, grounding, appetite; writer off) on claude sonnet via the CLI. Baseline v3 (production), candidate v6 (v5 with the per-unit clause corrected), v3-repeat as the noise floor, v5, v4, v2 and v1 for history. Reference: the golden labels. Judges: two source-attribution classifiers, validated against human labels in weigh/labels/extraction-prompts.jsonl. 27 cases, two variants, 7 binary checks. Tokens and cost are metered from the provider's usage report.
Every check is pass or fail per case. Rates carry Wilson 95% intervals. A check both variants pass everywhere is saturated: it guards against regressions but no longer discriminates.
| check | kind | v3 | v6 | delta | 95% interval (candidate) | |
|---|---|---|---|---|---|---|
| extraction_schema Extraction JSON matched the schema within the pipeline's two attempts. | code | 27/27 100.0% | 27/27 100.0% | +0.0% | 88% to 100% | saturated |
| fields_correct Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). | code | 27/27 100.0% | 27/27 100.0% | +0.0% | 88% to 100% | saturated |
| construction_canonical constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. | code | 18/18 100.0% | 18/18 100.0% | +0.0% | 82% to 100% | saturated |
| nulls_stay_null Fields the source never states come back null (no guessed TIV, year built, state or subclass). | code | 5/5 100.0% | 5/5 100.0% | +0.0% | 57% to 100% | saturated |
| decision_correct The routing decision matches the golden label. | code | 22/27 81.5% | 27/27 100.0% | +18.5% | 88% to 100% | |
| attribution_jev Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. | judge | 0/0 n/a | 0/0 n/a | n/a | 0% to 100% | |
| attribution_claude Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful. | judge | 17/27 63.0% | 21/27 77.8% | +14.8% | 59% to 89% |
No per-case regression: every check the baseline passed, the candidate passed too.
One row per case. Grey is v3; the colored dot is v6. Mean change +7.5%, median change 376 tokens, bootstrap 95% interval on the mean difference 367 to 399 tokens.
Dots are checks in the order of the table above; hollow means the check did not apply. Tokens are prompt tokens the model read (fresh input plus cache read plus cache creation).
| case | v3 | v6 | tokens v3 | tokens v6 | change | decision | fitScore | retries | fieldsWrong | modelErrors |
|---|---|---|---|---|---|---|---|---|---|---|
| GOLD-01 | 5,324 | 5,700 | +7.1% | IN APPETITE | 90 | 0 | 0 | |||
| GOLD-02 | 5,357 | 5,735 | +7.1% | IN APPETITE | 86 | 0 | 0 | |||
| GOLD-03 | 5,586 | 5,884 | +5.3% | IN APPETITE | 84 | 0 | 0 | |||
| GOLD-04 | 5,140 | 5,516 | +7.3% | IN APPETITE | 88 | 0 | 0 | |||
| GOLD-05 | 5,306 | 5,682 | +7.1% | IN APPETITE | 86 | 0 | 0 | |||
| GOLD-06 | 5,419 | 5,723 | +5.6% | DECLINED | 12 | 0 | 0 | |||
| GOLD-07 | 5,252 | 5,556 | +5.8% | DECLINED | 12 | 0 | 0 | |||
| GOLD-08 | 5,115 | 5,483 | +7.2% | INCOMPLETE | 62 | 0 | 0 | |||
| GOLD-09 | 4,986 | 5,362 | +7.5% | INCOMPLETE | 78 | 0 | 0 | |||
| GOLD-10 | 5,202 | 5,624 | +8.1% | REFERRED | 35 | 0 | 0 | |||
| V-01 | 5,047 | 5,423 | +7.4% | IN APPETITE | 90 | 0 | 0 | |||
| V-02 | 4,930 | 5,395 | +9.4% | IN APPETITE | 84 | 0 | 0 | |||
| V-03 | 4,851 | 5,316 | +9.6% | IN APPETITE | 92 | 0 | 0 | |||
| V-04 | 4,970 | 5,346 | +7.6% | IN APPETITE | 88 | 0 | 0 | |||
| V-05 | 4,999 | 5,375 | +7.5% | IN APPETITE | 90 | 0 | 0 | |||
| V-06 | 5,014 | 5,390 | +7.5% | IN APPETITE | 86 | 0 | 0 | |||
| V-07 | 5,020 | 5,396 | +7.5% | IN APPETITE | 92 | 0 | 0 | |||
| V-08 | 4,924 | 5,300 | +7.6% | INCOMPLETE | 86 | 0 | 0 | |||
| V-09 | 4,964 | 5,340 | +7.6% | IN APPETITE | 84 | 0 | 0 | |||
| V-10 | 5,041 | 5,417 | +7.5% | DECLINED | 28 | 0 | 0 | |||
| V-11 | 5,090 | 5,466 | +7.4% | IN APPETITE | 86 | 0 | 0 | |||
| V-12 | 5,049 | 5,425 | +7.4% | IN APPETITE | 84 | 0 | 0 | |||
| V-13 | 5,007 | 5,476 | +9.4% | IN APPETITE | 90 | 0 | 0 | |||
| V-14 | 4,908 | 5,284 | +7.7% | INCOMPLETE | 72 | 0 | 0 | |||
| V-15 | 5,108 | 5,484 | +7.4% | IN APPETITE | 88 | 0 | 0 | |||
| V-16 | 4,928 | 5,393 | +9.4% | IN APPETITE | 86 | 0 | 0 | |||
| GOLD-27 | 5,066 | 5,436 | +7.3% | IN APPETITE | 88 | 0 | 0 |
Each judge check is a classifier. Its verdicts count only as far as they agree with a person's labels on a split it never saw. Positive means a failure is present. TPR is the share of human-labelled failures the judge caught; TNR the share of human-labelled passes it left alone. Two judges on the same property can be compared row for row.
| split | n | human fails | human passes | TPR | TNR |
|---|---|---|---|---|---|
| all | 54 | 14 | 40 | 71.4% 45% to 88% | 70.0% 55% to 82% |
| train | 12 | 1 | 11 | 0.0% 0% to 79% | 90.9% 62% to 98% |
| dev | 22 | 3 | 19 | 100.0% 44% to 100% | 63.2% 41% to 81% |
| test | 20 | 10 | 10 | 70.0% 40% to 89% | 60.0% 31% to 83% |
54 labels matched a judge verdict; 0 did not.
Rows are the last stage that succeeded, columns the first that failed. The matrix says where to look before any judge is built.
| last success \ first failure | appetite | grounding |
|---|---|---|
| grounding | 5 | |
| extraction | 2 |
7 of 27 traces failed a stage.
| last success \ first failure | grounding |
|---|---|
| extraction | 4 |
4 of 27 traces failed a stage.
Variants. v2: Extraction prompt v2: canonical construction classes and the broker-working-value rule. Shipped 2026-07-27. v3: Extraction prompt v3, production: null-first subclass rule and source tied to a citable span, after a live over-extraction Arize caught. v3-repeat: v3 run a second time on identical inputs: the noise floor. v4: Extraction prompt v4, measured 2026-10-20: a bare construction word is a stated class (bare masonry is Joisted Masonry), both means each document states it, outside-knowledge values keep source null. From the measured v3 failures. v6: Extraction prompt v6, candidate: v5 with one clause corrected, a per-unit target rate times the unit count is the target premium. From the measured v5 misses on V-12 and V-16. v5: Extraction prompt v5, measured 2026-10-20: v4 plus one targetPremium rule; fixed GOLD-09 and GOLD-02, missed per-unit targets on V-12 and V-16. v1: Extraction prompt v1, frozen: plain instructions, no construction mapping, no few-shot examples.
Checks. extraction_schema (code): Extraction JSON matched the schema within the pipeline's two attempts. fields_correct (code): Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). construction_canonical (code): constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. nulls_stay_null (code): Fields the source never states come back null (no guessed TIV, year built, state or subclass). decision_correct (code): The routing decision matches the golden label. attribution_jev (judge): Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. attribution_claude (judge): Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful.
Gate. Candidate pass-all may not drop by more than 0%, and the rate of decision_correct and fields_correct may not be lower on the candidate. Regressions per case are listed, not gated.
Numbers. Token counts and cost come from the provider's usage report on every call (metered). Cost is the API-equivalent figure; these calls ran on a subscription. Latency is wall time per case including every call the variant made. Intervals: Wilson for rates, seeded percentile bootstrap (2,000 resamples) for paired means. No p-values.
Pairing. Only cases both variants completed are compared, in suite order, so every paired number has the same n (27). Errors: 0 on v3, 0 on v6.
Reproduce.
bun src/cli.ts run <path> --run-id 2026-10-20-v6 --concurrency 4 bun src/cli.ts report extraction-prompts 2026-10-20-v6