Weigh evals you can watch
2026-11-12 06:56 UTC

The new prompt v6 beat the current prompt v3: 8 more cases passed every check.

Ship it gate

27 of 27
Decisions right

Matches the expert's answer; current prompt 22 of 27.

21 of 27
Cases that passed every check pass-all

All applicable checks passed; current prompt 13 of 27.

$0.0149
Cost per case

API-equivalent average; current prompt $0.0119.

The current prompt, run twice, moved 0.0 points. Anything smaller is not a real change. noise

finishedGraded 189 of 189 cases.started 2026-10-21 01:00 · ended 2026-10-21 01:11

Progress counts each case once for each prompt being graded.

What changed, in words

9 cases fixed. 0 cases broke.
A case counts once in each group if any check changed. A case can have both a fix and a break; an unanswered check doesn't count as either.

9 cases gained a passing check; 0 cases lost one, compared with the current prompt.

Fixed: GOLD-03, GOLD-07, GOLD-08, GOLD-10, V-02, V-03, V-09, V-13, V-16

Broke: None.

What we grade

Each check asks one question. Counts compare the current prompt with the new prompt on the same cases.

The answer has the required structure extraction_schema

The extracted answer fits the required fields within two attempts.

Every case passes, so this check can no longer tell prompts apart saturated

Current prompt v3
27 of 27 passed
New prompt v6
27 of 27 passed

The facts match the expert's answers fields_correct

Every labelled value matches, including amounts, construction class and missing facts.

Every case passes, so this check can no longer tell prompts apart saturated

Current prompt v3
27 of 27 passed
New prompt v6
27 of 27 passed

Construction type is the standard class construction_canonical

The class is one the insurance rules can use.

Every case passes, so this check can no longer tell prompts apart saturated

Current prompt v3
18 of 18 passed
New prompt v6
18 of 18 passed

Missing facts stay empty nulls_stay_null

The model doesn't guess facts the source never states.

Every case passes, so this check can no longer tell prompts apart saturated

Current prompt v3
5 of 5 passed
New prompt v6
5 of 5 passed

The decision matches the expert's decision_correct

The next step for the submission matches the expert's answer.

Current prompt v3
22 of 27 passed
New prompt v6
27 of 27 passed

Each fact is backed by its named source attribution_jev

Jev checks that the named document actually states the value.

An AI grader without human validation recorded judge

Current prompt v3
0 of 0 passed
New prompt v6
0 of 0 passed

Each fact is backed by its named source attribution_claude

Claude Haiku checks the source text against the extracted values.

An AI grader we checked against human answers judge

Current prompt v3
17 of 27 passed
New prompt v6
21 of 27 passed

Every case

The result and the next step, side by side. The change measures words sent to the model (what it costs) prompt tokens. Open check details to see every verdict.

CaseCurrent prompt v3New prompt v6Words sent
GOLD-01Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.1% more
GOLD-02Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.1% more
GOLD-03Grader flagged a sourcequote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
5.3% more
GOLD-04Grader flagged a sourcequote it IN APPETITEGrader flagged a sourcequote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
7.3% more
GOLD-05Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.1% more
GOLD-06Passed the answered checksDECLINEDPassed the answered checksDECLINED
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
5.6% more
GOLD-07Grader flagged a sourceDECLINEDPassed the answered checksDECLINED
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
5.8% more
GOLD-08Grader flagged a sourcefollow up with the broker INCOMPLETEPassed the answered checksfollow up with the broker INCOMPLETE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.2% more
GOLD-09Passed the answered checksfollow up with the broker INCOMPLETEPassed the answered checksfollow up with the broker INCOMPLETE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.5% more
GOLD-10Wrong decisionfollow up with the broker INCOMPLETEGrader flagged a sourcesend to an underwriter REFERRED
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: failed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
8.1% more
V-01Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.4% more
V-02Wrong decisionfollow up with the broker INCOMPLETEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: failed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
9.4% more
V-03Wrong decisionfollow up with the broker INCOMPLETEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: failed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
9.6% more
V-04Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.6% more
V-05Grader flagged a sourcequote it IN APPETITEGrader flagged a sourcequote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
7.5% more
V-06Grader flagged a sourcequote it IN APPETITEGrader flagged a sourcequote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
7.5% more
V-07Grader flagged a sourcequote it IN APPETITEGrader flagged a sourcequote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
7.5% more
V-08Passed the answered checksfollow up with the broker INCOMPLETEPassed the answered checksfollow up with the broker INCOMPLETE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.6% more
V-09Grader flagged a sourcequote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.6% more
V-10Passed the answered checksDECLINEDPassed the answered checksDECLINED
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.5% more
V-11Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.4% more
V-12Grader flagged a sourcequote it IN APPETITEGrader flagged a sourcequote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: failed
7.4% more
V-13Wrong decisionfollow up with the broker INCOMPLETEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: failed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
9.4% more
V-14Passed the answered checksfollow up with the broker INCOMPLETEPassed the answered checksfollow up with the broker INCOMPLETE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.7% more
V-15Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.4% more
V-16Wrong decisionfollow up with the broker INCOMPLETEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: failed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: doesn't apply
  • Missing facts stay empty nulls_stay_null: doesn't apply
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
9.4% more
GOLD-27Passed the answered checksquote it IN APPETITEPassed the answered checksquote it IN APPETITE
Check details

Current prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed

New prompt

  • The answer has the required structure extraction_schema: passed
  • The facts match the expert's answers fields_correct: passed
  • Construction type is the standard class construction_canonical: passed
  • Missing facts stay empty nulls_stay_null: passed
  • The decision matches the expert's decision_correct: passed
  • Each fact is backed by its named source attribution_jev: the grader gave no verdict, so it is left out unanswered
  • Each fact is backed by its named source attribution_claude: passed
7.3% more

Cost per case

Cost per case: 25.2% more with the new prompt. Words sent to the model: 7.5% more with the new prompt.

Each line joins the same case on the current and new prompts. Words sent to the model (what it costs) prompt tokens; exact usage and likely ranges are in For engineers.

01,4712,9424,4135,884words sent
Current prompt v3New prompt v6 used fewerNew prompt v6 used more
For engineers

The measurements

Current prompt v3 and new prompt v6; 27 paired cases.

Cases that passed every check pass-all: current prompt 13 of 27 (48%); new prompt 21 of 27 (78%). Likely range CI: 30.7% to 66.0% current, 59.2% to 89.4% new.

Measurement per caseCurrent promptNew promptChangeLikely range CI on mean difference
Words sent to the model (what it costs) prompt tokens5,0965,4797.5% more367 to 399; median difference 376
API-equivalent cost$0.0119$0.014925.2% more$0.0006 to $0.0054; median difference $0.0028
Time including all calls13.6 s13.2 s3.0% less-739 ms to -100 ms; median difference -405 ms

Words sent: likely range CI on relative mean change 7.2% more to 7.8% more.

Repeat-run change noise: repeat prompt v3-repeat moved 0.0 points higher in cases passing every check; new prompt moved 29.6 points higher. The new prompt's drop is within that change.

The decision matches the expert's decision_correct: repeat moved 3.7 points higher; new prompt moved 18.5 points higher.

The facts match the expert's answers fields_correct: repeat moved 3.7 points lower; new prompt moved 0.0 points higher.

CheckCurrent promptNew promptChangeLikely range CI for new prompt
The answer has the required structure extraction_schema

Every case passes, so this check can no longer tell prompts apart saturated

27/27 (100.0%)27/27 (100.0%)0.0 points higher88% to 100%
The facts match the expert's answers fields_correct

Every case passes, so this check can no longer tell prompts apart saturated

27/27 (100.0%)27/27 (100.0%)0.0 points higher88% to 100%
Construction type is the standard class construction_canonical

Every case passes, so this check can no longer tell prompts apart saturated

18/18 (100.0%)18/18 (100.0%)0.0 points higher82% to 100%
Missing facts stay empty nulls_stay_null

Every case passes, so this check can no longer tell prompts apart saturated

5/5 (100.0%)5/5 (100.0%)0.0 points higher57% to 100%
The decision matches the expert's decision_correct22/27 (81.5%)27/27 (100.0%)18.5 points higher88% to 100%
Each fact is backed by its named source attribution_jev0/0 (Not available)0/0 (Not available)Not available0% to 100%
Each fact is backed by its named source attribution_claude17/27 (63.0%)21/27 (77.8%)14.8 points higher59% to 89%

0 checks on cases got worse regression: none. The shipping rules compare rates, not individual cases.

Usage for each paired case

CaseCurrent words sentNew words sentChangedecisionfitScoreretriesfieldsWrongmodelErrors
GOLD-015,3245,7007.1% morequote it IN APPETITE9000
GOLD-025,3575,7357.1% morequote it IN APPETITE8600
GOLD-035,5865,8845.3% morequote it IN APPETITE8400
GOLD-045,1405,5167.3% morequote it IN APPETITE8800
GOLD-055,3065,6827.1% morequote it IN APPETITE8600
GOLD-065,4195,7235.6% moreDECLINED1200
GOLD-075,2525,5565.8% moreDECLINED1200
GOLD-085,1155,4837.2% morefollow up with the broker INCOMPLETE6200
GOLD-094,9865,3627.5% morefollow up with the broker INCOMPLETE7800
GOLD-105,2025,6248.1% moresend to an underwriter REFERRED3500
V-015,0475,4237.4% morequote it IN APPETITE9000
V-024,9305,3959.4% morequote it IN APPETITE8400
V-034,8515,3169.6% morequote it IN APPETITE9200
V-044,9705,3467.6% morequote it IN APPETITE8800
V-054,9995,3757.5% morequote it IN APPETITE9000
V-065,0145,3907.5% morequote it IN APPETITE8600
V-075,0205,3967.5% morequote it IN APPETITE9200
V-084,9245,3007.6% morefollow up with the broker INCOMPLETE8600
V-094,9645,3407.6% morequote it IN APPETITE8400
V-105,0415,4177.5% moreDECLINED2800
V-115,0905,4667.4% morequote it IN APPETITE8600
V-125,0495,4257.4% morequote it IN APPETITE8400
V-135,0075,4769.4% morequote it IN APPETITE9000
V-144,9085,2847.7% morefollow up with the broker INCOMPLETE7200
V-155,1085,4847.4% morequote it IN APPETITE8800
V-164,9285,3939.4% morequote it IN APPETITE8600
GOLD-275,0665,4367.3% morequote it IN APPETITE8800

Checking the AI grader

Each fact is backed by its named source attribution_claude. An AI grader we checked against human answers judge.

Human answer groupAnswersReal failuresGood answersFailures caughtGood answers flaggedGood answers left alone
all541440Catches 71.4% of real failures TPR
likely range CI: 45% to 88%		Wrongly flags 30.0% of good answers TNR70.0% left alone; likely range CI: 55% to 82%
train12111Catches 0.0% of real failures TPR
likely range CI: 0% to 79%		Wrongly flags 9.1% of good answers TNR90.9% left alone; likely range CI: 62% to 98%
dev22319Catches 100.0% of real failures TPR
likely range CI: 44% to 100%		Wrongly flags 36.8% of good answers TNR63.2% left alone; likely range CI: 41% to 81%
test201010Catches 70.0% of real failures TPR
likely range CI: 40% to 89%		Wrongly flags 40.0% of good answers TNR60.0% left alone; likely range CI: 31% to 83%

54 human answers matched a grader verdict; 0 did not.

Where runs fail

Rows show the last successful step; columns show the first failed step.

Current prompt v3

7 of 27 full run records failed a step trace.

Last success / first failureappetitegrounding
grounding5
extraction2

New prompt v6

4 of 27 full run records failed a step trace.

Last success / first failuregrounding
extraction4

Method

Exchange Triage extraction prompts v1 to v4 over the 27-case golden set, through the production pipeline (intake, extraction, grounding, appetite; writer off) on claude sonnet via the CLI. current prompt v3 (production), new prompt v6 (v5 with the per-unit clause corrected), v3-repeat as the repeat-run change noise, v5, v4, v2 and v1 for history. Reference: the golden labels. AI graders judge: two source-attribution classifiers, validated against human labels in weigh/labels/extraction-prompts.jsonl.

Versions variant

Version v2: Extraction prompt v2: canonical construction classes and the broker-working-value rule. Shipped 2026-07-27.

Version v3: Extraction prompt v3, production: null-first subclass rule and source tied to a citable span, after a live over-extraction Arize caught.

Version v3-repeat: v3 run a second time on identical inputs: the repeat-run change noise.

Version v4: Extraction prompt v4, measured 2026-10-20: a bare construction word is a stated class (bare masonry is Joisted Masonry), both means each document states it, outside-knowledge values keep source null. From the measured v3 failures.

Version v6: Extraction prompt v6, new prompt: v5 with one clause corrected, a per-unit target rate times the unit count is the target premium. From the measured v5 misses on V-12 and V-16.

Version v5: Extraction prompt v5, measured 2026-10-20: v4 plus one targetPremium rule; fixed GOLD-09 and GOLD-02, missed per-unit targets on V-12 and V-16.

Version v1: Extraction prompt v1, frozen: plain instructions, no construction mapping, no few-shot examples.

Check definitions

The answer has the required structure extraction_schema: Extraction JSON matched the schema within the pipeline's two attempts. Checked by code.

The facts match the expert's answers fields_correct: Every labelled field matches the golden value under the app's own field scorer (0.5% on dollar amounts, canonical construction class, null means null). Checked by code.

Construction type is the standard class construction_canonical: constructionType is the canonical ISO class the appetite rules consume, not the broker's wording. Checked by code.

Missing facts stay empty nulls_stay_null: Fields the source never states come back null (no guessed TIV, year built, state or subclass). Checked by code.

The decision matches the expert's decision_correct: The routing decision matches the golden label. Checked by code.

Each fact is backed by its named source attribution_jev: Jev (TypeSafe yes/no classifier) says every `source` attribution is truthful: the value is stated in the document it names. Pass at probability 0.5 or above. An AI grader without human validation recorded judge

Each fact is backed by its named source attribution_claude: Claude Haiku, given only the source text and the extracted fields, says every `source` attribution is truthful. An AI grader we checked against human answers judge

The grader gave no verdict, so it is left out unanswered. A case's result uses only the checks that answered.

Shipping rules gate: the share of cases passing every check may fall by at most 0%. These checks must not fall: The decision matches the expert's decision_correct, The facts match the expert's answers fields_correct. The measured comparison clears these rules.

Usage and costs are metered from the provider's report on every call. Costs are API-equivalent; these calls used a subscription. Time includes every call made for each case.

Likely range CI: Wilson for rates; seeded percentile bootstrap with 2,000 resamples for paired means. No p-values. Only cases completed by both prompts are compared, in suite order (27 pairs). Errors: 0 current, 0 new.

Words sent include fresh input, cache read and cache creation. Cached answers are reused when checks are rerun. 0 of 189 stored records were marked cached.

Reproduce

bun src/cli.ts run <path> --run-id 2026-10-20-v6 --concurrency 4
bun src/cli.ts report extraction-prompts 2026-10-20-v6 --baseline v3 --candidate v6

Static report · Summary (Markdown) · Full records as JSON lines trace

All comparison measurements (JSON)
{
  "comparison": {
    "baseline": {
      "variant": "v3",
      "n": 27,
      "errors": 0,
      "passAll": {
        "id": "pass_all",
        "passed": 13,
        "n": 27,
        "rate": 0.48148148148148145,
        "ci": {
          "lo": 0.3074295064970908,
          "hi": 0.6601467541378958
        }
      },
      "checks": [
        {
          "id": "extraction_schema",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        {
          "id": "fields_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        {
          "id": "construction_canonical",
          "passed": 18,
          "n": 18,
          "rate": 1,
          "ci": {
            "lo": 0.8241154494176252,
            "hi": 1
          }
        },
        {
          "id": "nulls_stay_null",
          "passed": 5,
          "n": 5,
          "rate": 1,
          "ci": {
            "lo": 0.5655085052479191,
            "hi": 1
          }
        },
        {
          "id": "decision_correct",
          "passed": 22,
          "n": 27,
          "rate": 0.8148148148148148,
          "ci": {
            "lo": 0.6330094349480825,
            "hi": 0.9181941342571468
          }
        },
        {
          "id": "attribution_jev",
          "passed": 0,
          "n": 0,
          "rate": null,
          "ci": {
            "lo": 0,
            "hi": 1
          }
        },
        {
          "id": "attribution_claude",
          "passed": 17,
          "n": 27,
          "rate": 0.6296296296296297,
          "ci": {
            "lo": 0.442289046234463,
            "hi": 0.7846771293206314
          }
        }
      ],
      "usage": {
        "promptTokens": [
          5324,
          5357,
          5586,
          5140,
          5306,
          5419,
          5252,
          5115,
          4986,
          5202,
          5047,
          4930,
          4851,
          4970,
          4999,
          5014,
          5020,
          4924,
          4964,
          5041,
          5090,
          5049,
          5007,
          4908,
          5108,
          4928,
          5066
        ],
        "outputTokens": [
          940,
          959,
          987,
          993,
          938,
          951,
          946,
          1051,
          781,
          1673,
          892,
          868,
          967,
          893,
          1044,
          944,
          1138,
          845,
          907,
          877,
          1123,
          1239,
          869,
          1241,
          896,
          1133,
          880
        ],
        "costUsd": [
          0.009943799999999997,
          0.0101371,
          0.010440000000000001,
          0.010455399999999998,
          0.009922,
          0.010063299999999999,
          0.0099966,
          0.0110329,
          0.008320000000000001,
          0.017261600000000002,
          0.0094361,
          0.0091844,
          0.010166500000000002,
          0.009438400000000001,
          0.010951299999999999,
          0.009952799999999998,
          0.011893399999999998,
          0.0089538,
          0.0095778,
          0.0092855,
          0.011750399999999998,
          0.012906299999999999,
          0.0213818,
          0.0129122,
          0.009482200000000001,
          0.0239008,
          0.0218097
        ],
        "latencyMs": [
          13164,
          13320,
          13388,
          12906,
          12681,
          12499,
          12913,
          14083,
          12673,
          18232,
          12694,
          12676,
          13347,
          12277,
          12713,
          12501,
          14217,
          12677,
          14880,
          12235,
          15728,
          14492,
          13520,
          14747,
          13246,
          15031,
          14348
        ],
        "metered": true,
        "providers": [
          "claude-cli"
        ]
      }
    },
    "candidate": {
      "variant": "v6",
      "n": 27,
      "errors": 0,
      "passAll": {
        "id": "pass_all",
        "passed": 21,
        "n": 27,
        "rate": 0.7777777777777778,
        "ci": {
          "lo": 0.5924270453692919,
          "hi": 0.8939290451059105
        }
      },
      "checks": [
        {
          "id": "extraction_schema",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        {
          "id": "fields_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        {
          "id": "construction_canonical",
          "passed": 18,
          "n": 18,
          "rate": 1,
          "ci": {
            "lo": 0.8241154494176252,
            "hi": 1
          }
        },
        {
          "id": "nulls_stay_null",
          "passed": 5,
          "n": 5,
          "rate": 1,
          "ci": {
            "lo": 0.5655085052479191,
            "hi": 1
          }
        },
        {
          "id": "decision_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        {
          "id": "attribution_jev",
          "passed": 0,
          "n": 0,
          "rate": null,
          "ci": {
            "lo": 0,
            "hi": 1
          }
        },
        {
          "id": "attribution_claude",
          "passed": 21,
          "n": 27,
          "rate": 0.7777777777777778,
          "ci": {
            "lo": 0.5924270453692919,
            "hi": 0.8939290451059105
          }
        }
      ],
      "usage": {
        "promptTokens": [
          5700,
          5735,
          5884,
          5516,
          5682,
          5723,
          5556,
          5483,
          5362,
          5624,
          5423,
          5395,
          5316,
          5346,
          5375,
          5390,
          5396,
          5300,
          5340,
          5417,
          5466,
          5425,
          5476,
          5284,
          5484,
          5393,
          5436
        ],
        "outputTokens": [
          925,
          958,
          919,
          997,
          939,
          911,
          863,
          1058,
          789,
          1839,
          905,
          940,
          902,
          896,
          1051,
          968,
          948,
          846,
          903,
          896,
          1142,
          1223,
          908,
          1147,
          922,
          923,
          905
        ],
        "costUsd": [
          0.0189418,
          0.0264513,
          0.026657300000000002,
          0.0132786,
          0.0131871,
          0.0129385,
          0.012168799999999999,
          0.0264433,
          0.011304100000000001,
          0.022395800000000004,
          0.012177700000000001,
          0.012509300000000001,
          0.012020000000000003,
          0.0120176,
          0.0135783,
          0.012757600000000001,
          0.0125504,
          0.011493500000000002,
          0.012067499999999998,
          0.0120754,
          0.0147236,
          0.015354000000000001,
          0.012372900000000001,
          0.0147086,
          0.0126034,
          0.0123196,
          0.012257
        ],
        "latencyMs": [
          13858,
          14233,
          12544,
          12766,
          12278,
          13021,
          12468,
          13743,
          11361,
          18695,
          11842,
          12692,
          12573,
          12266,
          13698,
          13570,
          12662,
          11872,
          13250,
          12459,
          14933,
          14378,
          12555,
          13740,
          12841,
          13122,
          12606
        ],
        "metered": true,
        "providers": [
          "claude-cli"
        ]
      }
    },
    "passAllDelta": {
      "n": 27,
      "baselineMean": 0.48148148148148145,
      "candidateMean": 0.7777777777777778,
      "meanDelta": 0.2962962962962963,
      "medianDelta": 0,
      "pctChange": 0.6153846153846154,
      "ci": {
        "lo": 0.14814814814814814,
        "hi": 0.48148148148148145
      }
    },
    "promptTokens": {
      "n": 27,
      "baselineMean": 5096.407407407408,
      "candidateMean": 5478.777777777777,
      "meanDelta": 382.3703703703704,
      "medianDelta": 376,
      "pctChange": 0.07502743399489836,
      "ci": {
        "lo": 366.6666666666667,
        "hi": 399.1111111111111
      }
    },
    "outputTokens": {
      "n": 27,
      "baselineMean": 999.074074074074,
      "candidateMean": 986.0370370370371,
      "meanDelta": -13.037037037037036,
      "medianDelta": 3,
      "pctChange": -0.013049119555143651,
      "ci": {
        "lo": -43.22222222222222,
        "hi": 12.851851851851851
      }
    },
    "costUsd": {
      "n": 27,
      "baselineMean": 0.011872448148148149,
      "candidateMean": 0.014864925925925929,
      "meanDelta": 0.0029924777777777796,
      "medianDelta": 0.0027898999999999997,
      "pctChange": 0.25205229287478864,
      "ci": {
        "lo": 0.0006170148148148155,
        "hi": 0.005359051851851854
      }
    },
    "latencyMs": {
      "n": 27,
      "baselineMean": 13599.555555555555,
      "candidateMean": 13186.148148148148,
      "meanDelta": -413.4074074074074,
      "medianDelta": -405,
      "pctChange": -0.030398596904038257,
      "ci": {
        "lo": -738.5185185185185,
        "hi": -100.29629629629629
      }
    },
    "checks": [
      {
        "id": "extraction_schema",
        "baseline": {
          "id": "extraction_schema",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        "candidate": {
          "id": "extraction_schema",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        "delta": 0,
        "saturated": true
      },
      {
        "id": "fields_correct",
        "baseline": {
          "id": "fields_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        "candidate": {
          "id": "fields_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        "delta": 0,
        "saturated": true
      },
      {
        "id": "construction_canonical",
        "baseline": {
          "id": "construction_canonical",
          "passed": 18,
          "n": 18,
          "rate": 1,
          "ci": {
            "lo": 0.8241154494176252,
            "hi": 1
          }
        },
        "candidate": {
          "id": "construction_canonical",
          "passed": 18,
          "n": 18,
          "rate": 1,
          "ci": {
            "lo": 0.8241154494176252,
            "hi": 1
          }
        },
        "delta": 0,
        "saturated": true
      },
      {
        "id": "nulls_stay_null",
        "baseline": {
          "id": "nulls_stay_null",
          "passed": 5,
          "n": 5,
          "rate": 1,
          "ci": {
            "lo": 0.5655085052479191,
            "hi": 1
          }
        },
        "candidate": {
          "id": "nulls_stay_null",
          "passed": 5,
          "n": 5,
          "rate": 1,
          "ci": {
            "lo": 0.5655085052479191,
            "hi": 1
          }
        },
        "delta": 0,
        "saturated": true
      },
      {
        "id": "decision_correct",
        "baseline": {
          "id": "decision_correct",
          "passed": 22,
          "n": 27,
          "rate": 0.8148148148148148,
          "ci": {
            "lo": 0.6330094349480825,
            "hi": 0.9181941342571468
          }
        },
        "candidate": {
          "id": "decision_correct",
          "passed": 27,
          "n": 27,
          "rate": 1,
          "ci": {
            "lo": 0.8754409628553641,
            "hi": 1
          }
        },
        "delta": 0.18518518518518523,
        "saturated": false
      },
      {
        "id": "attribution_jev",
        "baseline": {
          "id": "attribution_jev",
          "passed": 0,
          "n": 0,
          "rate": null,
          "ci": {
            "lo": 0,
            "hi": 1
          }
        },
        "candidate": {
          "id": "attribution_jev",
          "passed": 0,
          "n": 0,
          "rate": null,
          "ci": {
            "lo": 0,
            "hi": 1
          }
        },
        "delta": null,
        "saturated": false
      },
      {
        "id": "attribution_claude",
        "baseline": {
          "id": "attribution_claude",
          "passed": 17,
          "n": 27,
          "rate": 0.6296296296296297,
          "ci": {
            "lo": 0.442289046234463,
            "hi": 0.7846771293206314
          }
        },
        "candidate": {
          "id": "attribution_claude",
          "passed": 21,
          "n": 27,
          "rate": 0.7777777777777778,
          "ci": {
            "lo": 0.5924270453692919,
            "hi": 0.8939290451059105
          }
        },
        "delta": 0.14814814814814814,
        "saturated": false
      }
    ],
    "regressions": []
  },
  "noise": {
    "variant": "v3-repeat",
    "passAllDrop": 0,
    "candidatePassAllDrop": -0.29629629629629634,
    "checks": [
      {
        "id": "decision_correct",
        "noiseDrop": -0.03703703703703709,
        "candidateDrop": -0.18518518518518523
      },
      {
        "id": "fields_correct",
        "noiseDrop": 0.03703703703703709,
        "candidateDrop": 0
      }
    ],
    "within": true
  },
  "gate": {
    "pass": true,
    "reasons": []
  },
  "judges": [
    {
      "joined": 54,
      "unmatched": 0,
      "splits": [
        {
          "split": "all",
          "n": 54,
          "humanFails": 14,
          "humanPasses": 40,
          "tpr": 0.7142857142857143,
          "tnr": 0.7,
          "tprCi": {
            "lo": 0.4535045882751561,
            "hi": 0.882788120898909
          },
          "tnrCi": {
            "lo": 0.545696893456796,
            "hi": 0.8192532908430471
          }
        },
        {
          "split": "train",
          "n": 12,
          "humanFails": 1,
          "humanPasses": 11,
          "tpr": 0,
          "tnr": 0.9090909090909091,
          "tprCi": {
            "lo": 0,
            "hi": 0.7934567085261071
          },
          "tnrCi": {
            "lo": 0.6226353745137962,
            "hi": 0.9837682477371741
          }
        },
        {
          "split": "dev",
          "n": 22,
          "humanFails": 3,
          "humanPasses": 19,
          "tpr": 1,
          "tnr": 0.631578947368421,
          "tprCi": {
            "lo": 0.43849391955098227,
            "hi": 1
          },
          "tnrCi": {
            "lo": 0.4103915749815672,
            "hi": 0.8085072762372616
          }
        },
        {
          "split": "test",
          "n": 20,
          "humanFails": 10,
          "humanPasses": 10,
          "tpr": 0.7,
          "tnr": 0.6,
          "tprCi": {
            "lo": 0.39677321997956516,
            "hi": 0.892210712513788
          },
          "tnrCi": {
            "lo": 0.3126695474501864,
            "hi": 0.8318224187964902
          }
        }
      ],
      "check": "attribution_claude",
      "labels": "hub:extraction-prompts/2026-10-20-v6 (54 labels)",
      "at": "2026-10-21T23:04:53.580Z"
    }
  ]
}