# Judge protocol > **Public version, and it has to be public.** The generated scripts stay private, so a reader cannot check the conclusions. Only the method. This document and [`01-rubric.md`](01-rubric.md) are what the artifact rests on. > > **Status:** this protocol governs Tier 2 (the judged criteria C1, C2, C4 and C6). **Tier 2 has not been run.** Nothing in this document is a result. It is the procedure, published in advance, so that the numbers it eventually produces can be checked against the design that was committed to beforehand. --- ## 1. Judge selection The three candidate writers span three different labs, so **no obvious frontier judge is neutral.** Two runs, both reported: | Run | Judge | Purpose | | --- | --- | --- | | **Primary** | A model from a **fourth family** (for example Mistral Large, Llama, DeepSeek or Qwen) | The headline scoring run. No family overlap with any candidate. | | **Bias measurement** | One judge per candidate family, all three | Measures how much each judge favours its own sibling. | **The question this design answers: does the ranking survive changing the judge?** A stable ranking across all four judges is a robustness finding and strengthens the primary result. A ranking that flips depending on who judges is a more interesting result and probably the headline. Publishable either way, which is the point. **Report per judge**, never averaged into a single number. Averaging would hide precisely the effect being measured. --- ## 2. The judge prompt Used verbatim. Any change requires re-validation. > **Revision note.** An earlier version of this prompt counted elaborating sentences per individual beat, matching the original C2. The pilot showed that criterion was measuring prose form rather than depth, and C2 was rewritten to grade at beat-group level (see [`01-rubric.md`](01-rubric.md) §4). **The prompt below has been brought into line with the revised C2.** Because Tier 2 has not been run, no agreement figure has ever been reported against either version. ### System ``` You are grading a single Act 1 section of a video script against a fixed rubric. You will be given: 1. The outline beats assigned to this section, organised into headed groups. 2. The outline's example text for those beats. 3. The section to grade. Rules: - Apply only the criteria given. Do not import your own taste about what makes writing good. - Do not reward or penalise length. A longer section is not a better section. - Do not reward or penalise layout. Prose and enumerated lists are judged by what they supply, never by their form. - Work through the analysis steps in order and write them out BEFORE giving any verdict. Do not state a verdict early and justify it afterwards. - If a criterion is genuinely ambiguous for this section, say so explicitly and mark it AMBIGUOUS rather than guessing. ``` ### User template ``` === ASSIGNED BEATS === {beat_list} === OUTLINE EXAMPLE TEXT === {example_column} === SECTION TO GRADE === {section_text} === ANALYSIS (complete in order, before any verdict) === STEP 1 - Beat coverage. For each assigned bullet, quote the sentence or sentences in the section that express it, or write ABSENT. Wording will differ from the outline; judge on meaning, not phrasing. STEP 2 - Development, by beat GROUP. The assigned beats are organised into headed groups. Work group by group, not bullet by bullet. For each group, list every elaborating MOVE the section supplies across that group's material. A move is exactly one of five things: MECHANISM - how something works or why it follows CONSEQUENCE - what resulted from it EXAMPLE - a specific instance of it CONTRAST - a comparison, counter-case or reversal QUALIFICATION - a limit, caveat or exception For each move, quote the text supplying it and name which of the five it is. A sentence that restates a beat in different words is NOT a move. Moves count whether they appear in prose or in an enumerated list. Then state per group: DEVELOPED (2 or more moves) or UNDEVELOPED (fewer than 2). STEP 3 - Additions. List any material in the section that does not correspond to an assigned beat. For each, classify as TRANSITION (connective material serving assigned beats) or SCOPE_CREEP (a new claim, tease or argument the outline did not ask for). STEP 4 - Framing. State the topic and audience the section is written for. Could that framing have been predicted from the outline? YES or NO. === VERDICTS === C1_COVERAGE: of C2_DEVELOPMENT: of C4_ADDITIONS: PASS if no SCOPE_CREEP, otherwise FAIL C6_ON_BRIEF: PASS or FAIL AMBIGUOUS: ``` ### Notes on the design **Reasoning before verdict is not a style preference.** A model conditions each token on what came before, so a verdict stated first turns everything after it into post-hoc justification, which will read as convincing. Reasoning first means the verdict is conditioned on the analysis. Same words, opposite causality. **Every step produces an artifact before a judgement.** Quoted sentences, named moves, a classified list. When judge and human disagree you compare artifacts and see exactly where the divergence happened, instead of two parties trading opinions. **The judge returns counts, not pass or fail, for C1 and C2.** Thresholds are applied afterwards in code. A threshold can then be changed without re-running the judge, and the threshold decision stays with the rubric author rather than being delegated to the model. **Naming the move type is not decoration.** Forcing the judge to classify each move into one of five named categories is what stops "it elaborates" from being asserted. A move that cannot be named is not counted, which is the mechanism that keeps decorative elaboration from passing quietly. **AMBIGUOUS is a permitted answer.** A judge forced to guess produces confident noise. An ambiguity rate is itself a measurement of rubric quality. **C5 and G1 never reach the judge.** They are code checks. --- ## 3. Validation procedure Run in this order. The order is the safeguard. 1. **Pilot-label 6 sections by hand**, blind, recording every ambiguity encountered. Fix the rubric. 2. **Set `N`** for C1 from the pilot. 3. **Freeze the rubric and the judge prompt.** 4. **Hand-label the validation sample blind**, roughly 12 sections, before any judge output exists. Written labels, timestamped, not revisited. 5. **A second human labels the same sample**, from the written rubric alone, with no discussion beforehand. Roughly 15 sections, two criteria, 45 to 60 minutes. 6. **Compute human to human agreement.** This is the ceiling. 7. **Run the judge** on the same sample, order randomised. 8. **Compute judge to human agreement**: percent agreement and Cohen's kappa. 9. **Run the adversarial fixtures** (§4). 10. **Read every disagreement individually** and record what caused it. ### The statistic **Percent agreement and Cohen's kappa. Not correlation.** The criteria are binary, and correlation is for scales. Kappa matters here specifically: if 90% of sections pass a criterion, a judge that stamps PASS on everything scores 90% raw agreement while being worthless. Kappa corrects for agreement expected by chance and exposes that. ### The target **Not 100%. Human to human agreement.** If two careful people applying this rubric agree 78% of the time, a judge agreeing with the author 85% of the time is performing *above the task's own consistency*, and pushing for 95% would mean the judge is more consistent than the task actually is. **If no second labeller can be found**, the report has to state plainly: *"Single labeller. No inter-human baseline was established, so judge agreement cannot be calibrated against the task's own ceiling."* That sentence is not a weakness in the write-up. Omitting it would be. ### On iteration Iterating the judge prompt is expected. **Reporting agreement on the sample used for tuning is not.** Either hold out a fresh set for the reported number, or state explicitly that iteration occurred and on which sample. --- ## 4. Adversarial validation Aggregate agreement can look healthy while the judge fails the one job it exists for, because most real sections are not padded and it gets those right. Run the judge on the three fixtures in [`04-test-cases.md`](04-test-cases.md), which are matched to roughly 300 words each so length cannot explain any difference: | Fixture | Engineered flaw | Judge must return | | --- | --- | --- | | **T1 Dense** | none, control | C1 pass, C2 pass | | **T2 Repetition-padded** | restates the same points | **C1 fail** | | **T3 Breadth-padded** | many claims, none developed | C1 pass, **C2 fail** | **A judge that misses T2 or T3 is not usable, whatever its aggregate agreement score.** Report the result per fixture, never averaged: catching repetition padding but missing breadth padding is a specific, nameable weakness that an average would conceal. This test matters most for **C2**, because a verbosity-biased judge conflates more text with more elaboration. That failure mode would reward exactly the padding C2 exists to catch. --- ## 5. Controls - **Answer order randomised** on every comparative call, or swapped and averaged. The pipeline's original checker always audited the same writer first. - **No quality criterion rewards length**, and the judge is instructed explicitly on this. - **Judge prompt frozen** across the whole run. Any change invalidates the agreement figures. - **Blind labelling**, enforced procedurally rather than by care, because contamination of the author's own labels is unfalsifiable after the fact. - **The rubric author is not the second labeller**, and no model that helped write the rubric may serve as a human baseline. A model can be a second *judge*. It can never be a human one. --- ## 6. Reporting template Tier 2 has not been run. This is the shape the report will take, published in advance so the eventual numbers cannot be selectively framed. ``` JUDGE primary judge bias-measurement judges prompt frozen, published verbatim above iterations VALIDATION pilot sample 6 sections validation sample sections human-human agreement <%> k = [or: NOT ESTABLISHED, single labeller] judge-human agreement <%> k = per criterion ambiguity rate <%> of criterion applications marked AMBIGUOUS ADVERSARIAL FIXTURES T1 dense T2 repetition-padded T3 breadth-padded RESULTS (per model, per section, across the Act sections of the run) C1 coverage C2 development C4 additions C5 paraphrase C6 on brief G1 length (gate) RANKING STABILITY ranking under primary judge ranking under each family judge self-preference effect ``` **Report every number per criterion and per model.** A single composite score would hide which criterion drove the result, and the per-criterion breakdown is the part with information in it.