Evaluating a multi-agent script pipeline

Public version. Part of the evaluation artifact. See the summary above for the results and the documents section for what is published and what is held back.

Scope: Act 1 sections. Not hooks, not later acts, not conclusions. Section type demonstrably changes what “adequate development” means, so the rubric does not claim to travel beyond the type it was calibrated on.

1. What is being evaluated

Three frontier models, spanning three different labs, each render the same Act 1 section from a shared outline produced in an earlier stage of the pipeline. The outline carries two relevant columns: a beat-by-beat column with example content, under an explicit instruction not to copy it word for word, and a structural column.

The writers are not generating substance. They are rendering supplied substance in their own words. The rubric measures rendering quality, not invention.

What the evaluation may claim

Given a fixed outline, which model produces the densest, best-developed Act 1 prose while following the paraphrase instruction.

What it may not claim

Which model “writes better scripts.” The outline is held constant, so the comparison is of marginal variation on a fixed skeleton. That is a genuine strength, because most model comparisons are confounded by models choosing different structures, but it caps the claim and the cap has to be stated.

2. What “good” means

Written before any output was scored.

An Act 1 section is good when it covers every beat it was assigned, develops each one rather than merely asserting it, adds nothing the outline did not ask for, and does so in its own words rather than the outline’s.

Act 1 is the setup act. It carries more obligation to develop than a fifteen-second hook, and less than a body section. It is expected to move quickly. It is not expected to be a list.

3. Gate

Checked before quality scoring. Never contributes to the score.

G1: Section length

The section meets the length required for video pacing.

• Grader: code (word count).
• Result: pass or fail. A fail is reported separately and the section is still scored, so a length miss never masquerades as a quality result.

Why this is a gate and not a criterion. Word count is the proxy that was already being gamed: writers hit the target by restating the same point in different words. Scoring length as quality would rebuild the exact incentive this rubric exists to remove, and would bake verbosity bias into the instrument. Length is a real production constraint, so it is checked. It is never rewarded.

4. Quality criteria

C1: Outline coverage

List the beats assigned to this section in the outline. For each, mark whether it appears in the section as a distinct claim, regardless of wording. Fail if fewer than N of the assigned beats appear.

• Grader: model, human-validated.
• Scale: pass or fail, plus the raw count for analysis.
• N is unset. It is calibrated on the pilot sample before any scoring run. See §7.

Granularity comes from the outline, but the outline has to be read at the right level. The beat list is nested: headed groups containing bullets. The unit is the individual bullet, never the heading. One act in the pilot episode has 4 headed groups containing 15 bullets, and C1 reads completely differently at 4 than at 15.

Corrected after the pilot. An earlier draft of this rubric claimed the outline settled granularity by itself, so graders could not disagree. That was wrong. Nested structure leaves the unit ambiguous unless the rubric names the level.

Declared limitation: because the outline supplies the beats, C1 measures rendering fidelity, not the model’s capacity to generate substance. It would not transfer to a free-form writing task.

C2: Development

Revised after the pilot. The original version failed the best section in the sample. See “Why C2 was rewritten” below.

Assess development at the level of the beat GROUP, not the individual bullet. A group is the headed block in the outline.

A group is DEVELOPED if the section supplies at least two distinct elaborating moves across that group’s material, where a move is: a mechanism, a consequence, an example, a contrast, or a qualification. Moves count whether they appear in prose or in enumerated form. Fail if more than one third of the section’s groups are undeveloped.

• Grader: model.
• Scale: pass or fail, plus the per-group move counts and what each move was.
• Primary quality signal. With substance supplied, depth of treatment is the model’s actual job.

Restating is not elaborating. Repeating a beat in new words supplies none of the five moves, so padding cannot pass, and length gains a section nothing.

Form is not depth. A numbered list item that states a mechanism is a mechanism. The criterion counts what the text supplies, never how it is laid out.

Declared limitations:

• Beatable by decorative elaboration: text that technically adds a consequence or example while carrying no real content.
• The five moves are a closed list chosen by one author. A section could develop a group in a way none of them capture.
• Both thresholds (two moves per group, one third of groups) are judgements, not derived values. Neither has been empirically tuned.
• Grading at group level means a section can pass C2 while leaving individual bullets undeveloped, provided the group as a whole is treated with depth. That is deliberate, but it does make C2 coarser than C1.

Why C2 was rewritten

The original criterion counted elaborating sentences per bullet and failed any section where more than a third of bullets had fewer than two.

In the pilot, one section rendered a four-item group as four one-line numbered items, mirroring the outline’s own numbered layout. Under the original C2 that is four undeveloped beats and an automatic fail. But a numbered set of tactics is good script writing, not padding.

The result was that the original C2 failed the section with perfect outline coverage, no scope creep and full paraphrase compliance, while passing the two sections that copied the example column near-verbatim and overran into the next act’s material. The rubric was ranking prose form, not quality.

The fix uses the outline’s own structure. The beat list is nested, so each level is read for what it is good at: bullets give C1 an unambiguous unit for coverage, groups give C2 the right unit for depth. A group rendered as a list is developed if its items supply the moves. A group rendered as prose is judged the same way. Layout stops mattering, which is the point.

Consequence still to be checked: the three fixtures in the adversarial fixtures were built against the original C2. T3 (breadth-padded) has to be re-verified against this revision. It should still fail, because twelve one-line assertions with no mechanism, consequence, example, contrast or qualification leave every group undeveloped. That expectation is a prediction, not a result, until it is actually run.

C3: Factual support

Removed from the text rubric. Checked at the research stage against the production research document, not against the script.

Why. Act 1 sections make hard factual claims (dates, percentages, named people) and carry no citations, because scripts do not carry citations. Applied to script text, C3 fails every section from every model, which makes it useless rather than strict. The claim’s correctness is a property of the research that fed the outline, and that is where it has to be checked.

C4: Additions

Identify any material in the section that does not correspond to a beat in the outline. Classify each as either a transition (connective material serving assigned beats) or scope creep (new claims, teases, or arguments the outline did not ask for). Fail if the section contains any scope creep.

• Grader: model finds candidates, human classifies.
• Scale: pass or fail, plus a list of additions with their classification.

Why this exists. C1 counts what should be there. C4 catches what should not. Without it, a model that adds three unrequested claims looks identical to one that followed the brief, and in the labelled sample one model’s extra material was mostly appended closing teases.

Declared limitation: the transition versus scope-creep line is a judgement call and is expected to be the lowest-agreement criterion in the set. It is kept because the distinction matters to production, and its disagreement rate is reported rather than hidden.

C5: Paraphrase compliance

Compute the longest common contiguous word sequence and the proportion of overlapping 8-grams between the section and the outline’s example column for the corresponding beats. Fail if any single verbatim run exceeds 12 words, or if more than 10% of the section’s 8-grams appear in the example column.

• Grader: code. No judgement involved.
• Scale: pass or fail, plus both raw measures.

Why this exists. The pipeline already instructs writers not to copy the example column word for word. Nobody was measuring compliance. It is the only criterion in the set that is fully mechanical, and therefore free to run on every sample.

Hypothesis, stated in advance and refuted by the run. The prediction was: two of the three writers hew close to the example column while the third paraphrases further.

Result: wrong. The model identified as the paraphraser fails more sections than the other two, 8 of 9 against 7 of 9. 22 of 27 sections fail C5, and copying tracks the section rather than the model. Full results in the full C5 results.

Pre-registering the hypothesis was deliberate, and its refutation is the more valuable outcome. A prediction that survives is evidence. A prediction that is published and then reported as wrong is proof the prediction preceded the data rather than being fitted to it.

Confound found by the run, which may invalidate C5 as a compliance check: the example column is finished, on-voice, shippable prose assembled from prior drafts, not a sketch. Instructing a model not to copy a passage that is already exactly what is wanted is close to an impossible brief. See the full C5 results §4.

Declared limitations: both thresholds (12 words, 10%) are author-chosen and not derived. N-gram overlap detects copying, not close paraphrase that reuses structure while changing every word.

C6: On brief

The section addresses the topic and audience the outline specifies, and does not drift into a different framing. Fail if the section’s framing could not be predicted from the outline.

• Grader: model.
• Scale: pass or fail.

Expected to pass almost always, since the outline is prescriptive. It is retained as a guard rather than a discriminator: a criterion that catches a rare catastrophic failure earns its place even when it almost never fires.

5. Grader allocation

Principle: the cheapest grader that can do the job. Code is free and deterministic, so it runs on everything. Model graders cost money and vary between calls, so they are spent where judgement is genuinely required. Human time is scarcest and is spent validating the judge and reading disagreements, never on bulk grading.

6. Run protocol

One episode. All sections. Three models. 27 real outputs, at zero additional generation cost.

Generating new episodes costs real API spend, so the run uses data that already existed. The episode already had nine sections (a hook, five acts, a checkpoint, a story beat and a call to action), each written by all three models. That is a genuine dataset that was sitting unused.

Two tiers, because the criteria have different requirements

Tier 1: C5 paraphrase compliance, across all 27 sections. C5 compares each section to its own example column, so it is section-type agnostic: it works on the hook and the call to action exactly as well as on Act 2. It is a code check, so it needs no judge, no blind protocol, no human labeller and no agreement statistics.

This is the tier that could be finished and published quickly, and it carries the pre-registered hypothesis. It is the tier reported in this artifact.

Tier 2: C1, C2, C4 and C6, on the Act sections only. These need judgement, so they need the judge and a second human labeller. Restricted to sections of a single type at a time, since C2’s thresholds are type-specific. Tier 2 has not been run and nothing about it is reported here.

What this costs the claim, stated plainly

• One episode means one topic, one research base, one outline author. Anything found here is directional and may not generalise. It cannot be reported as a rate.
• Sections within one episode are not independent observations. They share a topic, a research document and an outline style, so 27 sections is not 27 independent trials.
• No repeat runs, so sampling noise is not separated from model difference. A single output per model per section is partly luck.

These are real limitations and they are the price of not spending weeks and an API budget. A narrow finding published honestly beats a broad one that never ships, but the narrowness has to be on the page, not left for a reader to work out.

Reported per model: result per criterion per section, with raw counts. Never a single composite score.

7. Calibration, before any scoring run

1. Pilot-label 6 sections against C1, C2, C4 and C6, blind.
2. Set N in C1 at the threshold that separates sections judged adequate from sections judged thin. If no such threshold exists, C1 is not discriminative and has to be reworked rather than tuned.
3. Record every ambiguity encountered while pilot-labelling. Each one is a criterion that is not yet checkable enough for a second person.
4. Only then freeze the rubric and begin the run.

The pilot exists so that the second human labeller is not spent on a rubric that still has holes in it.

8. Judge validation

Full procedure and the judge prompt: the judge protocol.

What has to be true before any judge output is reported:

• Labels written blind, before any judge output is seen.
• Judge drawn from outside the three candidate families, or the family-preference effect measured explicitly.
• Answer order randomised.
• Agreement reported as percent agreement and Cohen’s kappa, not correlation, because the criteria are binary.
• Agreement compared against human to human agreement, not against 100%.
• The judge has to separate the matched-length dense and padded fixtures in the adversarial fixtures. Failing that, it is not usable regardless of its aggregate score.
• If the judge prompt was iterated, the reported number comes from a set it was not tuned on, or the iteration is stated plainly.

9. Known limitations

Published in full. Volunteered limitations read as rigour. The same limitations found by a reader read as sloppiness.

Design

• All criteria authored by one person.
• All thresholds (N, one third, 12 words, 10%) are author-chosen judgements, not derived values.
• C4’s transition versus scope-creep boundary is expected to be the weakest-agreement criterion in the set.
• C2 is beatable by decorative elaboration. C5 is beatable by close structural paraphrase.

Scope

• Act 1 sections only for the judged criteria. The rubric is not calibrated for hooks, body sections or conclusions, and section type demonstrably affects what “adequate development” means.
• Because the outline supplies the beats, results measure rendering fidelity and would not transfer to free-form writing.
• One content domain, one production pipeline, one language.

Scale

• One episode, nine sections, three models. Small enough that a single unusual section still carries weight.
• The three writer models are not fully independent in their output: two produced near-verbatim passages in the pilot sample, so they should not be treated as three separate observations of “how frontier models write.”

Measurement

• The outline stage itself is ungraded, so outline quality is an uncontrolled confound beneath every writer comparison.
• Trajectory (retries, latency, cost per stage) was not logged for this run, so the comparison is on output quality alone.
• If no second human labeller is available for Tier 2, no inter-human baseline exists and judge agreement cannot be calibrated against the task’s own ceiling. That limitation has to be stated explicitly rather than omitted.

Prior-work context

• The pipeline’s original quality checker shared a model family with one of the writers, and applied no order randomisation. Comparative results it produced before this evaluation carry an unquantifiable bias. Both defects are corrected in this design.

Download original

Public version, and it has to be public. The generated scripts stay private, so a reader cannot check the conclusions. Only the method. This document and the rubric are what the artifact rests on.

Status: this protocol governs Tier 2 (the judged criteria C1, C2, C4 and C6). Tier 2 has not been run. Nothing in this document is a result. It is the procedure, published in advance, so that the numbers it eventually produces can be checked against the design that was committed to beforehand.

1. Judge selection

The three candidate writers span three different labs, so no obvious frontier judge is neutral. Two runs, both reported:

The question this design answers: does the ranking survive changing the judge? A stable ranking across all four judges is a robustness finding and strengthens the primary result. A ranking that flips depending on who judges is a more interesting result and probably the headline. Publishable either way, which is the point.

Report per judge, never averaged into a single number. Averaging would hide precisely the effect being measured.

2. The judge prompt

Used verbatim. Any change requires re-validation.

Revision note. An earlier version of this prompt counted elaborating sentences per individual beat, matching the original C2. The pilot showed that criterion was measuring prose form rather than depth, and C2 was rewritten to grade at beat-group level (see the rubric §4). The prompt below has been brought into line with the revised C2. Because Tier 2 has not been run, no agreement figure has ever been reported against either version.

System

You are grading a single Act 1 section of a video script against a fixed rubric.

You will be given:
  1. The outline beats assigned to this section, organised into headed groups.
  2. The outline's example text for those beats.
  3. The section to grade.

Rules:
- Apply only the criteria given. Do not import your own taste about what makes
  writing good.
- Do not reward or penalise length. A longer section is not a better section.
- Do not reward or penalise layout. Prose and enumerated lists are judged by
  what they supply, never by their form.
- Work through the analysis steps in order and write them out BEFORE giving any
  verdict. Do not state a verdict early and justify it afterwards.
- If a criterion is genuinely ambiguous for this section, say so explicitly and
  mark it AMBIGUOUS rather than guessing.

User template

=== ASSIGNED BEATS ===
{beat_list}

=== OUTLINE EXAMPLE TEXT ===
{example_column}

=== SECTION TO GRADE ===
{section_text}

=== ANALYSIS (complete in order, before any verdict) ===

STEP 1 - Beat coverage.
For each assigned bullet, quote the sentence or sentences in the section that
express it, or write ABSENT. Wording will differ from the outline; judge on
meaning, not phrasing.

STEP 2 - Development, by beat GROUP.
The assigned beats are organised into headed groups. Work group by group, not
bullet by bullet.

For each group, list every elaborating MOVE the section supplies across that
group's material. A move is exactly one of five things:
  MECHANISM      - how something works or why it follows
  CONSEQUENCE    - what resulted from it
  EXAMPLE        - a specific instance of it
  CONTRAST       - a comparison, counter-case or reversal
  QUALIFICATION  - a limit, caveat or exception

For each move, quote the text supplying it and name which of the five it is.
A sentence that restates a beat in different words is NOT a move.
Moves count whether they appear in prose or in an enumerated list.

Then state per group: DEVELOPED (2 or more moves) or UNDEVELOPED (fewer than 2).

STEP 3 - Additions.
List any material in the section that does not correspond to an assigned beat.
For each, classify as TRANSITION (connective material serving assigned beats) or
SCOPE_CREEP (a new claim, tease or argument the outline did not ask for).

STEP 4 - Framing.
State the topic and audience the section is written for. Could that framing have
been predicted from the outline? YES or NO.

=== VERDICTS ===

C1_COVERAGE:    <number of bullets present> of <total bullets assigned>
C2_DEVELOPMENT: <number of UNDEVELOPED groups> of <total groups>
C4_ADDITIONS:   PASS if no SCOPE_CREEP, otherwise FAIL
C6_ON_BRIEF:    PASS or FAIL
AMBIGUOUS:      <list any criterion you could not apply confidently, or NONE>

Notes on the design

Reasoning before verdict is not a style preference. A model conditions each token on what came before, so a verdict stated first turns everything after it into post-hoc justification, which will read as convincing. Reasoning first means the verdict is conditioned on the analysis. Same words, opposite causality.

Every step produces an artifact before a judgement. Quoted sentences, named moves, a classified list. When judge and human disagree you compare artifacts and see exactly where the divergence happened, instead of two parties trading opinions.

The judge returns counts, not pass or fail, for C1 and C2. Thresholds are applied afterwards in code. A threshold can then be changed without re-running the judge, and the threshold decision stays with the rubric author rather than being delegated to the model.

Naming the move type is not decoration. Forcing the judge to classify each move into one of five named categories is what stops “it elaborates” from being asserted. A move that cannot be named is not counted, which is the mechanism that keeps decorative elaboration from passing quietly.

AMBIGUOUS is a permitted answer. A judge forced to guess produces confident noise. An ambiguity rate is itself a measurement of rubric quality.

C5 and G1 never reach the judge. They are code checks.

3. Validation procedure

Run in this order. The order is the safeguard.

1. Pilot-label 6 sections by hand, blind, recording every ambiguity encountered. Fix the rubric.
2. Set N for C1 from the pilot.
3. Freeze the rubric and the judge prompt.
4. Hand-label the validation sample blind, roughly 12 sections, before any judge output exists. Written labels, timestamped, not revisited.
5. A second human labels the same sample, from the written rubric alone, with no discussion beforehand. Roughly 15 sections, two criteria, 45 to 60 minutes.
6. Compute human to human agreement. This is the ceiling.
7. Run the judge on the same sample, order randomised.
8. Compute judge to human agreement: percent agreement and Cohen’s kappa.
9. Run the adversarial fixtures (§4).
10. Read every disagreement individually and record what caused it.

The statistic

Percent agreement and Cohen’s kappa. Not correlation. The criteria are binary, and correlation is for scales.

Kappa matters here specifically: if 90% of sections pass a criterion, a judge that stamps PASS on everything scores 90% raw agreement while being worthless. Kappa corrects for agreement expected by chance and exposes that.

The target

Not 100%. Human to human agreement. If two careful people applying this rubric agree 78% of the time, a judge agreeing with the author 85% of the time is performing above the task’s own consistency, and pushing for 95% would mean the judge is more consistent than the task actually is.

If no second labeller can be found, the report has to state plainly: “Single labeller. No inter-human baseline was established, so judge agreement cannot be calibrated against the task’s own ceiling.” That sentence is not a weakness in the write-up. Omitting it would be.

On iteration

Iterating the judge prompt is expected. Reporting agreement on the sample used for tuning is not. Either hold out a fresh set for the reported number, or state explicitly that iteration occurred and on which sample.

4. Adversarial validation

Aggregate agreement can look healthy while the judge fails the one job it exists for, because most real sections are not padded and it gets those right.

Run the judge on the three fixtures in the adversarial fixtures, which are matched to roughly 300 words each so length cannot explain any difference:

A judge that misses T2 or T3 is not usable, whatever its aggregate agreement score. Report the result per fixture, never averaged: catching repetition padding but missing breadth padding is a specific, nameable weakness that an average would conceal.

This test matters most for C2, because a verbosity-biased judge conflates more text with more elaboration. That failure mode would reward exactly the padding C2 exists to catch.

5. Controls

• Answer order randomised on every comparative call, or swapped and averaged. The pipeline’s original checker always audited the same writer first.
• No quality criterion rewards length, and the judge is instructed explicitly on this.
• Judge prompt frozen across the whole run. Any change invalidates the agreement figures.
• Blind labelling, enforced procedurally rather than by care, because contamination of the author’s own labels is unfalsifiable after the fact.
• The rubric author is not the second labeller, and no model that helped write the rubric may serve as a human baseline. A model can be a second judge. It can never be a human one.

6. Reporting template

Tier 2 has not been run. This is the shape the report will take, published in advance so the eventual numbers cannot be selectively framed.

JUDGE
  primary judge           <model, version, date>
  bias-measurement judges <three models, versions>
  prompt                  frozen, published verbatim above
  iterations              <n, and on which sample>

VALIDATION
  pilot sample            6 sections
  validation sample       <n> sections
  human-human agreement   <%>  k = <k>     [or: NOT ESTABLISHED, single labeller]
  judge-human agreement   <%>  k = <k>     per criterion
  ambiguity rate          <%>  of criterion applications marked AMBIGUOUS

ADVERSARIAL FIXTURES
  T1 dense                <pass/fail vs expected>
  T2 repetition-padded    <pass/fail vs expected>
  T3 breadth-padded       <pass/fail vs expected>

RESULTS  (per model, per section, across the Act sections of the run)
  C1 coverage             <pass rate, raw counts>
  C2 development          <pass rate, undeveloped-group counts>
  C4 additions            <pass rate, scope-creep instances>
  C5 paraphrase           <pass rate, max verbatim run, 8-gram overlap %>
  C6 on brief             <pass rate>
  G1 length (gate)        <pass rate, reported separately>

RANKING STABILITY
  ranking under primary judge      <order>
  ranking under each family judge  <order x 3>
  self-preference effect           <per judge, toward its own family>

Report every number per criterion and per model. A single composite score would hide which criterion drove the result, and the per-criterion breakdown is the part with information in it.

Download original

Public version. Written after the pilot found three defects in the outline, not in the models. These are fixes to the outline template, not to the rubric. Until they are applied, C4 and G1 cannot be scored at all.

Episode specifics are removed throughout. The defects are structural and survive the removal intact.

Why the brief needed fixing at all

Three models were given the same outline and resolved its internal contradictions differently. That is not a model difference. It is an instruction defect, and any evaluation run on top of it would attribute the outline’s ambiguity to the writers.

An outline that can be read two ways is not a controlled brief.

This is the part of the exercise that is easy to skip. It is much more comfortable to grade the models than to discover that the instrument handing them their work is inconsistent with itself. The pilot found the defects only because the criteria were applied to real production data rather than to a curated sample.

Fix 1: Declare which column is authoritative

Problem. The outline has a What to Include column (the assigned beats) and an Example column (sample prose). Two of three writers tracked the Example column and stopped where it stopped. The third followed the beat list, which continues further. Nothing in the document says which one governs.

Fix. Add this block to the top of every outline:

COLUMN AUTHORITY
  What to Include  = the assignment. Every beat listed here must appear.
  Example          = illustration only. Never copy it. It is not the assignment,
                     and it is not a completeness signal: it may be shorter than
                     the beat list.
  Synthesis Decisions = editorial commentary, written after the fact.
                     NOT instructions. Writers must ignore this block.

That last line matters most. See Fix 2.

Fix 2: Resolve the section-assignment contradiction

Problem. The What to Include column for ACT 2 does not mention a particular named figure. ACT 3 assigns her explicitly. But the Synthesis Decisions block at the foot of the same document says:

[original, ACT 2 integration] Merged her introduction into ACT 2 (not split across ACT 2 and ACT 3)...

Two of three writers put her in Act 2. One did not. The same document supports both behaviours, so it is impossible to say whether that is scope creep or obedience. This single contradiction decides C4 for two of the three sections.

Fix, and it is an editorial call rather than a technical one. Pick one and make the beat table match:

• Option A: keep her in ACT 3. Delete the ACT 2 integration line from Synthesis Decisions. Simplest, and it matches the section titles, since ACT 3’s title is her part of the story.
• Option B: move her into ACT 2, as the synthesis note intends. Then move the bullets themselves into ACT 2’s What to Include, remove them from ACT 3, and rebalance both word counts.

Recommendation: Option B, if the merge was a real editorial decision. The note reads like one, and it describes a deliberate narrative choice worth keeping. But the decision has to live in the beat table, not in a commentary block the writers were never told to read.

The general rule: if an editorial decision changes what a writer should produce, it belongs in What to Include. If it does not, it belongs nowhere the writer can see.

Fix 3: One word-count target per section

Problem. ACT 2’s row header says ~450 words. The verification block at the foot of the same outline says ACT 2: ~420. The three sections came in at ~420, ~420 and ~450, so the writers picked different targets from the same document.

Fix. The row header is authoritative. Regenerate the verification block from the row headers rather than maintaining it by hand, so the two cannot drift apart.

Until this is fixed, G1 is unscoreable. A gate with two different pass lines is not a gate.

Fix 4: Number the beats (recommended, not required)

Problem. Beats are nested: headed groups containing bullets. One act in the pilot episode has 4 headed groups containing 15 bullets. C1 counts beats, and the count is 4 or 15 depending on how the outline is read.

Fix. Number the groups and bullets explicitly. Illustrated below on the neutral demonstration topic used by the fixtures in the adversarial fixtures, so the convention can be shown without publishing production material:

ACT 1 - assigned beats

A. THE PROBLEM
   A1. 1937: shoppers carry hand baskets
   A2. A full basket is heavier than most people will carry
   A3. When the basket gets heavy, the shopping trip ends
   A4. Every heavy basket is a sale that never happened

B. THE INVENTOR
   B1. Store owner in Oklahoma City notices the pattern
   B2. Diagnoses it as a physical limit, not a demand limit
   B3. Builds a wheeled cart from a folding chair frame
   B4. Solves the engineering problem completely

C. THE REJECTION
   C1. Nobody uses it
   C2. Men refuse: pushing a cart reads as weak
   C3. Women refuse: it resembles pushing a pram
   C4. The social problem is harder than the engineering one

D. THE FIX
   D1. Hires actors of both sexes and various ages to push carts
   D2. Places a greeter at the door to hand out carts
   D3. Refusing now means refusing a person, not an object
   D4. Within a year the carts are never idle

Why this is worth doing. It gives the rubric an unambiguous unit at both levels: bullets for coverage (C1), groups for development (C2). It also makes grading machine-checkable, because a grader can return A1: present, A2: present, C4: absent instead of prose. And it costs nothing at authoring time.

Applying these

These are template changes, so they apply to every future episode. The pilot episode can be retro-fitted by hand for the run, or simply excluded from C4 and G1 scoring with the reason stated.

This document is part of the artifact on purpose. “The evaluation found defects in the instrument that produces the work, not only in the work” is a stronger finding than a model ranking, and it is the kind of thing that only turns up when someone actually applies a rubric to real production data instead of to a benchmark.

Download original

Public version. Three purpose-built test sections, published in full so a reader can see the criteria operating rather than taking the author’s word for it.

These are test fixtures. They are not pipeline output and they are not findings. They were written as test material, on a neutral topic, tied to no channel and no client.

Why these exist

The real generated scripts stay private. That leaves a reader unable to see the criteria actually working, and “trust me, I applied it” is the weakest sentence in any evaluation. Fixtures solve that: they can be published in full.

There is a second reason, independent of privacy. Real data gives you the distribution. Constructed cases probe specific failure modes on demand. Each case below is engineered to trip exactly one criterion, which is how you find out whether a criterion works at all.

Note on factual accuracy

Content is illustrative and not fact-checked. This is deliberate and does not weaken the fixtures, because C1 and C2 measure structure, not truth: claim distinctness and depth of treatment. Factual accuracy is C3’s job and is checked separately against the production research document. Do not cite these as history.

Design

Three sections on one brief, matched to roughly 300 words each, so that length cannot explain any score difference. Length matching is the point. It is what stops a verbosity-biased judge from passing the test by accident.

T2 and T3 are the two padding modes, separated. A judge that catches one and not the other is only half working, and an aggregate agreement score would hide that.

Brief used for all three: Act 1 opening, roughly 300 words, on how the shopping cart was adopted.

Status note. These fixtures were written against the original C2, which counted elaborating sentences per bullet. C2 has since been rewritten to grade at beat-group level. T3 is expected to still fail C2 under the revision, because twelve one-line assertions supply no mechanism, consequence, example, contrast or qualification, leaving every group undeveloped. That is a prediction, not a result, until it is run.

T1: Dense (control, should pass both)

ACT 1: Nobody Wanted It (~300 words)

Picture a grocery store in 1937. You are carrying a wire basket that cuts into your fingers, and it is half full, because half full is all you can lift. When it gets heavy, you stop shopping. You go and pay.

That last sentence was the whole problem, and one store owner was the only person who noticed it. Sylvan Goldman ran a chain in Oklahoma City, and he worked out that his customers were not leaving because they had everything they needed. They were leaving because their arms hurt. Every basket that got heavy was a sale that never happened.

So he built a cart: two wire baskets on a folding frame with wheels, borrowed from the folding chair in his office. By any reasonable measure it was a good idea. It solved a real problem that was costing him real money.

Nobody used it.

Men refused, because pushing a cart looked weak and they were not going to be seen doing it. Women refused too, because after years of pushing prams they had no appetite for pushing another one around a shop. Goldman had solved the engineering problem and walked straight into a social one, and the social one was harder.

His fix was to lie. He hired actors, men and women of different ages, and paid them to push carts around his stores looking perfectly content. He put a greeter at the door to hand a cart to every arriving customer, so refusing meant refusing a person rather than an object. Within a year the carts were never idle.

The shopping cart did not win because it was better. It won because Goldman manufactured the appearance that everyone else had already accepted it.

T2: Repetition-padded (should fail C1)

ACT 1: Nobody Wanted It (~300 words)

Picture a grocery store in 1937. You are carrying a wire basket, and it is heavy. It is genuinely heavy, the kind of heavy that you feel. Baskets in those days were heavy things to carry, and carrying them was hard work.

When the basket got heavy, shoppers stopped shopping. They would stop. The weight made them stop. Once a basket became too heavy to carry comfortably, the shopping trip was effectively over, because nobody wants to carry something uncomfortable. So they went and paid, and that was the end of it.

One store owner noticed this. Sylvan Goldman ran a chain in Oklahoma City, and he noticed what was happening in his stores. He saw the problem clearly. He understood it. He recognised that something was going wrong, and he was the one who spotted it when others had not.

So he built a cart. It was a cart, with wheels, designed for carrying shopping. The idea was that the cart would carry the things the basket had been carrying, so the customer would not have to.

It was a good idea. It was, in fact, a very good idea, and a sensible one, and it addressed something real. Good ideas like this one deserve to succeed.

Nobody used it. People did not use it. Adoption was essentially zero, and the cart sat there unused, because customers were not using it.

Goldman had a problem on his hands. The problem was that his solution was not being taken up, which is a problem when you have built something and nobody takes it up. He had built the thing. The thing was not being used. That was the situation he found himself in as 1937 went on.

T3: Breadth-padded (should pass C1, fail C2)

ACT 1: Nobody Wanted It (~300 words)

Picture a grocery store in 1937. Shoppers carried wire baskets. The baskets were small. They cut into your fingers. Most people could manage about fifteen pounds. Heavy baskets ended shopping trips early. Lost sales followed.

Sylvan Goldman ran a chain in Oklahoma City. He noticed the pattern. He had a background in retail. His stores were doing reasonably well. He wanted them to do better.

He built a cart. It used a folding frame. The design came from a folding chair. It held two wire baskets. It had four wheels. It could be stacked for storage. It cost very little to make. He patented it.

Nobody used it. Men thought it looked weak. Women associated it with prams. Older shoppers found it unfamiliar. Staff were unsure how to explain it. The carts sat by the entrance. Weeks passed.

Goldman hired actors. They pushed carts around the stores. They looked relaxed. They were different ages. Some were men. Some were women. He also hired a greeter. The greeter stood at the door. The greeter handed out carts.

It worked. Adoption climbed. Within a year the carts were in constant use. Stores had to buy more. Other chains copied the design. Goldman licensed it. He became wealthy. The cart got bigger over the following decades. Average basket size grew with it. Supermarkets redesigned aisles around cart width. The layout of retail changed permanently.

The lesson is about social proof. People follow other people. Objects need permission before they get adopted. Engineering was never the hard part. The hard part was persuasion. Goldman understood this before most people did. That insight was worth more than the patent.

How to use these

1. Hand-label all three against C1 and C2 first, blind if possible, before showing them to any judge.
2. Run each judge on all three. A judge that passes T2 or T3 is broken for this purpose regardless of its aggregate agreement number on real sections.
3. Report the results in full, so a reader can see the criteria operating.
4. Report per case, never averaged. A judge that catches repetition padding but not breadth padding has a specific, nameable weakness, and averaging hides it.

Known weaknesses of these fixtures

• Written by one author in one sitting, so they may share stylistic tells that a judge picks up on instead of the criteria. A judge could in principle learn “the padded ones sound flat” rather than applying C1 and C2.
• Deliberately extreme. Real padding is subtler than T2. These test whether a criterion works at all, not where its threshold sits. Threshold calibration needs real sections.
• One section type only. Section type appears to matter, so equivalent fixtures are needed for body sections and conclusions before the rubric can be applied across a whole script.

Download original

Public version. Run 2026-07-31 via the measurement code, published in full. Deterministic, no model calls, no API cost. One episode, nine sections, three models = 27 measurements. Thresholds: longest verbatim run greater than 12 words, or 8-gram overlap greater than 10%, fails a section. Either condition alone is sufficient. Mandated brand phrases are stripped from both texts before comparison. Three phrases: a required signature opening, a presenter self-identification, and a branded sign-off. Reproducing those is compliance with the outline, not copying. The phrases themselves are not published. Writers appear as Model 1, 2 and 3. The mapping to named models is not published, because the finding is not a model ranking.

Results

Each cell is longest verbatim run in words / proportion of the section’s 8-grams also present in the example column.

22 of 27 sections fail.

1. The pre-registered hypothesis was refuted

Written into the rubric before this data was seen:

“Two of the three writers hew close to the outline’s example column while the third paraphrases further.”

Wrong. The model identified as the paraphraser, on the evidence of Act 1 and Act 2, fails more sections than the other two: 8 of 9 against 7 of 9. The inference generalised from two sections to nine, and nine sections disagreed.

This is reported first, and prominently, on purpose. A refuted pre-registration is worth more than a confirmed one. It demonstrates that the prediction preceded the data rather than being fitted to it, and it is the clearest available evidence that the evaluation was not steered toward a convenient story. An evaluation whose every hypothesis is confirmed should be read with suspicion, not admiration.

2. Section-level variance dominates model-level variance

The same model copies heavily in one section and paraphrases well in another.

• Model 2’s ACT 2 has the lowest verbatim run in the entire table: 21 words, against 104 for the other two. The original Act 2 reading was correct, for Act 2.
• Model 2’s ACT 5 is a 210-word section with a 210-word verbatim run and 100% 8-gram overlap. The example column, reproduced in full, word for word.

The same model produced both. “Which model follows the paraphrase instruction” is therefore the wrong question. The better one is “which sections get copied”, and the pattern points at the examples rather than at the models. That reframing is the single most useful thing this run produced, and it is a direct consequence of measuring all nine sections instead of the two that prompted the hypothesis.

3. Only the opening passes reliably

All three models pass HOOK once the mandated signature phrases are stripped: runs of 5 to 7 words, 0% overlap. Given a one-line brief and three short example variants, all three wrote genuinely original copy.

The call to action is the threshold-sensitivity case. Models 1 and 3 pass cleanly (run 6, 0%). Model 2 fails on overlap alone: its run of 11 words is under the 12-word limit, but 16.7% overlap is over the 10% limit. A single marginal fail driven by one of two thresholds is a useful reminder that both numbers are author-chosen judgements, not derived values.

4. Confound: the example column is finished prose, not a sketch

This is the most important caveat in the report, and it is why the headline is not “the models disobeyed.”

The outline’s own Synthesis Decisions block shows the example column was assembled from prior complete drafts. It is polished, on-voice, correctly paced copy. Not a guideline, but a shippable draft.

Instructing a model not to copy a passage that is already exactly what is wanted is close to an impossible brief. This is plausibly a pipeline design finding rather than a model compliance failure.

The decision this forces

1. Accept it. If polished examples are what the pipeline wants, reproduction is the expected outcome, and C5 should not be framed as a violation check at all. Reframe it as a divergence measurement: a number that says how far each writer moved from the reference, with no pass line attached.
2. Make the examples skeletal. Bullet fragments and tonal notes rather than finished prose, leaving the model something to actually write. Then C5 measures compliance meaningfully.

Option 2 makes C5 a real criterion. Option 1 keeps the current pipeline and drops the pretence that the instruction is being followed. This decision is open at the time of publication. It is recorded here rather than resolved quietly, because which option is chosen determines whether the 22 of 27 figure means anything at all.

5. Limitations

• One episode, one topic, one outline author. Directional. Not a rate that generalises.
• Sections within one episode are not independent observations. 27 measurements, far fewer than 27 independent trials.
• One output per model per section. No repeat runs, so sampling noise is not separated from model behaviour.
• Both thresholds (12 words, 10%) are author-chosen and were not tuned. The call to action result for Model 2 shows the outcome is threshold-sensitive at the margin.
• N-gram overlap detects copying, not close structural paraphrase that reuses shape while changing every word. A writer who follows the example sentence by sentence with entirely different vocabulary scores as compliant.
• The measurement code is published; the input data is not. A reader can audit exactly how the numbers were produced but cannot reproduce them, because the scripts and the outline stay private. That is a real limit on verifiability and it is stated rather than glossed.
• Model identities are known to the author. Irrelevant to this particular result, since the measurement is deterministic code with no judgement in it, but stated for completeness.

6. What this result actually supports

Defensible:

“Across one episode of nine sections, three frontier models reproduced the outline’s example prose beyond a 12-word verbatim threshold in 22 of 27 sections. Copying tracked the section rather than the model, and one section was reproduced in full. The example column consists of finished prose, which the report treats as the likeliest cause.”

Not defensible: any statement ranking the models on instruction-following, any pass rate presented as generalising beyond this episode, and any claim that the models “ignored” the instruction without noting the confound.

Download original

The C5 check exactly as run. No model is called: it is deterministic text measurement, so it returns the same numbers every time.

/**
 * C5 - paraphrase compliance check.
 *
 * Measures how closely each generated section reproduces the outline's Example
 * column for that same section.
 *
 * Two measures per section:
 *   1. longest verbatim run - longest continuous sequence of matching words
 *   2. 8-gram overlap       - proportion of the section's 8-grams that also
 *                             appear in the example text
 *
 * C5 fails a section if EITHER the longest verbatim run exceeds MAX_RUN words,
 * or 8-gram overlap exceeds MAX_OVERLAP.
 *
 * Mandated brand phrases (a required signature opening, a presenter
 * self-identification, and a branded sign-off) are stripped from BOTH texts
 * before comparison. Reproducing them is compliance with the outline, not
 * copying; leaving them in would report the hook as the worst offender in the
 * script. The phrases live in mandated_phrases.txt, one per line, which is not
 * published.
 *
 * PUBLIC RELEASE NOTE
 * This is the measurement code exactly as run. The two input files it reads,
 * sections.md (generated script sections) and examples.md (the outline's
 * example column), are production material and are NOT published, so the
 * published numbers can be audited for method but not re-run. Both are plain
 * markdown in the formats parsed below, so the script runs against any
 * equivalent pair.
 *
 * Where a section has several example variants, the section is compared against
 * EACH variant and the WORST result is kept: a writer copying variant 2 is
 * copying regardless of what variants 1 and 3 say.
 *
 * No model is called. This is a deterministic text measurement.
 *
 * Usage:  node c5-paraphrase-check.mjs
 */

import { readFileSync, existsSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";

const HERE = dirname(fileURLToPath(import.meta.url));

const MAX_RUN = 12;       // words
const MAX_OVERLAP = 0.10; // proportion
const NGRAM = 8;

function loadMandated() {
  const f = join(HERE, "mandated_phrases.txt");
  if (!existsSync(f)) return [];
  return readFileSync(f, "utf8")
    .split(/\r?\n/)
    .map((l) => l.trim())
    .filter((l) => l && !l.startsWith("#"));
}

/** Parse "# SCRIPT X" / "## SECTION" into { script: { section: text } }. */
function parseSections(path) {
  const out = {};
  let script = null, sec = null, buf = [];
  const flush = () => {
    if (script && sec) {
      out[script] ??= {};
      out[script][sec] = buf.join("\n").trim();
    }
  };
  for (const line of readFileSync(path, "utf8").split(/\r?\n/)) {
    if (/^# (?!#)/.test(line)) { flush(); buf = []; script = line.slice(2).trim(); sec = null; }
    else if (/^## (?!#)/.test(line)) { flush(); buf = []; sec = line.slice(3).trim(); }
    else buf.push(line);
  }
  flush();
  return out;
}

/** Parse "## SECTION" / "### VARIANT n" into { section: [variant, ...] }. */
function parseExamples(path) {
  const out = {};
  let sec = null, buf = [];
  const flush = () => {
    if (sec) {
      const t = buf.join("\n").trim();
      if (t) (out[sec] ??= []).push(t);
    }
  };
  for (const line of readFileSync(path, "utf8").split(/\r?\n/)) {
    if (/^## (?!#)/.test(line)) { flush(); buf = []; sec = line.slice(3).trim(); }
    else if (/^### /.test(line)) { flush(); buf = []; }
    else buf.push(line);
  }
  flush();
  return out;
}

function normalise(s, mandated) {
  let t = s.toLowerCase();
  for (const p of mandated) t = t.split(p.toLowerCase()).join(" ");
  t = t.replace(/[‘’“”—–]/g, " ");
  t = t.replace(/[^a-z0-9\s]/g, " ");
  return t.replace(/\s+/g, " ").trim();
}

/** Longest common contiguous word sequence. */
function longestRun(a, b) {
  if (!a.length || !b.length) return { len: 0, text: "" };
  let prev = new Array(b.length + 1).fill(0);
  let best = 0, bestEnd = 0;
  for (let i = 1; i <= a.length; i++) {
    const cur = new Array(b.length + 1).fill(0);
    const ai = a[i - 1];
    for (let j = 1; j <= b.length; j++) {
      if (ai === b[j - 1]) {
        cur[j] = prev[j - 1] + 1;
        if (cur[j] > best) { best = cur[j]; bestEnd = i; }
      }
    }
    prev = cur;
  }
  return { len: best, text: a.slice(bestEnd - best, bestEnd).join(" ") };
}

function ngrams(ws, n) {
  const s = new Set();
  for (let i = 0; i + n <= ws.length; i++) s.add(ws.slice(i, i + n).join(" "));
  return s;
}

function compare(sectionText, exampleText, mandated) {
  const a = normalise(sectionText, mandated).split(" ").filter(Boolean);
  const b = normalise(exampleText, mandated).split(" ").filter(Boolean);
  const run = longestRun(a, b);
  const ga = ngrams(a, NGRAM), gb = ngrams(b, NGRAM);
  let hit = 0;
  for (const g of ga) if (gb.has(g)) hit++;
  return { words: a.length, run: run.len, runText: run.text, overlap: ga.size ? hit / ga.size : 0 };
}

const pad = (s, n) => String(s).padEnd(n);
const padL = (s, n) => String(s).padStart(n);

function main() {
  const mandated = loadMandated();
  const scripts = parseSections(join(HERE, "sections.md"));
  const examples = parseExamples(join(HERE, "examples.md"));

  console.log(`C5 paraphrase compliance   MAX_RUN=${MAX_RUN}  MAX_OVERLAP=${(MAX_OVERLAP * 100).toFixed(0)}%  n-gram=${NGRAM}`);
  console.log(`mandated phrases excluded: ${mandated.length}\n`);
  const header = pad("script", 12) + pad("section", 16) + padL("words", 6) + padL("run", 6) + padL("8gram", 8) + "  verdict";
  console.log(header);
  console.log("-".repeat(header.length));

  const totals = {};
  for (const [script, secs] of Object.entries(scripts)) {
    for (const [sec, body] of Object.entries(secs)) {
      const variants = examples[sec];
      if (!variants) { console.log(pad(script, 12) + pad(sec, 16) + padL("", 6) + padL("", 6) + padL("", 8) + "  NO EXAMPLE"); continue; }
      let worst = null;
      for (const v of variants) {
        const r = compare(body, v, mandated);
        if (!worst || r.run > worst.run || (r.run === worst.run && r.overlap > worst.overlap)) worst = r;
      }
      const fail = worst.run > MAX_RUN || worst.overlap > MAX_OVERLAP;
      (totals[script] ??= []).push(fail);
      console.log(pad(script, 12) + pad(sec, 16) + padL(worst.words, 6) + padL(worst.run, 6) + padL((worst.overlap * 100).toFixed(1) + "%", 8) + "  " + (fail ? "FAIL" : "pass"));
      if (fail) console.log(" ".repeat(34) + `longest run: "${worst.runText.slice(0, 100)}..."`);
    }
    console.log("");
  }

  console.log("summary");
  console.log("-".repeat(48));
  for (const [script, v] of Object.entries(totals)) {
    const f = v.filter(Boolean).length;
    console.log(pad(script, 12) + `${f} FAIL of ${v.length} sections (${Math.round((f / v.length) * 100)}%)`);
  }
}

main();

Download original