I have been turning a batch of meeting transcripts into SFT data, so a small model can write meeting notes against a fixed template.
After the pipeline was running and the filters were written, I checked something I should have checked much earlier: judge the same note twice, same judge model, same parameters. The first run reported zero factual errors. The second reported more than a dozen.
Here is what was worth writing down.
The human note is not ground truth
My first plan was to pair each transcript with its human-written note. That does not work.
A human note is what somebody typed from memory after the meeting. It skips decisions, conflates items, and sometimes contains things that never appear in the transcript because they were agreed beforehand. Train on it and the model learns somebody else’s recall errors.
What you actually want is a different pair: the transcript, and a note where every sentence can be traced to a span of that transcript.
The base model’s problem is not format, it is filling blanks
The base model’s first draft looks structurally fine. Checking line by line against the transcript is where it falls apart: the transcript never says who chaired, so it invents a name; an action item has no deadline, so it writes “this week”; a discussion that ended unresolved becomes “the meeting decided.”
All of that passes a format validator. They are hallucinations, and they read as perfectly consistent with everything around them.
Three roles, one job each
- Student: the base model being fine-tuned. Produces the draft.
- Judge: a large model that receives only the transcript and one unattributed candidate note. It never sees the human note. It splits the note into claims and rules on each — supported, partially supported, unsupported — then checks in reverse for anything important the note dropped.
- Teacher: a second large model that rewrites, given the transcript, the draft, and the judge’s verdict.
flowchart TD
A["Meeting transcript"] --> B["Student model
writes a draft"]
A --> C
B --> C["Judge
checks every claim"]
C --> D["Teacher
emits structured fields"]
A --> D
D --> E["Renderer
deterministic layout"]
E --> F["Judge re-checks
three independent votes"]
F --> G{"Three gates"}
G -->|"format passes
zero factual errors
retention above floor"| H["Into the training set"]
G -->|"any gate fails"| I["Rejected
reason recorded"]
The teacher does not write Markdown
This is the most useful piece of the design. The teacher emits fields only — what the topic is, who owns this action item, whether a field is known or not mentioned — each carrying an evidence pointer into the transcript. Code does the layout.
The model says “chair: not mentioned” and the renderer writes the mandated placeholder. The model says “no action items” and the renderer emits a one-row table with the required sentinel.
Format compliance is therefore 100%. Not because the model got obedient, but because the format never goes through it.
Side benefit: the same fields render into any format. When the template revs, you change the renderer and regenerate nothing.
Assign splits before deriving any target
Meeting data comes in series. The same weekly meeting recurs with the same people and the same project, and last week’s action items are still open.
Split individual meetings at random and your test set holds other instalments of a series the model trained on. Part of the test score is then just recall of last week, and the metrics will not show you that.
So split by series, whole group on one side, and freeze it before generating a single target. Once the test identities are fixed and hashed, no later pipeline change can leak into training.
The matching rule: anything descended from an evaluation benchmark stays out of training. The judge’s verdict on it, the teacher’s rewrite, the rendered note — all marked evaluation-only, and that marking only ever gets stricter.
Retention: too much rewriting is its own failure
Three gates: format, zero factual errors, retention. The third is the one I underrated.
Retention measures how much of the student’s already-correct wording the teacher kept.
More rewriting is not better. Fine-tuning corrects a specific behaviour on top of what the base model does; it is not supposed to replace its voice. If the teacher rephrases everything in its own style, the data teaches two lessons — stop inventing facts, and also talk like me. The second one burns training budget and dilutes the signal you wanted: keep what has evidence, mark what does not as unknown.
One long meeting passed the factual gate cleanly, formatted fine, and retained about a third of the original wording. The teacher had rewritten it from scratch. Rejected — not for being wrong, for teaching more than intended.
The judge is not reproducible
Temperature zero, greedy decoding. I expected two identical calls to agree. They did not:
- First pass zero factual errors, second pass more than a dozen
- And the reverse: first pass a dozen, second pass two or three
Several causes stack. Server-side batching drops identical requests into different batches, which changes the numerics. Prefix caching changes how the forward pass is chunked. With multiple replicas behind a balancer, two requests may not hit the same process. And “check a long document claim by claim” sits at the edge of what the model does reliably, with long output, so an early divergence gets amplified.
Gating on “zero factual errors” from one verdict is a coin flip. Some rows I had marked as passing had simply got lucky — one pass said zero, a re-check found plenty.
Three votes, unanimous or nothing
The fix is unsubtle: judge each rewrite three times, require all three to report zero errors, reject on any dissent.
The rule is biased, in the right direction. Noise kills good rows (one of three votes drew badly) but cannot admit bad ones (a genuinely broken note gets caught by at least one vote). For training data, less is fine and dirty is not.
That matches what I see: clean rows come back zero, zero, zero; problematic rows produce wildly varying counts. The noise is itself a signal — an unstable verdict usually means the row sits at the edge of the judge’s competence. Since you are judging three times anyway, record the spread as a confidence score and put the high-variance rows in front of a human.
Reproducible is not the same as trustworthy
Two ways to make a pipeline rerun the same, at very different costs.
Real determinism: the serving stack guarantees identical outputs for identical requests. Needs batch-invariant kernels, prefix caching off, and one of a short list of supported attention implementations. Of my two models the teacher managed byte-for-byte equality and the judge did not — the sparse attention implementation it depends on is not on that list.
Replay: hash every model call by request content, store the response, and replay the cache on rerun.
Replay is far cheaper and does not depend on the server. But it guarantees that this run used last run’s verdict, not that the verdict is any good. Tuning a threshold makes the difference obvious: replay re-derives results for free, but if you want to know whether a verdict is stable you have to actually judge again.
Keep both. Replay for auditability, votes for trust.
Notes to self
An evaluation model used constantly and evaluated never is dark matter in your pipeline. The cheapest check is running it twice.
Models produce meaning, code produces format. That holds for anything with a fixed output structure.
Separate “asserted something the transcript never said” from “omitted something the transcript contains.” The first is hallucination and has to be rejected. The second is incompleteness. Collapsing them into one gate throws away a lot of usable data.
This pipeline yielded far less data than I expected. The useful output was not the data — it was finding out which stages were untrustworthy. The day it first ran end to end I was staring at the pass rate and reaching for the thresholds, still treating a noisy measurement as ground truth.
