A representative evaluation gives a tuning team a stable comparison between a candidate adapter and a known base. It can show whether a dataset changed tool use, format adherence, domain recall, or refusal behavior.
Its result says what the suite observed. The claim reaches only as far as the suite's coverage and data quality.
One tool-use evaluation made this concrete. The original score treated a model as failing strict selection because the harness did not recognize its tool-call format. Traces exposed the mismatch. Once the evaluator accepted the model's actual format, selection proved reliable while later execution and follow-up stages still failed. The score had hidden separate findings that required different fixes.
Representation follows the work
An evaluation becomes representative by covering the work the model is expected to do. Randomly sampled examples may preserve the common request shape while missing incomplete context, ambiguous tool arguments, policy boundaries, stale state, long inputs, or a local format that downstream systems rely on.
The categories should come from the job. A tool-using specialist may need valid calls, missing arguments, disallowed calls, recovery after a failed command, and handoffs where the model should ask for more information. A document workflow may need overloaded requests, conflicting sources, and cases where the answer must state uncertainty.
One well-chosen case can reveal a failure. A pattern needs several cases. The suite needs routine work, difficult work, and the conditions where a plausible response would still cause harm.
Small sets hide local movement
Specialist datasets often have specialist evaluations. A small suite is a reasonable starting point, provided the result is read at the same level of detail as the behavior it measures.
A tuning change may help a narrow class while leaving the aggregate score almost unchanged. It may improve one category and regress another by the same amount. One unusual completion can also move a small score enough to look important.
Report results by the categories the suite was designed to cover. Compare the same cases against the baseline and candidate, then read the changed outputs. The score can support a decision when the movement appears in the behavior the product relies on. It can also support a decision to collect more cases when the current set cannot distinguish the models.
The evaluator has data failures too
Expected answers can go stale after a workflow changes. Labels can be copied from an earlier case. A chat template can render the wrong request. A parser can accept malformed output or reject a correct response. Duplicate rows can give a familiar pattern more weight than intended.
Leakage creates a different false signal. A held-out example may be nearly identical to a training row, or a generated corpus may already contain the expected answer. The adapter then appears to generalize from a completion it already knows.
Preserve the source and revision of every evaluation row. Keep training and evaluation sets separate by more than a filename. Check duplicates and near-duplicates. Version the rendering template and scorer. When a result moves in a surprising way, review passing and failing outputs before changing the training data.
Compare behavior before choosing a checkpoint
The baseline belongs in every useful evaluation. A candidate run shows whether the model can produce an acceptable response. A baseline comparison shows whether the training change earned its added cost and risk.
Use the same template, decoding settings, tools, and evaluator version for both runs. Otherwise the surrounding system becomes another variable in the result. Some cases also need direct review. Tool arguments, safety boundaries, and explanations can fail while an exact-match scorer reports success.
Evaluation guides the next data revision. An unexpected failure can reveal a missing class of examples. A regression can show that one response style has become too dominant. A suspicious improvement can send the team back to the evaluator instead of the weights.
The value of a representative suite is that it makes those questions cheaper to ask again with evidence attached.
The one-line version
A representative evaluation is only as useful as the work it covers and the evidence behind each result.