While testing for models for tool selection compliance, I got what looked like a simple result: the model was failing the evaluation.
Unbeknownst to me, the harness expected a particular tool-call representation. I swapped in a different model and kept that expectation. Its tool calls used a format the harness did not recognize, so a valid selection never reached the part of the evaluation that could judge it looking like a failure.
I didn't see that at the time. The score only said that the run had failed. Trace observation made the missing boundary visible later.
After the evaluator accepted the model's actual format, tool selection proved more reliable than the first run had suggested. Other parts of the run still failed. The revised evaluation identified where the model still needed work.
Tool selection has a prerequisite: the harness must recognize a tool call before it can judge the choice.
An evaluation observes a path
An evaluation does not observe a model's internal decision directly. It observes output that has travelled through a path.
The model receives a prompt and responds. The system interprets that response as a tool event. The evaluator compares the event with its policy. The runtime may then execute the tool, return a result, and observe what the model does next.
Each step has its own potential for failure. A model can choose the wrong tool. It can choose the right tool with invalid arguments. The parser can fail to recognize a valid call. The adapter can map a call into the wrong internal shape. The tool can fail after a correct selection. The model can then mishandle the tool result or fail to recover from the error.
An aggregate failure score collapses those outcomes into one word. That is fine for a regression gate after the path is understood. It is weak evidence while a new model, template, or adapter is being introduced.
The first run in this case had a format failure in the evaluator path. Calling that a tool-selection failure would have sent the next tuning step toward the wrong data or the wrong model setting.
Format belongs in the contract
Tool use is often described as a capability. That description hides the contract around the capability.
The model needs a prompt format that tells it which tools exist and how calls should be expressed. The harness needs a corresponding rule for recognizing the returned call. Those two sides have to agree on names, arguments, role markers, and the representation that carries a tool result back into the next turn.
One model may produce a call in the shape the harness was built around. Another may use a different representation while still following its own expected format. A model swap can materially change the effective interface even when the task, available tools, and expected choice stay the same.
That is why a tool-use harness should treat its rendering and parsing rules as versioned evaluation inputs. They are as important to the result as the prompt, the available tool set, decoding settings, and expected answer.
The contract also needs an owner. A model-specific adapter can normalize calls into one internal event shape. A shared evaluator can then judge tool selection against that shape. The boundary is useful because it prevents provider details from leaking into every evaluation case while keeping the translation visible and testable.
Traces turn a score into evidence
The answer came from observing the trace rather than trusting the final score.
A useful tool-use trace records the rendered request, the raw model response, the parsed tool event, the selection verdict, the execution result, and the next model response. It should identify the template, adapter, and evaluator version that produced those records.
That record answers concrete questions. Did the model emit a call? Did the harness parse it? Which tool name and arguments did it extract? Did the policy accept the call? Did execution succeed? Which later step actually failed?
Without that sequence, investigation tends to start with the weights because the model is the visible component. In this case, the evaluator was the first component that needed changing. It needed to understand the output it had been given.
Trace observation does more than find harness defects. It also prevents a format fix from becoming an excuse to call the model ready. Once the parser recognized the call, the run could show that tool selection was comparatively reliable and that other stages still needed attention. A single configuration edit cannot establish general reliability. The evaluation could now locate the remaining failures.
Strictness needs layers
Strict tool selection is a worthwhile test, especially when a wrong call can change data, consume money, or cross an authority boundary. It should be a named layer of the evaluation rather than a proxy for every part of tool use.
The selection layer asks whether the model chose an allowed tool for the task. The format layer asks whether that choice crossed the model-to-harness boundary correctly. Argument validation asks whether the call is safe and complete. Execution asks whether the tool actually performed the requested work. The follow-up layer asks whether the model used the result correctly or recovered when the tool failed.
Those layers can share one run and one trace. They should retain separate verdicts. A team can then say that selection is reliable for a model and template combination while execution recovery remains weak. That statement is far more useful than a single pass rate.
It also changes the next experiment. A parser failure calls for an interface fix and a regression case around the emitted format. A selection failure may call for different supervision, a clearer tool description, or a narrower tool set. An execution failure belongs in the runtime path. Each result points at a different owner and a different remedy.
The evaluator is part of the system under test
Models do fail tool-use evaluations. Harnesses also fail to observe models correctly. Both facts can be true in the same run.
Treating the evaluator as part of the system under test keeps those facts separate. It requires format fixtures for each supported model path, trace-level inspection when a new path is added, and regression cases that prove the normalizer emits the expected internal event.
That effort is smaller than tuning a model against a false signal. It also makes later failures more valuable because the team can trust that the reported layer is the one that actually broke.
The one-line version
A tool-use score is only meaningful after the evaluator can distinguish a model that chose the wrong tool from a harness that failed to read the right one.