The model swap that failed every evaluation

We had all our evaluation tests passing for Smoketest’s browser agent. Then I tried a different model, one I expected to be more capable, and every one of those same tests failed.

We paused the migration. I had expected to compare models, but the results made me look more closely at the system we had built around the one we were already using.

Smoketest turns plain-English instructions into software tests across surfaces, including browsers, CLIs, APIs and mobile apps. This experiment involved its browser agent, which works through tasks on websites using browser tools. The model helps decide what to do next, how to interpret the result, and when the task is finished.

Those decisions depend on more than a model’s general capability. They also depend on how it behaves inside the system you’ve given it. And by the time that system works reliably, you’ve probably made quite a few decisions around that behavior.

How the dependency builds up

You start with a model and build something around it. You run evaluations, inspect failures and make changes until the application does what you need.

A tool description might need to be more explicit. An instruction might need an example. You might change what information a tool returns, or how the agent recovers when an action fails. Each change solves a problem you encountered with the model you’re using.

Over time, those decisions add up. The prompts, tools and control flow begin to fit that model’s behavior. You don’t have to deliberately write model-specific code for this to happen. You’re improving the application through a feedback loop, and one particular model is part of that loop.

That’s the dependency I hadn’t given enough weight to. Passing evaluations tells you that the current combination works. It doesn’t establish that the surrounding system is equally suitable for another model.

An API wrapper can make the replacement easy to connect. It can’t guarantee that the replacement will interpret the same instructions or use the same tools in the same way.

Similar capability, different behavior

When I say models “think differently,” I mean the differences we can observe while they work: how they approach a task, what they do with incomplete information, which tools they choose, and whether they keep going after an unexpected result.

Two models can have similar benchmark scores without making similar decisions in your application. A score summarizes performance on a set of tasks. It doesn’t describe every behavior your system may have come to rely on.

After pausing the migration, we evaluated several other models. We found that moving from an earlier Gemini to a newer Gemini more or less preserved our successful evaluation results, with what we observed to be better reasoning. Switching to an OpenAI model, on the other hand, broke the setup.

In our experiments, the newer model from the same provider behaved more like the one the agent already worked with. That made it a less disruptive change. I wouldn’t assume that every same-provider upgrade will behave this way, but the contrast helped explain why greater capability alone wasn’t enough to make a replacement work for us.

Model flexibility has to be built and maintained

You can deliberately design an application to support different models. But that means testing those models as the application evolves and allowing for differences in their behavior, not just making the model name configurable.

That flexibility takes work. Depending on the application, it might mean different instructions, tool descriptions or recovery behavior for different models. If you only evaluate one model while improving the system, you have little evidence that the alternatives still work with it.

For a customer-facing product, keeping every model interchangeable may not be the priority. Someone using Smoketest needs their software tested correctly. They shouldn’t have to care which model is doing the work in the background. It’s reasonable to focus on a combination that gives them reliable results at an acceptable cost and speed.

The part worth keeping in mind is what that choice means for a later migration. A new model may need changes to the system around it, even if it is more capable. The work you did to make the old setup reliable doesn’t automatically transfer.

We eventually chose the best balance of cost, accuracy and speed among the models we evaluated. But the failed swap changed how I think about that choice. We had spent time getting an agent to work well with a particular model. Trying another meant checking which of those decisions still made sense, and being prepared to change the ones that didn’t.