Part 2 of a five-part series: From Cloud AI to Local Models.
The most common dismissal of local models is also the least interesting one:
"They're slower."
Sometimes they are. Sometimes they are not. More importantly, that sentence is usually missing the only word that matters.
Slower at what?
Speed is not a single property. A model can be slower at generating one perfect answer and faster at making the engineering workflow move. A frontier model can win every benchmark and still be the wrong thing in the loop. A smaller local model can be objectively less capable and still make the system faster because it is closer, cheaper, more available, and easier to call repeatedly.
The question is not "is the model slower?"
The question is "is the system faster?"
Cloud latency is not just inference latency
When people compare cloud and local models, they often compare the wrong two numbers.
They imagine the cloud model's raw generation speed on one side and the local model's raw generation speed on the other. But the user does not experience raw inference. The user experiences the whole path.
For a hosted model, that path includes:
- network round trips
- provider queueing
- rate limits
- request serialization
- context upload
- policy and safety layers
- occasional retries
- regional availability
- the provider's current load
None of these make the cloud model bad. They are just part of the actual system. If the model is being used in a chat window, the difference may not matter. If it is inside a tight agent loop that makes hundreds of small decisions, it matters a lot.
The moment the model becomes part of automation, latency stops being about one answer. It becomes about the cadence of the loop.
Can the agent inspect a file, make a small plan, run a tool, read the result, adjust, and continue without every step feeling like a remote procedure call to another company?
That is a different speed question.
Local latency is real, but it is tunable
Local models have their own latency story.
They depend on hardware, model size, quantization, context length, runtime, batching, memory bandwidth, and whether someone chose a model because it fit the task or because it looked good on a leaderboard. A bad local setup can be painfully slow. A model that is too large for the machine is not infrastructure; it is an expensive pause button.
But local latency has one property cloud latency often does not: you can shape it directly.
You can choose a smaller model. You can quantize. You can keep context small. You can route short tasks to fast models and hard tasks to slower ones. You can pin the runtime near the repository. You can warm the model. You can run parallel local workers for bounded jobs. You can decide that this particular workflow needs a 1-second answer more than it needs the strongest possible reasoning.
That last decision is the one teams underuse.
Not every AI task deserves the smartest model you can rent.
Overkill is a performance problem
Engineering work is full of small tasks that do not need a frontier model.
Rename this symbol consistently.
Explain this function.
Generate a first-pass test.
Classify this log line.
Summarize this diff.
Find the likely owner.
Rewrite this comment in the house style.
Extract the migration steps from this document.
None of these are trivial, but many of them are bounded. They have local context, clear constraints, and a cheap way to check the result. Paying the latency and cost of a frontier model for every one of them is like calling a principal engineer into a meeting to rename a variable. Sometimes the answer is good. The resource choice is still strange.
This is where smaller local models become useful. They do not have to be the best general intelligence in the world. They have to be good enough for the task, fast enough for the loop, and cheap enough that the team is willing to use them everywhere the workflow benefits.
That changes behavior.
When every failed cloud call is a billable event, experimentation has friction. When every attempt waits behind a remote service, the loop slows down. When a local model can be called casually, repeatedly, and privately, people build different systems around it.
Agentic systems care about iteration rate
The speed question gets sharper with agents.
An agent rarely does one giant model call and finishes. The useful ones tend to work in small steps:
- inspect the repository
- read a file
- form a hypothesis
- run a command
- interpret the result
- edit a small slice
- verify it
- adjust when the verification fails
That loop is where AI-assisted engineering either feels fluid or feels like waiting for a remote service to think out loud.
For agentic work, the unit of performance is not tokens per second on a single answer. It is useful steps per minute, under the team's constraints. A model that is less brilliant but always available, cheap to call, and close to the tools can win because it lets the system keep moving.
This does not mean every agent should be local. Hard planning, ambiguous debugging, security review, architecture tradeoffs, and cross-domain reasoning may justify a stronger cloud model. But a lot of agent steps are not that. They are inspection, classification, summarization, ranking, and bounded generation.
If you send all of those to the same frontier endpoint, you are not designing a system. You are outsourcing the routing decision to habit.
Availability is part of speed
There is another speed teams forget until the day they need it: availability.
A model that responds in 500 milliseconds when the provider is healthy and not at all during an outage has a very different latency distribution than a local model that responds in three seconds consistently.
For some workflows, that does not matter. For others, it matters a lot.
Incident response is the obvious one. If the AI assistant is part of how your team summarizes logs, searches runbooks, explains alerts, or drafts updates, then the worst possible time to discover a provider dependency is during the same incident that is already stretching the team. The assistant does not need to be perfect in that moment. It needs to be available, allowed to see the context, and fast enough to keep the response moving.
Availability is not separate from performance. It is performance over time.
The useful benchmark is the workflow
The benchmark that matters is not "which model is faster in isolation?"
It is:
- How long does the code review take?
- How quickly can the agent finish a safe refactor?
- How many test-generation attempts can run before the developer loses patience?
- How fast can the incident assistant read local runbooks and logs?
- How many internal docs can be summarized without creating a privacy review?
- How much experimentation happens when failed attempts are cheap?
Those are system-level questions. They include model quality, but they do not end there.
The best deployment model is the one that makes the workflow faster without hiding unacceptable tradeoffs. Sometimes that will be a frontier model in the cloud. Sometimes it will be a small local model in the editor. Sometimes it will be a router that starts local, escalates when confidence drops, and saves the expensive call for the moments where the expensive call earns its keep.
The reframe
"Local models are slower" sounds precise, but it usually collapses half a dozen different questions into one vague objection.
Slower at raw generation?
Slower after the network, queue, and rate limit?
Slower for one hard answer, or faster across a thousand small steps?
Slower when the provider is healthy, or faster when the provider is down?
Slower for a benchmark, or faster for the engineering system?
That last one is the question worth keeping.
The future of AI-assisted engineering will not be decided by one model being globally faster than another. It will be decided by teams that understand where speed actually lives in their workflows and place models accordingly.
For some work, the fastest answer is the smartest model in the cloud.
For a lot more work than people expect, the fastest answer is the model already close enough to the code to answer now.
Previous: Part 1 — The Cloud Was the First AI Deployment Model Next: Part 3 — Capability Is Not a Single Number