notes on distributed systems, infrastructure, and production engineering
Short, practical writing on systems work: schedulers, data pipelines, observability, performance, infrastructure, and the tooling around it.
AI can raise the quality of an artifact while quietly weakening the judgment that produced it. The risk is real, but the outcome depends on how the tool is used and which parts of the work we continue to own.
An AI harness is the runtime that turns a request into controlled and observable work. Following one failing test exposes the parts hidden behind the term.
AI makes proofs of concept cheap enough to participate in system design. Exploratory harnesses expose boundaries before the planned implementation begins.
Representative evaluation sets are essential for tuning, yet a small or corrupted suite can hide a real improvement or report one that does not exist.
Hand-authored training data is slow to create. Conversation history can provide high-signal raw material when it is curated with provenance, review, and clear boundaries.
A tool-use evaluation can only judge model behavior after its harness recognizes the model's emitted tool-call format.
Prompt tokens are part of every training sequence. Reducing unnecessary context makes fine-tuning faster, cheaper, and easier to evaluate.
Fine-tuning improves through controlled changes evaluated against explicit behavior and preserved experiment evidence.
A model can only act on memory that the application retrieves and includes in the current request. Storing a preference is only the first step.
A model can sound aware of ongoing work when retrieval finds the relevant memory and assembles it into the current context. A hybrid AST, syntax, and reranking path makes that retrieval faster and more precise.
Streaming alerts need durable evidence beyond threshold breaches, so operators can inspect what happened after the market has moved.
Frontend hosting becomes part of application behavior when it determines page speed, streaming, runtime configuration, and the shape of production failures.
Batch and live market-data paths share domain concepts while relying on different runtime contracts for reconstruction, continuity, and failure recovery.
Ingestion is only the visible part of a market-data system. The durable work is making moving data replayable, inspectable, bounded, safe, and explainable.
A market-data product becomes a distributed system when it starts depending on durable coordination, replayable work, live state, deployment boundaries, and explanations operators can trust.
Agent observability has to capture runtime decisions along with infrastructure health.
When code volume outpaces human review capacity, testing strategy has to change. The old model doesn't scale.
The total cost of a managed service includes the dependency, coupling, and operational surface you inherit.
The outage gets the attention. The first 15 minutes decide the outcome. The postmortem decides whether it happens again.
Conway's Law can be used deliberately when team structure, ownership, and communication paths are part of the architecture discussion.
Latency budgets are product decisions with architectural consequences.
AI gets you to launch fast. What happens two years later when nobody understands the system it built?
Technical debt only becomes manageable when teams distinguish strategic debt from accidental debt.
AI made production cheap. It made verification expensive. That's the bottleneck now.
When AI makes tactical competence cheap, the value of seniority shifts to judgment, verification, and teaching the meta-skill.
Caching improves reads by adding consistency, ownership, and recovery questions to the system.
You can build anything. Can you move it? Migration is where abstractions prove themselves or fail.
A lot of useful AI work should not stay AI work forever. Once the task resolves into explicit, repetitive, mechanically checkable rules, the better runtime is usually deterministic automation.
The hard part of deploying agents is not mostly reasoning quality. It is deciding what they are allowed to do, when they must ask, what they must prove before acting, and how you recover when they are wrong.
AI is compressing the market value of tactical execution. The people who stay relevant will be the ones who learn to direct the work, judge the output, and keep building real mastery instead of outsourcing the reps that create it.
A lot of the panic around reviewing AI output treats it like a new burden. It isn't. If you've spent years reviewing other people's code, libraries, and infrastructure decisions, you already know the job. AI just makes the need for that judgment harder to ignore.
AI can be useful without justifying current spending, pricing, or hype.
You can own a service and still abdicate responsibility for the outcomes it produces. Real ownership means caring about the consequences of your work, not just the artifacts.
The strongest argument for local and self-hosted models is not cost. It is the ability to turn company-specific knowledge into working engineering infrastructure.
Cloud AI pricing is easy to tolerate when AI is occasional. Once AI becomes part of every engineering workflow, the economics change.
Model capability depends on the task, the context, the tools, and the evaluation loop. The model alone is not the product.
A cloud model can be smarter and still make the workflow slower. Speed is not a benchmark number. It is a property of the whole system.
Hosted frontier models were the first practical way most teams touched AI. That does not make the cloud the final shape of AI-assisted engineering.
AI outages feel like a new kind of problem. They aren't. They're the same dependency risk we've always had, wearing a new label. The question that matters isn't whether AI is reliable. It's what happens to your business when the dependency disappears.
For years software companies built moats around technology that was hard to build. AI is dissolving that assumption. When intelligence is rented instead of owned, the model stops being a moat and becomes infrastructure. The durable advantage is whatever is harder to copy than intelligence itself.
There's a growing consensus that whoever wields the best model wins the AI revolution. I don't buy it. Code is becoming a commodity, and when output is free its value approaches zero. The advantage was never the model. It's the understanding the model can't supply.
For years I thought the goal was to become the person everyone depends on. It isn't. If the system only works because you're there, you haven't built leverage, you've become its failure mode.
The bugs that bankrupt projects aren't runtime bugs. They're decisions. Nobody files a ticket for the wrong database or the wrong ownership model, and nobody sees a stack trace, yet they cost more than every syntax error combined.
People keep saying AI is killing code review. It isn't. The cheap part is being automated and the expensive part is being pushed up to where it always mattered most: the design, the boundary, the conversation before anyone opened an editor.
We love to blame the outage on a bad deploy or a flaky dependency. Dig into the postmortem and the real cause usually isn't code. It's who owned what, and who thought someone else did.
Code review catches the bug in the function. Architecture review catches the bug in the plan, and the second one is almost always the one that sinks you.
A missing constraint sat in Zcash's Orchard pool for four years, surviving in-depth human audits. Four days after a new AI model shipped, an automated audit found it. When discovery accelerates like that, the security advantage stops being perfect code and becomes the speed from "found" to "users protected."
AI didn't remove the hard part of building software. It moved it. The constraint was never typing the code, and now that typing is cheap, everyone can finally see where the real work always lived.
Senior engineers aren't valuable because they know more. They're valuable because they've already seen the movie, and they recognize the failure mode before it finishes introducing itself.
Most systems don't fail when they get traffic. They fail when they get successful, because success arrives with everything that was easy to ignore on launch day.
The most dangerous engineer on a team isn't the junior. It's the one who stopped being curious, because everything underneath them keeps moving whether they keep learning or not.
A distributed system is a collection of carefully managed disappointments. The engineering isn't avoiding failure, it's choosing which failures you're willing to tolerate.
AI made the obvious mistakes rare and the expensive ones invisible. Code review didn't get easier. It moved up the stack, from implementation to judgment.
AI is great at getting an app running. It is not the same as getting it ready for real users, real payments, and real data. Here is where the gap hides.
AI is another abstraction in a long line of them. The real question was never whether engineers use it, but whether they understand what they ship.
Replay storms, cost amplification, cache pressure, and backpressure — what makes a recovery design survive contact with production.
Idempotent writes, versioned aggregates, and DO NOTHING vs DO UPDATE — merging historical replay into live state without a maintenance window.
Keep collecting in realtime, but finalize behind the present. A lagged watermark is what buys you deterministic replay.
Postgres advisory locks are useful coordination primitives, but they are not durable leadership or consensus.
Observability is part of runtime architecture, not just a read-only layer over production systems.
Live processing is the easy 20% of a streaming system. Deterministic replay is the requirement everything else hangs off of.