At some point a market-data app stops being an app with data in it.
It becomes a coordination problem.
That shift did not happen because the product suddenly needed a grand architecture. It happened one honest requirement at a time: backfill this range, skip work already processed, retry only the failed pieces, keep live charts current, switch pairs without corrupting state, deploy the frontend without leaking secrets, and explain why an alert fired after the chart moved on.
Each request sounds like product work. Together, they require coordination guarantees.
The boundaries fail in different ways. Batch coordination, live contracts, dashboard semantics, deployment shape, and alert explanations each bring their own failure modes. They still belong to one system.
The first shape was simple
The first version of a market-data product is easy to describe. Gather data. Process it. Show it to someone.
That description is useful for a prototype and misleading for a system people depend on. Market data is full of quiet coordination problems before the first dashboard exists. Symbols change. Reference data is incomplete. Exchanges expose different surfaces. Date ranges overlap. Some datasets are daily, some are monthly, some span a custom range. A retry can repeat work. A missing upstream piece can block ten downstream results.
The domain already has state, dependencies, and partial failure. The early app only gets away with pretending otherwise because the workflow is still small enough for a person to hold in their head.
That stopped being true once the product needed backfills.
Batch made coordination durable
The batch path looked like data processing from the outside. Underneath, it became a planning and coordination layer.
datasets.toml became a planning surface. It turned market-data intent into executable work: subjects, date ranges, task resolutions, worker pools, priorities, and dependencies. A pair like BTC-USDT over a date range could expand into stable DAG nodes, and each node carried enough identity to be claimed, skipped, retried, or linked to downstream work without relying on the operator's memory.
From there, the database was no longer just where results lived. PostgreSQL became the shared truth for the work itself.
The line is there. A data script can keep its state in local memory, a file, or the operator's terminal scrollback. A production backfill has to answer questions after the process that planned it exits:
What is ready? What is blocked? What is running? What failed? What can be retried? What already succeeded and should be left alone?
Once those questions matter, coordination is part of the product.
Workers made failure explicit
The worker layer forced the system to admit something useful: "run this task" contains a protocol.
Claiming had to be exclusive. Priority had to be respected. IO-heavy downloads, rate-limited API calls, and database-heavy processing needed separate pools. Restarts could interrupt a worker, but they could not erase the system's place.
The coordination mechanics were boring in the best way: PostgreSQL rows for DAG state, FOR UPDATE SKIP LOCKED for exclusive claims, NOTIFY and LISTEN for fast wakeups, advisory locks where propagation needed serialization, and idempotent processors so retrying a node did not corrupt outputs.
Those mechanics are what separate "the job is running" from "the system knows what work exists, who owns it, and what happens next."
The constraint that mattered was stateless workers. Scale them out, kill one, restart another, change pool sizes. The durable picture remains in PostgreSQL. That boundary is why failure can be handled as a state transition instead of a forensic exercise.
Live had a different contract
Live streaming looked adjacent to batch work because both paths dealt with the same market domain. That similarity can hide the contract change.
Batch wants replayability, dependency tracking, idempotent processors, and bounded chunks of historical work. Live wants freshness, bounded state, stable event contracts, and a frontend that does not tear itself apart every time the user switches pairs.
The live path moved through publishers, Valkey Redis Streams, stream workers, PostgreSQL streaming_* tables, a streaming API, and the frontend. That path did not replace the batch DAG. It handled a different set of failure modes.
The frontend could not consume "some events" and call that a product. The contract had to include a seed snapshot followed by an SSE tail, named events for trade bars, book bars, and book snapshots, pair lifecycle endpoints for deliberate add/drop behavior, and chart modes with stable data semantics.
Here, too, the app crossed a boundary. The API stopped being a convenience wrapper around internals and became a contract the UI depends on for continuity.
The dashboard exposed the backend
Dashboards punish vague semantics.
A backend can hide a lot behind a CLI. A chart cannot. If the snapshot does not merge cleanly with the tail, the user sees it. If pair switching leaks old state, the user sees it. If an event name is vague, the frontend has to guess. If the backend cannot explain whether a stream is active, missing, delayed, or never configured, the operator gets a blank screen with no useful next move.
The frontend became a test of whether the backend could explain its own state.
That pressure is useful. Operator views force architecture out of diagrams and into behavior. Lifecycle endpoints, status responses, snapshot windows, event names, and chart data shapes become product contracts once someone uses them to decide whether the system is healthy.
The UI revealed the complexity the system already owed.
Deployment became a correctness boundary
Deployment stopped being a packaging concern once runtime behavior depended on it. Keeping Amplify for dev and staging while adding an ECS production frontend path separated preview behavior from production promotion. It clarified which branch could produce which environment. It forced browser-visible runtime config to move through an allowlist. It made secrets in image layers a hard boundary rather than a cleanup task.
These rules affect what the product can safely do. Public runtime config without private credentials, branch-aware production promotion, and container images without baked-in secrets are all behavior the system has to preserve.
Infrastructure is part of the system when users and operators can feel the difference.
Alerts made explanation part of the data
Alerting is where the architecture stopped being only about movement and started being about memory.
A threshold rule sounds simple: alert when the z-score crosses a line. That leaves too much work for the operator. Market data moves. The UI changes. The operator arrives later. By then, the useful artifact is the reason the system believed the event mattered at the time.
That means the alert record has to preserve the signal value, baseline, window, z-score, sample count, severity, mode, and explanation. A spread-widening alert needs to say what the spread was, what baseline it was compared against, how many samples backed that baseline, and whether the comparison was local to one pair/exchange or relative to peer exchanges.
Without that stored explanation, an alert is just delayed debugging.
Here the batch and live lessons meet. Fixed windows keep semantics stable. Persistence lets the alert survive the browser session. Baseline modes separate pair-local behavior from cross-exchange divergence. The API has to show open alerts, pair-specific history, and eventually acknowledgment.
Alerting turns analytics into operations. Operations require memory.
One map, several deeper problems
Most market-data apps do not need this exact shape.
Distributed systems often arrive quietly. Nobody has to sit down and declare that the product is now a distributed system. The product just keeps asking for reasonable things.
Backfills ask for durable DAG state. Retries ask for idempotent processors. Parallel workers ask for safe claiming. Live charts ask for stream contracts. Pair switching asks for lifecycle semantics. Public frontends ask for secret boundaries. Production promotion asks for deployment rules. Alerts ask for persistent explanations.
Each requirement is defensible on its own. Together, they create a system whose correctness depends on coordination across processes, stores, queues, APIs, deployment paths, and operator interfaces.
That complexity is manageable when each boundary owns a clear job. DAG state owns batch coordination. Processors own idempotent execution. Valkey owns low-latency live handoff. The streaming API owns lifecycle and event contracts. Runtime config owns browser-visible public values. Alert records own post-hoc explanation.
Treating those boundaries as implementation details is how the complexity gets misfiled. They are the product becoming honest about what it already needs.
The same pattern shows up at each boundary. Batch work becomes durable coordination. Live streaming earns its own contract. Dashboards expose backend semantics. Frontend deployment changes runtime behavior. Alerts need stored explanations. The details differ. The pattern is the same.
The one-line version
A data product becomes a distributed system when users start depending on coordination guarantees: replayable work, live contracts, deployment boundaries, and explanations that survive the moment they describe.