Batch and live market data look close enough to invite the same design.
Both paths process trades, quotes, books, pairs, exchanges, bars, and time windows. Both may calculate the same aggregate and write into related tables. Both can feed the same chart.
The shared vocabulary is real. The runtime behavior is different.
A backfill reconstructs a known range of history. It can divide that range into bounded work, record dependencies, retry failed pieces, and converge on the expected result. A live stream has to preserve a coherent view while events are arriving, connections are changing, and the user is looking at the output.
Confusing those jobs usually produces a system that handles the happy path and has no precise answer for recovery.
The domain model should be shared
Separate runtime contracts do not require separate definitions of the market.
A trade should have the same identity, timestamp rules, pair, venue, price, and quantity regardless of whether it came from an archive or a live connection. A one-minute bar should use the same window boundaries and aggregation rules when it is computed during a backfill or updated from a stream. Pair normalization, exchange naming, decimal precision, and book-side semantics should also agree.
This shared language keeps historical and current views comparable. It also makes replay useful. If the backfill path and the live path interpret the same event differently, replay can complete successfully while producing an answer that disagrees with the one users saw in real time.
The useful sharing happens in domain types, validation rules, windowing logic, and processor behavior. Runtime ownership belongs elsewhere. Task claims, dependency state, stream offsets, connection freshness, and active pair state describe how work is being executed. They are specific to the path carrying the work.
That boundary allows the product to speak one market language without forcing two operating models into one abstraction.
Backfills reconstruct history
A backfill begins with a range and an expected outcome. The system knows the pair, source, dataset, resolution, and dates it intends to cover. That intent can be expanded into stable work units before processing begins.
A configuration entry for a pair and date range might produce a DAG of download, normalization, aggregation, and persistence tasks. Each node has an identity and a dependency set. Workers claim ready nodes from durable state, record their progress, and leave enough information for another worker to resume after a failure.
The database is doing more than storing results in this model. It records what work exists, which prerequisites are complete, who owns a claim, how many times the work has run, and whether the output can be accepted. That state survives the process that planned the backfill and the worker that happened to execute it.
Idempotency determines whether retries are routine or dangerous. A processor may receive the same source range twice because a worker died after writing its output and before recording success. Stable keys and conflict-safe writes let the second attempt converge on the same stored result. Without those properties, the retry mechanism adds another failure mode.
Bounded work keeps recovery proportional to the miss. If one hourly archive is corrupt, the system should be able to replace that hour and recompute the affected downstream windows. Rebuilding a month because the runtime cannot name one missing unit increases cost and makes the recovery harder to inspect.
Batch correctness is visible after the work settles. The expected range is covered, dependencies are satisfied, repeated execution converges, and every failed or skipped unit has a recorded reason.
Live streams maintain continuity
A live path operates without a completed range. Its upper boundary keeps moving. The user may switch pairs, the upstream may reconnect, and events may arrive late or out of order while the chart remains open.
The runtime state reflects those conditions. It needs to know which streams are active, the last event received from each source, the last event persisted, the current consumer position, and whether the data shown to the user is fresh. Those facts change continuously and cannot be represented accurately as a set of batch tasks waiting to finish.
Live views also need a defined starting point. Opening an event stream alone leaves the frontend with no history until enough new events arrive. Fetching a snapshot alone gives the user a view that immediately begins to age. A practical contract hydrates the view from a bounded snapshot and continues it from a live tail.
The join between them needs an explicit cursor, sequence, or timestamp rule. Otherwise an event can appear in both responses and be counted twice, or arrive between the snapshot query and stream subscription and disappear from the view. The frontend needs enough identity to deduplicate overlaps and enough ordering information to detect a gap.
Named events matter for the same reason. Trade bars, book bars, and book snapshots have different merge behavior. A generic update event pushes that knowledge into client-side guesses. Stable event names and payload schemas let the frontend apply the correct transition and reject data it does not understand.
Live correctness is visible while the system is moving. The current view has a known origin, events continue from that origin without silent gaps, state stays bounded, and lifecycle changes do not leak data from the previous subscription.
Their failures need different state
Batch failures are usually failures of reconstruction. A dependency is missing. A worker loses its claim. A task partially writes. Duplicate execution produces conflicting output. Cached reference data causes a historical processor to use the wrong market metadata.
These conditions belong in durable task state because the work can be inspected and resumed later. A failed node can wait for a dependency, return to the ready queue, or remain failed with the exact range and reason attached.
Live failures are failures of continuity. A connection drops. A publisher keeps running while a consumer falls behind. A snapshot is older than the retained stream tail. Events arrive out of order. A pair switch leaves the old subscription active long enough to write into the new chart.
The response is time-sensitive. The system may reconnect from a known offset, refresh the snapshot, mark the view stale, or stop the subscription and rebuild its state. A retry counter attached to a general job record does not say whether the current chart is safe to continue.
Keeping the states separate also makes monitoring more useful. Batch metrics can describe queue depth, task age, retry counts, blocked dependencies, and range coverage. Live metrics can describe source freshness, consumer lag, reconnects, event gaps, active subscriptions, and snapshot age. Combining them under a single processing status removes the details an operator needs during a failure.
Finality has to be named
Batch and live paths often meet in the same aggregate tables. That handoff needs a clear definition of finality.
A live bar may be updated several times while its window is open. A historical processor may later recompute the same bar from a complete source file. Both results can be valid for the stage they represent, yet they should not compete through last-write-wins behavior.
The stored model needs to identify whether a value is provisional or final, what source range produced it, and which write is allowed to replace another. A watermark can establish when the live path considers a window complete. A later backfill can verify or repair that window through an idempotent upsert with explicit precedence.
Shared processor logic lets the two paths apply the same windowing and aggregation rules while preserving different execution state. A recomputed result should agree with the finalized live result when both consumed the same inputs. A disagreement becomes a diagnosable data-quality event rather than an unexplained chart correction.
The contract also needs a policy for late data. Some products reopen a window, some publish a correction, and some preserve the original value while recording the late event for audit. Any of those policies can be operated when it is explicit. Silent replacement leaves downstream consumers unable to tell whether the market moved or history was revised.
The frontend tests the live contract
Runtime ambiguity becomes visible quickly in a dashboard.
Suppose the user changes from BTC-USDT to ETH-USDT. The frontend clears the old view, requests a seed snapshot, and opens a stream for the new pair. During that transition, an event from the old subscription can still be in flight. If events do not carry pair identity, or the client does not bind them to a subscription generation, the old event can appear in the new chart.
A similar problem occurs during reconnect. The browser has a last known event, the server has a retained stream range, and the snapshot store has its own freshness. The API has to decide whether it can resume from the browser's cursor or whether the client needs a new snapshot. Returning an open connection without making that decision gives the appearance of recovery while leaving a silent gap.
Lifecycle endpoints make these transitions deliberate. The system can expose whether a pair is available, whether its publisher is active, when the latest snapshot was written, and whether a requested removal is complete. The dashboard can show loading, stale, unavailable, and current states based on facts supplied by the backend.
This contract is part of product behavior. Users read meaning from chart motion, freshness labels, and pair changes. The frontend cannot recover semantics that the runtime never defined.
Share meaning and separate execution
The clean boundary is a shared domain model with distinct runtime contracts.
Trades, books, pairs, bars, windows, normalization rules, and aggregation logic should remain consistent across historical and live processing. That consistency allows the product to compare current data with history and allows replay to verify what happened during live operation.
Batch execution needs stable work identity, dependency tracking, durable claims, idempotent writes, and bounded replay. Live execution needs snapshot hydration, ordered tails, freshness state, bounded buffers, lifecycle operations, and payload contracts the frontend can enforce.
The paths can meet at explicit boundaries: finalized windows, shared schemas, repair policies, and APIs that identify the state of the value being returned. They remain easier to reason about when their execution state and recovery mechanisms stay separate.
The distinction is practical. Historical work has to converge after interruption. Live work has to preserve continuity through interruption. A system that names both obligations can recover without making the operator guess which guarantee still holds.
The one-line version
Batch reconstructs a trustworthy past, live streaming maintains a coherent present, and each needs a runtime contract built for that job.