Batch vs. streaming: the decision that shapes your data architecture most
Every data architecture has to answer one fundamental question before it’s designed: how often does the data need to be updated to be useful? The answer determines whether the architecture should be batch (periodic processing) or streaming (continuous, real-time processing).
Batch processing runs transformations on data accumulated over set intervals: every night, every hour, every week. It’s simple, predictable, and cost-efficient. For most management reports, accounting, and historical analysis, batch is the right choice.
Stream processing handles each event the moment it happens, updating downstream systems with latencies of milliseconds to seconds. It’s more complex and more expensive, but for some use cases there’s no real alternative.
When real time is essential
Fraud detection: a model that decides whether a bank transaction or card payment is fraudulent has seconds to act, not hours. Batch doesn’t work here.
Dynamic pricing: in industries such as travel, energy, or transportation, optimal prices shift with current demand, availability, and competitors’ moves. Information that arrives hours later is no longer actionable.
Industrial monitoring: a plant that detects anomalies in its sensors needs to respond before the problem becomes a failure. A batch system that checks the data every hour can miss the window to intervene.
Real-time operational dashboards: operations teams in call centers, logistics, or production need immediate visibility into the state of the system, not a snapshot from several hours ago.
Apache Kafka as streaming infrastructure
Apache Kafka is the reference distributed messaging system for real-time data architectures. It acts as an event bus that decouples the systems producing data from the systems consuming it, so multiple applications can process the same events independently and in the right order.
What makes Kafka the industry standard: high availability and fault tolerance, configurable message retention (consumers can replay past events), extremely high throughput (millions of events per second on well-configured clusters), and a mature ecosystem of connectors for virtually any data system.
Managed alternatives: Confluent Cloud (managed Kafka), Amazon Kinesis, Azure Event Hubs, and Google Pub/Sub are cloud alternatives that reduce the operational burden of running your own Kafka cluster.
Reference architectures for streaming
Lambda architecture: maintains two parallel layers — a batch layer for complete historical data and a speed layer for the most recent data in real time. Both are queried together. It guarantees consistency but means duplicating processing logic.
Kappa architecture: unifies batch and streaming in a single streaming pipeline, treating historical data as a very long stream. Simpler to maintain but more demanding on stream capacity.
Modern streaming with a lakehouse: today’s lakehouse platforms (Databricks, Microsoft Fabric) can ingest Kafka streams directly into Delta Lake tables with ACID guarantees, simplifying the architecture and removing the split between batch and streaming.
When real time is overkill
Not everything needs real time. Implementing streaming where batch is enough adds complexity, cost, and risk with no real benefit. If finance needs a consolidated monthly sales report, there’s no value in processing it in real time. The decision should always start from the business use case and the latency it actually needs, not from the most advanced technology available.
At AIRES, we assess the real latency requirements of each use case before proposing a streaming architecture, so we avoid unnecessary complexity.
Frequently asked questions
How much does a streaming architecture cost compared with batch? A well-designed streaming architecture can cost several times more than a batch equivalent, once you factor in infrastructure, development complexity, and maintenance. That cost is justified only when the use case has genuine low-latency requirements. The assessment should weigh the cost of the architecture against the business value of lower latency.
Is Kafka hard to manage? An on-premises Kafka cluster carries significant operational complexity: partition configuration, replication, retention, and consumer lag monitoring. Managed cloud versions (Confluent Cloud, AWS MSK, Azure Event Hubs) remove much of that complexity in exchange for a higher price. For most companies, starting with a managed service is the right call.
Can streaming be combined with existing AI models? Yes, and it’s one of the most valuable use cases. An ML model trained offline can serve real-time predictions once it’s built into the streaming pipeline: each event (a transaction, a sensor signal, a website visit) goes to the model for inference, and the result is published back to the stream. This architecture underpins real-time recommendation systems, fraud detection, and predictive alerts.