Service spotlight: Kinesis Data Streams, shards, and when SQS was simpler

I’ve seen more than one team reach for Kinesis Data Streams because “streaming” sounded like the right word for their use case, then spend weeks fighting shard math for a workload that one SQS queue and a Lambda would have handled with a fraction of the operational overhead. Kinesis is the right tool for a specific shape of problem — it’s worth being precise about what that shape is before picking it.

What it actually is

Kinesis Data Streams is an append-only, partitioned log: producers write records to a stream, the stream is divided into shards, and each shard is an ordered sequence that multiple independent consumers can read from the same position without deleting records for each other. That last part is the core difference from SQS: an SQS message is deleted once a consumer processes it, so only one logical consumer group gets each message, whereas a Kinesis record stays in the stream (default 24 hours, extendable to 365 days) and any number of consumer applications can read the full history independently, at their own pace, from their own checkpoint.

Each shard supports up to 1 MB/sec or 1,000 records/sec of writes and 2 MB/sec of reads (5 reads/sec) in standard consumer mode, or up to 2 MB/sec per shard per consumer with enhanced fan-out, which gives each registered consumer a dedicated read throughput instead of sharing the base 2 MB/sec across all of them.

Where it earns its keep

• Multiple independent consumers of the same event stream. A clickstream that needs to feed a real-time dashboard, a fraud-detection pipeline, and a data lake ingestion job simultaneously — each reading the full stream independently — is exactly what Kinesis’s replay-without- deletion model is for. SQS would need a fan-out pattern (SNS to multiple queues) to approximate this, and even then each queue consumer still deletes its own copy rather than sharing an ordered log.
• Strict ordering within a partition key. Records with the same partition key always land in the same shard and are delivered in the order written. SQS FIFO queues offer ordering too, but Kinesis’s partition-key-to-shard model scales that ordering guarantee across much higher throughput than a single FIFO message group.
• Replay and reprocessing. Because records persist in the stream for the configured retention window, a consumer that falls behind or a new consumer version that needs to reprocess history from an earlier point can do so — you can’t rewind an SQS queue.

Where SQS is genuinely simpler

If you have one logical consumer (or a consumer group where each message should be handled exactly once by exactly one worker), SQS is less to operate: no shard count to manage, no manual scaling decision when throughput grows, and its default at-least-once, auto-scaling queue model just works without capacity planning. Kinesis in provisioned mode requires you to explicitly resize (split or merge) shards as throughput changes — under-provisioned shards throttle producers with ProvisionedThroughputExceededException, and over-provisioned shards are paying for capacity you don’t use. On-demand mode removes manual shard management (Kinesis scales automatically based on observed throughput, within limits), trading that operational burden for a meaningfully higher per-stream price than provisioned mode at low-to-moderate volume.

Consumer-side, Kinesis Client Library (KCL) applications carry more operational surface than an SQS consumer loop — checkpointing, lease management via a DynamoDB table KCL creates for you, and shard rebalancing on scale-out are all things that can silently misbehave (a consumer falling behind and not alarming on IteratorAge is the classic failure mode) in ways a plain SQS ReceiveMessage loop doesn’t have to think about at all.

A practical tip

Before reaching for Kinesis, ask whether you actually need **multiple independent consumers replaying the same ordered stream**. If the honest answer is “no, I have one processing pipeline,” SQS (or SQS FIFO, if you need ordering within a single consumer group) is very likely the simpler, cheaper, lower-maintenance choice — save Kinesis for when the fan-out and replay semantics are the actual requirement, not just the more impressive-sounding service name.