Kafka Consumer Recovery: Resuming Processing After a Crash

Kafka consumers must pick up where they left off after a restart, crash or network failure. This article explains how committed offsets, consumer groups and manual commits govern recovery, using a C# example built on Confluent.Kafka. It also covers duplicate deliveries, poison messages and the monitoring signals that reveal a consumer quietly falling behind.

Key points

• A committed offset names the next record to read; committing an unprocessed record makes the consumer skip it.
• Set EnableAutoCommit = false and EnableAutoOffsetStore = false so you decide exactly when offsets become eligible for commit.
• Commit only after the business operation has succeeded, never while it is still running.
• Make handlers idempotent, for example by storing the Kafka event ID in a table with a unique constraint.
• Kafka transactions cover Kafka-to-Kafka flows only; a SQL Server transaction or external API call is not included.
• After bounded retries, send poison messages to a dead-letter topic and watch consumer lag, commit failures and rebalance frequency.

A Kafka consumer can run for hours and then stop abruptly after a crash, restart or network fault. Where it resumes depends on the committed offset, on whether the business work finished, and on how the application treats redelivered messages. Kafka has no way to know whether a database write or an external API call completed, so the application has to coordinate processing with offset commits.

The article walks through a Confluent.Kafka setup with auto-commit turned off and a commit after each successful handler, and explains why a committed offset points at the next record to read. It warns against advancing past failed records, and suggests idempotent writes, a unique-keyed table of processed events, bounded retries and a dead-letter topic for poison messages. It also covers rebalance handling and metrics such as consumer lag, commit failures and restart counts that reveal a consumer falling behind.

Real-world use

• An order-processing background service that restarts during a deployment and must not apply the same order twice.
• A consumer that calls a payment provider and needs idempotency keys to tolerate redelivered events.
• A data pipeline where malformed messages must be parked in a dead-letter topic so the rest of the partition keeps moving.

Why it matters

A consumer with auto-commit can lose in-flight work or skip records silently after a crash. Controlling commit timing and making handlers idempotent is what keeps Kafka-driven workflows reliable in production.

Related stories

• Collection Return Types in C#: Interfaces, Views, and SnapshotsDLNick Cosentino (Dev Leader)2 d ago
• Dynamic Gauss-Legendre numerical integration in C#: computing nodes and weights on the fly without overflowHBHabr C# (ru)2 d ago
• Events and Delegates in C#: Contracts, Subscriptions and NotificationDLNick Cosentino (Dev Leader)3 d ago