Seven emails later: duplicate messages are not a theoretical problem
Mauro Servienti uses a shipping notification that arrived seven times to explain how duplicate messages happen in messaging systems. The post separates duplicates caused by sending, receiving and processing, and lists practical defenses for each. It is a useful read for any .NET developer building queue-based systems.
Key points
- Duplicates come from three places: sending, receiving and processing, and each needs a different fix.
- On the sending side, deriving a stable message identifier with SetMessageId makes retried sends recognizable as duplicates.
- The outbox pattern stores outgoing messages in the same transaction as business data, which removes the non-atomic batch problem.
- Receivers should be idempotent, because message-ID deduplication only helps when duplicates share the same identifier.
- For side effects like emails or payments, choose consciously between at-most-once and at-least-once, and use idempotency keys where the third party supports them.
A duplicate message is one that gets sent, received or processed more than once in a distributed system. The author makes the point with a shipping notification that reached his inbox seven times, a harmless annoyance that becomes serious when the same thing happens to a card charge or a physical shipment.
The post splits duplicates into three sources: a sender that processes the same trigger twice or publishes a non-atomic batch, a broker's at-least-once delivery that redelivers the same message, and side effects such as emails that a database rollback cannot undo. The suggested defenses are stable message identifiers, the outbox pattern, idempotent business logic, and a dedicated handler with no retries for side effects.
Real-world use
- An ASP.NET Core API that publishes a message from an HTTP request, where client retries should not produce duplicate shipping notifications.
- A background worker reading from Amazon SQS, where redelivered messages must not process the same order twice.
- A checkout flow that sends payments or emails, where side effects need idempotency keys or a dedicated handler.
Why it matters
In a queue-based system, duplicates should be designed in as an expected event, not treated as a bug. Stable message IDs, an outbox and idempotent handlers together keep retries from corrupting data or repeating side effects.