Outbox in Postgres: concurrent workers vs reading the WAL

Milan Jovanović compared two ways to drain a transactional outbox on PostgreSQL: polling workers and a reader streaming the write-ahead log through logical replication. Both move thousands of messages per second, so the real split is latency against operational cost — a replication slot, WAL held on disk and heavier duplicate replays.

Key points

• FOR UPDATE SKIP LOCKED lets workers claim batches of up to 1,000 rows without collisions, even across separate processes; the transaction stays open until the broker confirms.
• Five workers drained ~23,000 messages per second, but a new message still waited 419 ms on average at 100 orders per second because of the one-second sleep after an empty batch.
• The Npgsql reader — LogicalReplicationConnection, a pgoutput slot — measured 0.83 ms commit-to-consumer and ~31,000 messages per second, about 57,600 without the batched UPDATE.
• Call SetReplicationStatus with commit.TransactionEndLsn only after the broker confirms, so a crash replays messages instead of losing them.
• NOTIFY sent from a trigger serializes commits: throughput fell from 8,070 to 256 orders per second with 100 concurrent users.
• A replication slot keeps WAL on disk while the reader is down; max_slot_wal_keep_size caps it, but Postgres invalidates the slot past that limit.

Milan Jovanović compared two ways to drain a transactional outbox on PostgreSQL: polling workers and a reader that streams the write-ahead log through logical replication. In one-minute backlog drains a single worker published about 15,000 messages per second, five workers reached roughly 23,000, and the WAL reader about 31,000. Latency separated them: at 100 orders per second five workers averaged 419 ms because they sleep a second after an empty batch, while the reader stayed under a millisecond.

The WAL path costs more to run: wal_level=logical, a publication, a replication slot that pins WAL on disk while the reader is down, and a cleanup job. One slot cannot be scaled out. A hard kill replayed 31,095 duplicates against 4,536 for the workers, so consumers must be idempotent. NOTIFY sent from a trigger is the variant to avoid: it took the API down to 256 orders per second.

Real-world use

• An order or payment API that must publish an event when the database transaction commits, even if the broker is briefly unavailable.
• A high-volume ingestion service that drains a backlog and needs more throughput by adding workers inside the same deployment.
• A notification or pricing feed where the gap between commit and consumer has to stay under a few milliseconds.
• A system where empty claim queries (about 289,000 a minute with five workers) are burning Postgres CPU and you want an idle-cost-free reader.

Why it matters

Polling keeps the outbox simple and scales out by adding workers; reading the WAL is the option when commit-to-consumer latency actually matters, paid for with a replication slot, WAL disk management and more duplicated sends.

Related stories

• Critter Stack in Review: message loss fixes, clean shutdown and multi-tenancyJXJasperFx (Critter Stack) BlogOct 30
• Booking cottages with range types in .NET 10: no double bookings via PostgreSQLDVDEV Community: #csharpOct 30
• CritterWatch 1.1 groups dead letters by failure cause and acts on each group at onceJXJasperFx (Critter Stack) BlogOct 29