Load Shedding: Dropping Requests to Save Your System
The author explains how traffic spikes and client retries can tip a service over, and why rate limiters and circuit breakers do not fully solve it. Load shedding, which drops requests based on current system health, is presented as the missing piece. The article walks through ASP.NET Core and Kestrel ideas for shedding early and by endpoint priority, useful for anyone running public APIs.
Key points
- Load shedding reacts to current system health, while rate limiting enforces per-client quotas.
- Answer overload with 503 Service Unavailable and Retry-After, not 429.
- Set MaxConcurrentConnections in Kestrel so rejections happen before HTTP parsing.
- Tag endpoints with a priority so telemetry and batch work are dropped before checkout requests.
- Trigger shedding on a rising latency trend such as P99 growth rather than a fixed CPU percentage.
- Verify under staging load that P99 for accepted requests improves; if not, the threshold is wrong.
A system under pressure either sheds load on purpose or collapses by accident. The author argues that rate limiters and circuit breakers, while useful, do not stop a retry storm from pushing a service over the edge. Load shedding fills that gap by rejecting requests based on current health rather than on who sent them, and the article recommends answering with a 503 plus Retry-After instead of a 429.
The key idea is to reject as early as possible, since each layer that has already done work makes a rejection more expensive. The post points to Kestrel connection limits, then custom middleware, then controllers, and suggests tagging endpoints with priorities so telemetry is dropped before payments. It also argues that a latency trend is a better trigger than a fixed CPU percentage, and notes that the ecosystem has no standard .NET library for adaptive shedding yet.
Real-world use
- A high-traffic e-commerce API that keeps checkout alive during a campaign while dropping analytics calls.
- An ASP.NET Core service where client retries amplify load and need a brake before collapse.
- A microservice with health and telemetry endpoints that must degrade gracefully under memory pressure.
Why it matters
Beyond rate limiting and circuit breakers, you decide which requests survive an overload. Choosing the right status code and priorities also keeps client retries from making things worse.