Mastering Rate Limiting and Resource Governance for MCP Servers at Scale
A production write-up on rate limiting and resource governance for MCP orchestration servers, where a single tenant's burst can exhaust shared LLM token quota and starve everyone else. It compares the common limiter algorithms and lands on per-tenant token buckets with Redis-backed state and quota-aware refill rates.
Key points
- Tenant traffic amplification, not raw scale, is what causes system-wide outages: one tenant pushed 10k requests per minute and drove token quota to 100 percent.
- Token bucket fits token-heavy LLM traffic, leaky bucket suits constant high RPS, and fixed windows alone invite thundering herd spikes at boundaries.
- Per-tenant partitions through RateLimitPartition with a tenant_id claim keep one noisy client from starving the pool, at the cost of one limiter state per tenant.
- Distributed token buckets need atomicity, usually via a Redis Lua script, plus synchronized clocks to avoid drift.
- Return HTTP 429 with a retry-after header when a bucket is empty instead of queueing, and cap queue depth hard.
- Scale workers on queue depth together with CPU (70 percent utilization, queue over 100) rather than on peak RPS.
A production playbook for keeping MCP (Model Context Protocol) orchestration servers alive when one tenant floods the pipeline. The author walks through token bucket, sliding window, fixed window and leaky bucket algorithms, comparing burst smoothing against storage cost, and argues per-tenant partitioning is what stops a noisy client from starving the rest of the region.
The concrete recipe mixes ASP.NET Core's AddRateLimiter with per-tenant token buckets keyed from the JWT, Redis Lua scripts so refill and consume stay atomic across nodes, a refill rate lowered when Azure OpenAI quota usage passes 80 percent, and Kubernetes autoscaling on CPU plus queue length. Fail fast with HTTP 429 and retry-after rather than silently queueing.
Real-world use
- A multi-tenant LLM orchestration API where one customer's batch job would otherwise consume the whole subscription's Azure OpenAI quota.
- A chat product whose traffic spikes after a marketing push and needs a hard per-tenant ceiling before it reaches inference workers.
- A Kubernetes-hosted tool-calling service that must scale on queue depth instead of paying for peak GPU capacity.
- A cost-sensitive vector search backend where expensive tool calls are routed to a cheaper CPU-only pool.
Why it matters
Without per-tenant limits and shared limiter state, one noisy client can exhaust token quota for the whole region, turning a traffic spike into an outage and a runaway bill.