Adaptive Rate Limiting: How Uber and OpenAI Are Scaling Access to AI

As demand for AI services surges, companies like Uber and OpenAI are moving beyond traditional rate limiting systems to more sophisticated, adaptive approaches. These shifts prioritize both operational resilience and a positive user experience, paving the way for broader access to powerful AI tools like Codex and Sora.

The Limitations of Traditional Rate Limiting

Historically, rate limiting often involved per-service limits based on counters, frequently implemented using technologies like Redis. Even as effective in preventing overload, this approach presented several challenges. Uber, for example, found that counter-based systems led to operational inefficiencies, including increased latency and the need for frequent deployments to adjust thresholds. Uber’s Engineering Blog details how inconsistent configurations and fragmented observability further complicated management and protection of services.

The Rise of Adaptive, Policy-Based Systems

Both Uber and OpenAI have developed proprietary, infrastructure-level rate-limiting platforms that move away from rigid, counter-based limits. These new systems employ “soft controls” – probabilistic shedding or credit-based waterfalls – to manage traffic without abruptly halting user access. This approach aims to maintain system resilience while minimizing disruption to user workflows.

Uber’s Global Rate Limiter (GRL)

Uber replaced its legacy rate limiters with a Global Rate Limiter (GRL) featuring a three-tier feedback loop. According to Uber’s blog, rate-limit clients within the service mesh enforce decisions locally, zone aggregators collect metrics, and regional controllers calculate global limits, pushing updates back to the clients. The GRL also utilizes a system that drops a configurable percentage of traffic, applying pressure on calling services to adapt rather than simply blocking them.

OpenAI’s Credit-Based Waterfall

OpenAI’s focus was on improving the user experience for Codex and Sora. Traditional rate limits frequently interrupted engaged users. OpenAI’s engineering team designed a system that combines rate limits, usage tracking, and credits. This “waterfall” model allows users access up to a certain limit, then draws from a credit balance, creating a seamless experience. As OpenAI explains, users don’t perceive distinct systems. they simply continue using the tools.

Real-Time Access and Asynchronous Billing

To support this model, OpenAI built a real-time access engine that consolidates usage tracking, rate-limit windows, and credit balances into a single evaluation path. This engine makes synchronous decisions, instantly checking for credit balances if rate limits are exceeded. Credit debits are settled asynchronously through a streaming processor, using idempotency keys to prevent double-charging. This architecture relies on three data streams: product usage events, monetization events, and balance updates, ensuring auditability and reconciliation.

Results and Scalability

These architectural shifts have yielded positive results for both companies. Uber’s GRL now processes over 80 million requests per second across 1,100 services, reducing tail latency by removing Redis dependencies and successfully mitigating a 15x traffic surge and DDoS attacks. InfoQ reports that OpenAI’s credit system has integrated seamlessly into the access path for Codex and Sora, providing real-time, accurate billing without sacrificing performance.

Key Takeaways

• Traditional rate limiting systems can create operational overhead and negatively impact user experience.
• Adaptive, policy-based rate limiting offers a more flexible and resilient approach.
• Infrastructure-level platforms enable automated, adaptive controls, scaling to handle massive demand.
• Combining rate limits with credit systems can provide a seamless user experience while ensuring fair access.

The move towards adaptive rate limiting represents a significant step in scaling access to AI services. By prioritizing both operational efficiency and user satisfaction, Uber and OpenAI are setting a new standard for managing demand in the age of increasingly powerful AI tools.