Cloudflare Outage Briefly Disrupted Service for Some Customers Due to Bot Management Configuration Error
Table of Contents
On November 19, 2023, Cloudflare experienced an outage that impacted some customers due to a configuration error within its Bot Management service.The issue, stemming from a change in how Cloudflare’s ClickHouse database generated a configuration file used to identify automated traffic, led to false positives in bot detection and, consequently, disrupted service for users relying on those rules. Customers not utilizing the bot score in their rules remained unaffected.
What Happened?
According to a post-mortem published by Cloudflare, the root cause was a change in the underlying ClickHouse query behavior that generates the configuration file used by the Bot Management system. This file contains “features” used to identify bots. The altered query inadvertently created a large number of duplicate entries within this file.
As Prince,a Cloudflare engineer,explained in the post,the duplicated data caused the configuration file to rapidly grow,exceeding pre-set memory limits. This ultimately led to the failure of “the core proxy system that handles traffic processing for our customers, for any traffic that depended on the bots module.” Essentially, the system responsible for filtering bot traffic became overwhelmed and stopped functioning correctly.
Impact on Customers
The impact of the outage varied depending on how customers had configured their Cloudflare settings.
* customers using Cloudflare’s bot score in their firewall rules: These customers experienced false positives, meaning legitimate user traffic was incorrectly identified as bot traffic and blocked. This resulted in service disruptions and accessibility issues.
* Customers not using the bot score in their rules: These customers were not affected by the outage, as their traffic processing wasn’t reliant on the malfunctioning Bot Management module.
Cloudflare resolved the issue by reverting the problematic query change and flushing the corrupted configuration file. the company states that the incident lasted approximately 26 minutes.
What is ClickHouse and Why does it Matter?
ClickHouse is a column-oriented database management system known for its speed and efficiency in handling large volumes of data. Cloudflare utilizes ClickHouse for various data-intensive tasks, including generating the configuration files for its Bot Management service. The database’s performance is critical to the real-time analysis of network traffic and accurate bot detection. A change in query behavior, as seen in this incident, can have important downstream effects if not carefully tested and monitored.
Key Takeaways
* A change to a ClickHouse query caused duplicate data in Cloudflare’s Bot Management configuration file.
* This led to the core proxy system failing, impacting customers who used bot scores in their rules.
* Customers not utilizing the bot score were unaffected.
* Cloudflare resolved the issue by reverting the query change and flushing the corrupted file.
Looking Ahead
Cloudflare has stated it is implementing additional monitoring and alerting to detect similar issues in the future. The company is also reviewing its change management processes to prevent such incidents from recurring. This incident highlights the importance of robust testing and monitoring when making changes to critical infrastructure components, particularly those involved in real-time data processing and security functions.
Worth a look