On November 18, Cloudflare, an internet infrastructure giant, experienced a service outage, rendering many major websites worldwide inaccessible.

According to Downdetector, a website fault tracking agency (which itself was inaccessible to some users for a while), Anthropic’s Claude chatbot, Trump’s Truth Social, and Elon Musk’s social media platform X were among those affected. Some digital services of the New Jersey Transit system in the United States also went down due to the outage.

Meanwhile, OpenAI’s status page later that day indicated that ChatGPT and its Sora short-video application had fully recovered after experiencing failures due to issues with a “third-party service provider.”

Cloudflare was initially formed at Harvard University in 2009 and officially launched its first beta versions in 2010. It went public on the New York Stock Exchange in 2019 and currently serves 30% of the Fortune 1000 companies. Its core services include DDoS (Distributed Denial of Service) defense, a type of attack that overwhelms a target website with a flood of fake requests, causing it to crash. According to foreign media reports, the company’s traffic management and security protection services cover approximately 20% of internet traffic.

Affected by the incident, Cloudflare’s stock price fell by 2.83% as of the close of the US stock market on the 18th.

Matthew Prince, Cloudflare’s co-founder and CEO, stated that this was the most severe outage Cloudflare had experienced since 2019. “An outage like today’s is unacceptable… On behalf of the entire Cloudflare team, I apologize for the disruption caused to the internet.”

Error messages displayed on affected websites

Dane Knecht, Cloudflare’s CTO, also posted on social media, expressing deep apologies for the failure. He explained that the incident was caused by a latent defect in a service supporting the company’s bot mitigation functionality. After a routine configuration change, the service began to crash, leading to widespread degradation of the network and other services, rather than being the result of an attack.

Knecht said that both the failure and its impact, as well as the recovery time, were unacceptable. “We have already begun working to ensure that such incidents do not happen again, but we are well aware of the real impact caused. The trust our customers place in us is our most precious asset, and we will do whatever it takes to regain that trust.”

Screenshot of Dane Knecht’s post on social media

Early on the morning of November 19 (local time), Cloudflare released a comprehensive report detailing the nearly five-hour incident. The impact began at 11:28 AM local time on the 18th, with errors first observed in customer HTTP traffic. By 2:30 PM, the main impact was resolved, and downstream affected services began to observe a reduction in errors, with most services starting to operate correctly. At 5:06 PM, all downstream services were restarted, and all operations were fully restored, marking the end of the impact.

Cloudflare stated that during the outage, the company “initially and incorrectly suspected that the symptoms observed were caused by a massive DDoS attack.” Later, it correctly identified the core issue: a change in the behavior of the ClickHouse query generating the underlying file, which contained a large number of duplicate “signature” lines. This caused the Bot Management module to trigger errors, leading the core proxy system to return HTTP 5xx error codes for any traffic dependent on that module. At the same time, when the erroneous file containing more signatures than the limit spread to servers, it triggered a system panic at Cloudflare. Additionally, this affected two services, Workers KV and Access, which the company’s customers rely on the core proxy for.

Subsequently, Cloudflare resolved the issue by stopping the generation and propagation of the erroneous signature files and manually inserting a known good file into the signature file distribution queue. Then, it forcibly restarted the core proxy, and the number of 5xx error codes returned to normal thereafter.

Timeline of Cloudflare’s outage incident

Cloudflare said, “Given Cloudflare’s importance in the internet ecosystem, any outage of any of our systems is unacceptable,” and expressed deep apologies for the impact on customers and the entire internet.

Cloudflare stated that the company has already begun researching ways to strengthen its systems to prevent similar failures in the future, including enhancing the ingestion and processing of configuration files generated by Cloudflare in the same way as user-generated input is handled; enabling more global emergency stop switches for features; eliminating the possibility of core dumps or other error reports exhausting system resources; and reviewing the failure modes of error conditions in all core proxy modules.

According to foreign media reports, less than a month before this incident, Amazon Web Services had just experienced a full-day outage that paralyzed multiple network services. Subsequently, Microsoft Azure cloud services and the 365 office suite also experienced global outages.

As early as July 2024, cybersecurity company CrowdStrike had caused a large-scale system failure due to a faulty software update, leading to a chain reaction of flight cancellations, disrupted financial services, and postponed surgeries at hospitals.