04/08/2024
Just four months after a complete power outage at a critical data center we were hit with the exact same scenario. Here’s how we did this time, and what’s next...
Continue reading »
11/04/2023
Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. Here are the details...
11/01/2023
Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023, due to the misconfiguration of a deployment tool used by Workers KV....
10/20/2023
On Wednesday, October 18, 2023, we discovered attacks on our system that we were able to trace back to Okta. We have verified that no Cloudflare customer information or systems were impacted by this event because of our rapid response. ...
10/04/2023
On 4 October 2023, Cloudflare experienced DNS resolution problems. Some users may have received SERVFAIL DNS responses to valid queries. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again...
August 02, 2023 1:05 PM
A deep dive into the recent incidents relating to Workers KV, and how we’re going to fix them...
July 11, 2023 8:48 AM
On July 9, 2023, users in the Asia Pacific region experienced connection errors due to origin DNS resolution failures to .com and .net TLD nameservers...
January 25, 2023 3:47 AM
Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products...
October 26, 2022 3:02 AM
Today, a change to our Tiered Cache system caused some requests to fail for users with status code 530...
August 09, 2022 3:56 PM
Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees...
June 21, 2022 12:38 PM
Today, June 21, 2022, Cloudflare suffered an outage that affected traffic in 19 of our data centers...
April 05, 2022 12:57 PM
This post tells the story of how a missing shell option called “pipefail” slowed Cloudflare down....
March 22, 2022 4:57 PM
Today at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result...
January 26, 2022 6:34 PM
On January 24, 2022, as a result of an internal product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin...
November 27, 2020 12:00 PM
When we review design documents at Cloudflare, we are always on the lookout for Single Points of Failure (SPOFs). In this post, we present a timeline of a real-world incident, and how an interesting failure mode known as a Byzantine fault played a role in a cascading series of ev...
