Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.

“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.

  • FauxLiving@lemmy.world
    link
    fedilink
    English
    arrow-up
    12
    ·
    3 hours ago

    If you want a technical breakdown that isn’t “lol AI bad”:

    https://blog.cloudflare.com/18-november-2025-outage/

    Basically, a permission change cause an automated query to return more data than was planned for. The query resulted in a configuration file with a large amount of duplicate entries which was pushed to production. The size of the file went over the prealloctaed memory limit for a downstream system which died due to an unhandled error state resulting from the large configuration file. This caused a thread panic leading to the 5xx errors.

    It seems that Crowdstrike isn’t alone this year in the ‘A bad config file nearly kills the Internet’ club.