• historicaldocuments@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 天前

    Regardless, what followed was the newly installed router “flooding” other routers in the Air Traffic Control system. Duffy said that there were “over 1,000 connection attempts every 4 to 5 seconds,”

    I’m not a networking person. Is that a lot?

    • blargh513@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      21
      ·
      1 天前

      I was a network person before i changed disciplines.

      A direct answer is no, that is not a lot of connections, in fact, it is trivial. What we don’t know is any level of meaningful detail. If there were thousands of endpoints each making thousands of connections all at once, that changes the scale of the problem. Furthermore, most of the time, we assume that one host is trying to connect to another host. However, if those connection attempts were network broadcasts, that’s a much bigger problem. A network broadcast is sent from one host to ALL hosts on a given network, and then they ALL respond. If all hosts on a network are sending broadcast frames endlessly, that will flood a network. It’s the same concept as a denial of service attack; one host can initiate traffic that will exhaust resources, but takes very little resource to do it. Thusly it can cripple a network with very little need for power or bandwidth.

      My guess is that they fucked up the config on the router and was passing traffic that it should not have or initiated traffic that it couldn’t handle the responses from. It started using up all resources on itself and probably other network equipment and the whole thing shits the bed. Techs go around trying to reboot network gear, not sure where the problem is, the gear is overwhelmed and won’t respond. They have to dick around with out-of-band management (does not use the clogged network for management, there is a separate network for management), try to reboot or do some diagnostics, but as soon as the network device comes back up, it’s saturated again making diagnosis slow and painful. People panic, bosses start freaking out, more and more people are called in adding to confusion and now the first people to respond are not working, they’re answering endless questions about what’s going on, even though they don’t really know yet. Vendors are called, contractors are called, calls spiral and fracture, bosses demand updates while people are guessing at what to do next because they aren’t allowed to say “I don’t know” when planes are in the air. Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched before everything shit the bed. They go look at the config, see the fuckup, back it out to the last known good config and the shitstorm dies.

      this is 100% speculation treat is as such.

      • h0tbeef@lemmy.zip
        link
        fedilink
        English
        arrow-up
        12
        ·
        edit-2
        1 天前

        I’m above average familiar with networking, but likely not as much as you. Currently working on going into that field tho.

        I’m also semi-familiar with how ATC works from touring their local facilities as part of a course on aerodynamics.

        I’m fairly certain they’re lying to cover their ass.

        Everything in the ATC system has redundancy. They’re saying they clipped one copper wire and the whole system went down, but there would be another established path for that connection, severing one copper wore wouldn’t stop any communications, they would transmit through a fail safe.

        Also, the last time they had to ground all flights was 09/11/2001, and let’s just say it was not because of issues with the extremely reliable technology used in ATC.

        They installed their AI slop bullshit a few days ago.

        They think we’re all stupid, so they didn’t bother asking anyone who understands the systems before formulating their low-quality lie (in a similar fashion to how they came up with their slop ATC system - without speaking to anyone who knows how the shit works).

        • blargh513@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          3
          ·
          23 小时前

          You are correct. One physical disconnect of a copper cable isn’t going to kill off an entire system. As you noted, they have redundant network paths, equipment, etc. Those systems were designed not to go down hard, and until this week, never did.

          Copper maxes out at 10gbps in a production environment and if they were still using copper Ethernet, they were trunking multiple cables together for higher throughput. The trunking also had to be redundant, so it would take a lot of cutting or unplugging to take it all down as an oopsy.

          Again, all speculation, but I’m still betting on a human-grade fuckup. When in doubt, blame DNS.

        • stevestevesteve@lemmy.world
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 天前

          With network-loop type shit, it’s less likely they clipped the copper wire, and more likely they brought up the fiber before taking down the copper link and things sent over the copper were coming back on the fiber, looping and exploding everything.

          • blargh513@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            1
            ·
            20 小时前

            It is possible but the old routing protocols that used to do this (spanning tree) have protections in place to prevent it.

            At one point, routing protocols were a little more promiscuous and you could create a loop by plugging a single Ethernet cable into two ports on the same switch. Cue Benny hill music as everyone went running around trying to figure out what happened.

            Now there are mechanisms like BPDU guard and portfast configs that should prevent this. In most cases, they would be enabled by default. However, I’m sure there are some folks out there who have no idea how to handle an old port config done at the command line. They could have been tinkering to try and make something work, turned off some of the protective features and now they’re back to the 90s with routing loops and broadcast storms. The song is Yakety Sax fwiw, always have it at the ready.

            If you ever wonder why the networking people are grouchy, this is why. Networks were and remain complex, highly interdependent, challenging to build and maintain. It takes only a few ignorant people to send something tumbling.

            • stevestevesteve@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              edit-2
              5 小时前

              Oh I’m very familiar with (R,M)STP, but yeah I’m assuming they were doing something incredibly incompetent though we have the technology to easily prevent it. Wouldn’t be the current administration if it wasn’t

              But you are right, it could have been a minor misconfig that exploded without any real incompetence involved; it’s just more fun to make fun

      • historicaldocuments@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 天前

        this is 100% speculation treat is as such.

        Will do.

        Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched

        I’d hope that those records exist. I think there’s value in treating this as a dry run response to a cyber attack, and so far I’m not inspired (mostly with the immediate call for money).

    • frongt@lemmy.zip
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 天前

      For anything made in the last 30 years, no. He could be completely off the mark (see also Sen. Ted Stevens’ “series of tubes” speech) or the equipment might actually be that old and unable to handle it.