I bet they duplicate everything and just switch off faulty units. Every year or so, they would emerge the whole thing and replace what they need at a large scale.
There was an Intel experiment a while back where they left a bunch of racks in the parking lot. They found that the failure rate wasn’t much higher than inside, and not needing a data center building saved money. Maybe this project just accepts the eventual failure of components.
They’re probably stacks of 8x NPU Huawei servers all cooperatively serving the same few models.
As an older example, I believe Deepseek V3 was most optimally served with ~384 GPUs in a single cluster, before they switched to Chinese NPUs. So they’d have some software that ties all these together as one “server” and maybe multiple of those all serving API requests for one endpoint.
But it doesn’t actually need all 384 in each server. Many models will fit in a single 8-GPU/NPU server, but the software pools more just to try and utilize the hardware better.
If one server fails, the system would return a few requests as empty and have to restart the serving software, but… that’s fine. All the data is ephemeral. Even if the whole 24MW unit fails, they can just route API requests somewhere else, and a few failed generations isn’t a big deal.
I’d really like to know how they handle all the small-scale HW issues. As a DC tech, I’m kept quite busy with those
Larger DCs don’t replace individual components, they wait until a percentage of servers on a rack have failed, then replace the rack or servers.
They will likely adopt this same model
https://xkcd.com/1737/
I bet they duplicate everything and just switch off faulty units. Every year or so, they would emerge the whole thing and replace what they need at a large scale.
Sounds expensive. I’m betting they just abandon it and sink a new one with new, faster hardware.
I’m thinking they replace the module every so often when it fails
There was an Intel experiment a while back where they left a bunch of racks in the parking lot. They found that the failure rate wasn’t much higher than inside, and not needing a data center building saved money. Maybe this project just accepts the eventual failure of components.
Sure, if there is zero weather a building wouldn’t be needed.
They did get rained on. I am having trouble finding an article about it now.
Computers can’t get wet from the rain if they’re underwater in the ocean.
👉😏
Probably have diver it techs… boy that’s kind of cool
They’re probably stacks of 8x NPU Huawei servers all cooperatively serving the same few models.
As an older example, I believe Deepseek V3 was most optimally served with ~384 GPUs in a single cluster, before they switched to Chinese NPUs. So they’d have some software that ties all these together as one “server” and maybe multiple of those all serving API requests for one endpoint.
But it doesn’t actually need all 384 in each server. Many models will fit in a single 8-GPU/NPU server, but the software pools more just to try and utilize the hardware better.
If one server fails, the system would return a few requests as empty and have to restart the serving software, but… that’s fine. All the data is ephemeral. Even if the whole 24MW unit fails, they can just route API requests somewhere else, and a few failed generations isn’t a big deal.