With AI seeming to consume all resources for hardware, I’m wondering what parts of those current systems we could see trickling down into componentry for desktop PC’s as they get outdated for AI tasks.
I know most of this hardware is pretty specific and integrated, but I do wonder if an eventual workaround to these hardware shortages are through recycling and repurposing of the very systems causing the shortage. We have seen things like dram, flash, and even motherboard chipsets be pulled from server equipment and find its way into suspiciously cheap hardware on eBay and AliExpress, so how much of the current crop of hardware will turn up there in the future?
How much of that hardware could even be useful to us? Will nvidia repo old systems and shoot them into the sun to keep it out of the hands of gamers? Perhaps only time will tell
Memory and CPU’s are about it.
GPU’s have all shifted to bespoke hardware that is physically impossible to run on consumer hardware platforms. All the Blackwell etc type chips are insanely dense. Most GPU’s built for datacenter use don’t even have video output hardware so it’s somewhat useless.
Memory (DIMM’s) are somewhat standard. Most servers use registered ECC which doesn’t work in consumer platforms, but the actual memory chips themselves could be removed and replaced onto normal consumer DIMM’s as they are basically univsersal.
x86 CPU’s are still CPU’s at least. You might need weird motherboards but those can still be run by us plebs.
Buying ECC RAM is a good idea.
You could just buy one of those workstations that are actually almost servers. Some of them have 2 CPUs, 8 slots for RAM and a PCIe slot for your GPU. Those motherboards can handle ECC.
The server boards would pretty much have to come with them. Also, and if those cpus go as high as 500W, and as a result a lot of homes might not have a powerful enough socket to power them. Even without GPUs, might need something like a dryer outlet to realistically power.
500w isn’t that high, sockets in Aus can push out 2000w+ without any issues, and your not gonna spend 1500w on the rest of the system.
Quick google suggests USA can do 1800w with a 15a circuit, or 2400w with 20a circuit, so plenty of headroom there as well.
Remember people plug space heaters into sockets, and those will out draw even a high end CPU easily.
Keep in mind these are dual socket systems, and that’s CPU without any GPU yet. So with the CPUs populated and a consumer-grade high end GPU added, those components are at 1500W, ignoring PSU inefficiencies and other components that can consume non-trivial power.
For USA, you almost never run a 20A circuit, most are 15A, but even then that’s considered short term consumption and if you run over a longer term it’s supposed to be 80%, so down to 1440W. Space heaters usually max out at 1400W in the USA when expected to plug into a standard outlet because of this. A die-hard enthusiast might figure out how to spread non-rendundant multiple PSUs across circuits, or have a rare 20A circuit run, but it’s going to be a very very small niche.
Removing and replacing memory chips is likely so labor expensive that it’ll never happen.
If the chips are just being hoarded to shut out competitors, which is what the OpenAI deal was rumoured to be about, we could see the unused chips getting bought and used, but equally likely we could see the chips (and dimms and cpus) deliberately shredded to prevent them falling into competitor hands.
My money is on an unprecedented level of e-waste, and nothing trickling down to consumers…
By the time the things are cycled out, they may not be terribly compute-competitive, in which case…shrugs
Also, a major unknown is where models go. Say that a bunch of people decide that they can’t get access to parallel compute hardware or a lot of memory, and they research looking at models split up into MoEs or otherwise broken up. Recent LLM models have been oriented towards MoEs. Llama.cpp, and I assume the other engines capable of running LLMs, has the ability to offload experts that don’t fit in GPU memory to main memory when they aren’t actively being used. Then maybe…having a bank of consumer-level 24GB GPUs or something like that is fine, and having chips with direct access to very large amounts of memory isn’t all that interesting. Then, what becomes essential to being competitive changes.
EDIT: I also think that it’s safe to say that more memory will probably always benefit. But I’ll also say that it is also probably very likely that our existing models are staggeringly inefficient with memory. We are still doing early passes at this.
Let me give you an example. I have a LLama 3-based model currently loaded on my Framework Desktop that’s using 96GB of memory for the model and associated storage.
Now, those are correct answers. But…in order to make an LLM capable of providing that correct response, to do it purely via running a neural net trained on natural language, we had to stick a really inefficient amount of data into memory. That same hardware that I’m running it on has the ability to do billions of integer computations per second. As of today, the software running that model doesn’t provide it access to the hardware, and the model was never trained to use it. But…it could be. And if it were, suddenly a lot of need for storing edges in some neural net wasted on arithmetic go away.
Plus, we could get better results:
That’s not far off — it is approximately 10.23 — but it should have been rounded to 10.24.
So we could probably get more-useful models that don’t waste a ton of space in the model if we gave the model access to the computational hardware that’s presently sitting idle and trained it to use it. That’s an off-the-cuff example, but I think that it highlights how we’re solving problems inefficiently in terms of memory.
Same sort of thing with a lot of other problems that we have (immensely-more-efficient and probably accurate) software packages that we can already solve problems with. If you can train the model to use those and run the software in an isolated sandbox rather than trying to do it itself, then we don’t need to blow space in the LLM on the capabilities there, shrink it.
If we reduce the memory requirements enough to solve a lot of problems that people want with a much-smaller amount of memory, or with a much-less-densely-connected set of neural networks, the hardware that people care about may radically change. In early 2026, the most-in-demand hardware is hugely-power-hungry parallel processors with immense amounts of memory directly connected to it. But maybe, in 2028, we figure out how to get models to use existing software packages designed for mostly-serial computation, and suddenly, what everyone is falling over themselves to get ahold of is more-traditional computer hardware. Maybe the neural net isn’t even where most of the computation is happening for most workloads.
Maybe the future is training a model to use a library of software and to write tiny, throwaway programs that run on completely different hardware optimized for this “model scratch computation” purposes, and mostly it consulting those.
Lot of unknowns there.
Agreed on all points, I think OP is hoping the bubble will burst, and the big players will have to unload their excess hardware all at once. But I don’t think that’s likely tbh.
I sadly believe you are right, there is probably a clause in the contract between manufacturers and AI companies stating that the the chips can’t be used outside of their intended purpose.
I envision a possible similar situation as with HDDs after the Tsunami, if AI companies goes bust after chips has been manufacturerd, who will use them?
They are made for specs used in AI data centers, that doesn’t mean that they are good for general purpose.
I could see the stocks being sold for cheap to low cost memory module manufacturers, producing some weird and possibly failure prone memory modules.