In my regular research (behind a paywall), I have been saying for a while that I think the future of AI is not large language models (LLM), but small language models (SLM) run on local desktop computers or even mobile phones.
I don’t doubt that locally hostable models will be important for agentic tasks but I suspect they are still going to be bigger than most can comfortably host for time being. I still think there will be a place for the super large models for more complex reasoning although how much will be due to the intrinsic knowledge in the weights and how much due to the plumbing around them remains too be seen.
And if course LLM’s are not going to be the end point of the search for AGI. Whatever their architecture they will still need copious amounts of compute.
I’m really interested to see what comes of the LLM ASIC idea - it worked wonders for hardware video codecs. I could see the smartphone industry picking up again on annual releases worthy of upgrades if they included dedicated AI chips not beholden to a cloud subscription. I think we’re reaching the point that the frontier models are becoming “good enough” to use for a while before uploading to the next.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
but I suspect they are still going to be bigger than most can comfortably host for time being
ARM and CXMT both have the potential to change this situation rather dramatically.
ARM: The RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86 (desktop OS) space, but ARM doesn’t have the same baggage.
CXMT: We know the DRAM cartel have previously engaged in price-fixing, and the current shortage looks suspiciously similar to their old behavior. When confronted by a new challenger, they might be forced to actually compete.
Hence “x86 (desktop OS) space”. It’s not intrinsically part of x86, but it has settled in as a conventional piece of x86 desktop OSes. x86 consoles and ARM desktops don’t assume the same principle, and they can get a lot more mileage out of SoCs as a result.
I don’t doubt that locally hostable models will be important for agentic tasks but I suspect they are still going to be bigger than most can comfortably host for time being. I still think there will be a place for the super large models for more complex reasoning although how much will be due to the intrinsic knowledge in the weights and how much due to the plumbing around them remains too be seen.
And if course LLM’s are not going to be the end point of the search for AGI. Whatever their architecture they will still need copious amounts of compute.
Alibaba just announced a chip specifically for running local models. We’ll see what it ends up going for. https://wccftech.com/alibabas-tsmc-built-5nm-risc-v-chip-xuantie-c950-now-runs-qwen-3-8-27b-model-natively-unlocking-massive-vertical-integration-tailwinds/
I’m really interested to see what comes of the LLM ASIC idea - it worked wonders for hardware video codecs. I could see the smartphone industry picking up again on annual releases worthy of upgrades if they included dedicated AI chips not beholden to a cloud subscription. I think we’re reaching the point that the frontier models are becoming “good enough” to use for a while before uploading to the next.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
NPU’s have been in phone SoCs for years now.
ARM and CXMT both have the potential to change this situation rather dramatically.
ARM: The RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86 (desktop OS) space, but ARM doesn’t have the same baggage.
CXMT: We know the DRAM cartel have previously engaged in price-fixing, and the current shortage looks suspiciously similar to their old behavior. When confronted by a new challenger, they might be forced to actually compete.
That has nothing to do with x86 though…
Hence “x86 (desktop OS) space”. It’s not intrinsically part of x86, but it has settled in as a conventional piece of x86 desktop OSes. x86 consoles and ARM desktops don’t assume the same principle, and they can get a lot more mileage out of SoCs as a result.