In my regular research (behind a paywall), I have been saying for a while that I think the future of AI is not large language models (LLM), but small language models (SLM) run on local desktop computers or even mobile phones.
Training happens once per model, but inference is an ongoing process. So, there’s going to be a huge amount of energy saving if we move to using local models.
SLM’s are still trained on massive datacenters as large models and then quantized down. But yes for inference there’s hope in the future
Training happens once per model, but inference is an ongoing process. So, there’s going to be a huge amount of energy saving if we move to using local models.