training a top-level model takes harvesting petabytes / exabytes of revelant data and then months of continuous 100% power on a large server farm costing many millions of dollars.
It’s a one-and-done process, then the normal Q&A usage has vastly lower requirements.
Local models generally means you download the result of all that , after it’s done, (the PT in ChatGPT stands for pre-trained) and then generate answers locally, but the training wasn’t done locally. The generating answers step is called “inference”.
Hardware cost to do inference with a top-level model is within reach of most small businesses. Hardware cost to do training of a top-level model is orders of magnitude higher.
So, there are open-source models that are trained by universities, and then used by affluent but not billionaire people at home.
There are open-source models that can do inference (answer questions) on a typical 8GB gaming graphics card, but they aren’t in the same order of magnitude of quality as the famous top-level models.
They’re the output of all that data centre processing. The model is fixed at that point and can’t “learn” anything more, so the pressure is always on to release a newer model that resulted from more data with better algorithms, but still small enough t9 be able to run on a consumer grade GPU
training a top-level model takes harvesting petabytes / exabytes of revelant data and then months of continuous 100% power on a large server farm costing many millions of dollars.
That was my understanding. So how do local models work? (Been fucking with computers since the 80s, AI is all new to me.)
It’s a one-and-done process, then the normal Q&A usage has vastly lower requirements.
Local models generally means you download the result of all that , after it’s done, (the PT in ChatGPT stands for pre-trained) and then generate answers locally, but the training wasn’t done locally. The generating answers step is called “inference”.
Hardware cost to do inference with a top-level model is within reach of most small businesses. Hardware cost to do training of a top-level model is orders of magnitude higher.
So, there are open-source models that are trained by universities, and then used by affluent but not billionaire people at home.
There are open-source models that can do inference (answer questions) on a typical 8GB gaming graphics card, but they aren’t in the same order of magnitude of quality as the famous top-level models.
They’re the output of all that data centre processing. The model is fixed at that point and can’t “learn” anything more, so the pressure is always on to release a newer model that resulted from more data with better algorithms, but still small enough t9 be able to run on a consumer grade GPU