Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks… Honestly, I need to tinker with it more but I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Mostly time. I didn’t buy the machine to run someone else’s models. I bought it to run my own models and the machine has a job to do. Tinkering with self hosting llms, I guess. I appreciate them as models. But my machine has a day job.
I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Can you expound a little more on what you mean by this?
I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don’t seem to support non-text base interaction very well.
128 gb uram 🤤
Fucking saved.
Do you know what kind of open weights it can run, and at which t/s?
Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks… Honestly, I need to tinker with it more but I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Someday TM
Why not llama.cpp? Ollama is just a wrapper over it.
Mostly time. I didn’t buy the machine to run someone else’s models. I bought it to run my own models and the machine has a job to do. Tinkering with self hosting llms, I guess. I appreciate them as models. But my machine has a day job.
Can you expound a little more on what you mean by this?
I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don’t seem to support non-text base interaction very well.