Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks… Honestly, I need to tinker with it more but I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Mostly time. I didn’t buy the machine to run someone else’s models. I bought it to run my own models and the machine has a job to do. Tinkering with self hosting llms, I guess. I appreciate them as models. But my machine has a day job.
I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Can you expound a little more on what you mean by this?
I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don’t seem to support non-text base interaction very well.
Yeah I’ve got the 128 gb Asus one as a processing machine, and honestly, I wish I would have just got it as a server. For my purposes, I need this bad-ass GPU/ CPU much closer to the storage.
Honestly, just a basic 2:1 is fine for me considering I’m rarely if ever running my compute locally.
I’m not sure I need a lot of compute on my NAS. I guess I might as well get as much as I can before it becomes cost prohibitive. Not planning on running any local LLMs on it though, so a GPU is probably overkill.
Yeah I do a ton of machine learning research and like, distance between the database and the place I’m doing the processing really matters. My NAS is on the other side of my house and it would be great to have maybe two NAS, a primary one for like, my movies, personal files, old projects etc, and then this one right next to it for active projects.
Right now I’m having to do it all on my primary machine, which means I often have to freeze updates for long periods of time once I get a particular compute configuration set up for a particular project.
This is what I’m drooling over currently:
https://store.minisforum.com/products/minisforum-n5-max-ai-nas
128 gb uram 🤤
Fucking saved.
Do you know what kind of open weights it can run, and at which t/s?
Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks… Honestly, I need to tinker with it more but I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.
Someday TM
Why not llama.cpp? Ollama is just a wrapper over it.
Mostly time. I didn’t buy the machine to run someone else’s models. I bought it to run my own models and the machine has a job to do. Tinkering with self hosting llms, I guess. I appreciate them as models. But my machine has a day job.
Can you expound a little more on what you mean by this?
I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don’t seem to support non-text base interaction very well.
I’ve been thinking about getting a real NAS machine instead of having my gaming PC always on. This looks pretty slick.
Yeah I’ve got the 128 gb Asus one as a processing machine, and honestly, I wish I would have just got it as a server. For my purposes, I need this bad-ass GPU/ CPU much closer to the storage.
Honestly, just a basic 2:1 is fine for me considering I’m rarely if ever running my compute locally.
I’m not sure I need a lot of compute on my NAS. I guess I might as well get as much as I can before it becomes cost prohibitive. Not planning on running any local LLMs on it though, so a GPU is probably overkill.
Yeah I do a ton of machine learning research and like, distance between the database and the place I’m doing the processing really matters. My NAS is on the other side of my house and it would be great to have maybe two NAS, a primary one for like, my movies, personal files, old projects etc, and then this one right next to it for active projects.
Right now I’m having to do it all on my primary machine, which means I often have to freeze updates for long periods of time once I get a particular compute configuration set up for a particular project.