Oh my god I feel so stupid. I’ve been arguing back and forth whether it was worth de-atomizing my steam deck to spin up alpaca in docker. I forgot they have a flatpak
Bazzite also has podman, though not specifically docker, in the core OS.
So… I have spun up one local LLM in Alpaca, told it what hardware, OS, and environment it is in/on, told it to generate a context prompt to inform itself of all that… and its now helping me try to figure out how/if it is possible to set up a podman container/environment… for LLMs that either Alpaca does not yet support, or I am too stupid to figure out.
Alpaca even has tools. You can give an LLM the ability to search the web for something, and find some info or what not.
ROCm on a Deck seems to kind of sort of work via … basically you spoof your gpu id in the podman environment, and then… you would either hwve to do the ole allocate more ram to gpu thing, or attempt to edit the LLM’s config and such, to try an run in a much lower than expected vram situation.
(WIP)
Presumably you could tell it to do a lot of things but that seems like a bad idea lol, anyway yeah, I was able to just tell it ‘go online and lookup bazzite, familiarize yourself with pertinent details, reformulate context prompt.’
Oh my god I feel so stupid. I’ve been arguing back and forth whether it was worth de-atomizing my steam deck to spin up alpaca in docker. I forgot they have a flatpak
Bazzite also has podman, though not specifically docker, in the core OS.
So… I have spun up one local LLM in Alpaca, told it what hardware, OS, and environment it is in/on, told it to generate a context prompt to inform itself of all that… and its now helping me try to figure out how/if it is possible to set up a podman container/environment… for LLMs that either Alpaca does not yet support, or I am too stupid to figure out.
Alpaca even has tools. You can give an LLM the ability to search the web for something, and find some info or what not.
ROCm on a Deck seems to kind of sort of work via … basically you spoof your gpu id in the podman environment, and then… you would either hwve to do the ole allocate more ram to gpu thing, or attempt to edit the LLM’s config and such, to try an run in a much lower than expected vram situation.
(WIP)
Presumably you could tell it to do a lot of things but that seems like a bad idea lol, anyway yeah, I was able to just tell it ‘go online and lookup bazzite, familiarize yourself with pertinent details, reformulate context prompt.’