SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · il y a 22 heuresDo you host your own AI?message-squaremessage-square173fedilinkarrow-up1143file-text
arrow-up1143message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · il y a 22 heuresmessage-square173fedilinkfile-text
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up1·edit-2il y a 2 heuresIf you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it. It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA. Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.