SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 22 小時前Do you host your own AI?message-squaremessage-square173fedilinkarrow-up1143file-text
arrow-up1143message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 22 小時前message-square173fedilinkfile-text
minus-squarehexagonwin@lemmy.todaylinkfedilinkEnglisharrow-up2·7 小時前a haswell xeon e5-1650 machine, i remember running llama 7b in llama.cpp in like 2023 and it was quite sluggish. guess i should try whisper at some point…
minus-squareSuspiciousCarrot78@aussie.zoneOPlinkfedilinkEnglisharrow-up7·edit-27 小時前Ha. You were doing inference on CPU on a haswell era. Been there, done that. OTOH…whisper.cpp is heavily optimised for it. Plus, you’re doing batch transcription, not real-time, so slow doesn’t actually matter. Fire Whisper small or medium overnight and wake up to searchable text. PS: if you want a good fast little llm, something like Qwen 3.6 2B will work well on the Xeon.
a haswell xeon e5-1650 machine, i remember running llama 7b in llama.cpp in like 2023 and it was quite sluggish. guess i should try whisper at some point…
Ha. You were doing inference on CPU on a haswell era. Been there, done that.
OTOH…whisper.cpp is heavily optimised for it.
Plus, you’re doing batch transcription, not real-time, so slow doesn’t actually matter.
Fire Whisper small or medium overnight and wake up to searchable text.
PS: if you want a good fast little llm, something like Qwen 3.6 2B will work well on the Xeon.