not_IO@lemmy.blahaj.zone to Lemmy Shitpost@lemmy.worldEnglish · 1 day agothe myth of consensual pc buildinglemmy.blahaj.zoneimagemessage-square16fedilinkarrow-up137
arrow-up137imagethe myth of consensual pc buildinglemmy.blahaj.zonenot_IO@lemmy.blahaj.zone to Lemmy Shitpost@lemmy.worldEnglish · 1 day agomessage-square16fedilink
minus-squareDaTingGoBrrr@lemmy.mllinkfedilinkarrow-up2·1 day agoIf you have enough regular ram and an nvme ssd you can run huge models on modest hardware using Colibrí. I got a Lenovo mini PC with 64GB of ram and a 500GB nvme and a 38B model runs with 11 tokens per second on my machine. The 38B model fits in my ram but if I had used a 120B parameter model then it would stream the rest of the model from my nvme disk.
If you have enough regular ram and an nvme ssd you can run huge models on modest hardware using Colibrí.
I got a Lenovo mini PC with 64GB of ram and a 500GB nvme and a 38B model runs with 11 tokens per second on my machine.
The 38B model fits in my ram but if I had used a 120B parameter model then it would stream the rest of the model from my nvme disk.