Qwen 3.6 27B MTP (Q4) is really great. ~24GB VRAM usage with one slot @ 262k . ~30GB with 3 slots at 128k. Also it does not struggle like Gemma if you use e.g. a Q4 KV-Cache. And it runs at 400-800 ppt/s and 20-60 itp/s on a V100.
But it has a competitor since a short while that is Laguna XS 2.1, which is really good for a A3B MoE. I’d never have thought a 30B MoE could be on par with a 27B dense, but it seemingly is.
Qwen 3.6 27B MTP (Q4) is really great. ~24GB VRAM usage with one slot @ 262k . ~30GB with 3 slots at 128k. Also it does not struggle like Gemma if you use e.g. a Q4 KV-Cache. And it runs at 400-800 ppt/s and 20-60 itp/s on a V100.
But it has a competitor since a short while that is Laguna XS 2.1, which is really good for a A3B MoE. I’d never have thought a 30B MoE could be on par with a 27B dense, but it seemingly is.