2x R9700, Ryzen 7700, 64GB RAM. The absolute best I’ve seen from llama.cpp was 60tps but the typical was 45-55tps. Radiance gave me 150tps on the first try. It can do parallel output while keeping 80-120tps with 2-4 streams. It can do more in parallel but I heven’t tested what the numbers look like. As far as I understand these numbers are possible on a single R9700 with the MXFP4 model. Haven’t tried it.
What did you switch to?
Radiance. Was using the latest llama.cpp. This thread caught my eye as simple enough to try.
What are the specs of the machine that runs it?
2x R9700, Ryzen 7700, 64GB RAM. The absolute best I’ve seen from llama.cpp was 60tps but the typical was 45-55tps. Radiance gave me 150tps on the first try. It can do parallel output while keeping 80-120tps with 2-4 streams. It can do more in parallel but I heven’t tested what the numbers look like. As far as I understand these numbers are possible on a single R9700 with the MXFP4 model. Haven’t tried it.