It’s not necessarily that any of the open models are cheaper to train (for similar model size), it’s more that China has deeper pockets. And MoE inference is cheaper than dense models I believe
If China manages to scale their hardware production, this could finally be “their day in the sun” where they clearly surpass the West in the way the West has outshone them for 100 years. MoE has its place, but a MoDE where each E is itself a dense model would be more powerful still, mostly you need the silicon gates and power to drive them.
It’s not necessarily that any of the open models are cheaper to train (for similar model size), it’s more that China has deeper pockets. And MoE inference is cheaper than dense models I believe
If China manages to scale their hardware production, this could finally be “their day in the sun” where they clearly surpass the West in the way the West has outshone them for 100 years. MoE has its place, but a MoDE where each E is itself a dense model would be more powerful still, mostly you need the silicon gates and power to drive them.
Three Gorges makes hydro-power, right? 22MW -> 100 TWh per year, just from that one structure, I bet that will run at least one AGI… https://www.fredgao.com/p/deepseeks-liang-wenfeng-breaks-his