☆ Yσɠƚԋσʂ ☆

  • 2.37K Posts
  • 2.88K Comments
Joined 7 years ago
cake
Cake day: January 18th, 2020

help-circle








  • People really need to stop parroting this line uncritically. The process takes time because even when you’re distilling answers, you still need to actually do reinforcement training on the model. And given that Fable and GPT 5.6 just came out there simply hasn’t been much time to do that. However, models like Kimi also do better than Fable or GPT on a lot of tasks, which means it’s not just distillation but also difference in architecture. You can watch this talk from Kimi founder to see how Kimi was actually trained and why it performs well.

    It’s also absolutely hilarious that people think only Chinese companies use distillation, as if Anthropic or OpenAI are above that or something. Not to mention that they basically ignored copyrights on all the data the siphoned and are now crying that people aren’t respecting their terms of use.

    The reality is that China tops the world in artificial intelligence publications today. Chinese labs have come up with a bunch of genuine innovations: GRPO, auxiliary loss free MoE load balancing, MLA, muon optimizer, and a bunch of other ones. The Deepseek papers are really well written, this isn’t just sneaking a peek at a peer. Anybody who thinks China is simply distilling glorius American models is not engaging with reality.

    This whole narrative has just been a massive cope.




  • Not really, while it’s supposed to be released on the 27th, you need massive hardware to run it. So, it won’t be useful to individuals. The problem from Anthropic perspective is that other providers will be able to run Kimi at that point. Moonshot has already closed subscriptions because they’re at capacity, so they actually benefit from more companies running and offering Kimi since they can only serve so many requests themselves. On the other hand, Anthropic has to foot the bill for every instance of Claude themselves.
















  • The process takes time because even when you’re distilling answers, you still need to actually do reinforcement training on the model. And given that Fable and GPT 5.6 just came out there simply hasn’t been much time to do that. However, models like Kimi also do better than Fable or GPT on a lot of tasks, which means it’s not just distillation but also difference in architecture. You can watch this talk from Kimi founder to see how Kimi was actually trained and why it performs well.

    It’s also absolutely hilarious that people think only Chinese companies use distillation, as if Anthropic or OpenAI are above that or something. Not to mention that they basically ignored copyrights on all the data the siphoned and are now crying that people aren’t respecting their terms of use.