Architectural improvements could help, but the big guys can’t even get away from “basic” problems like high temperature sampling. Corporate development is way more conservative than you’d think.
And it’s not getting better. See the “all star” ego Tech Bro teams at Meta, OpenAI and such vs. researchers that quit.
The Chinese are testing some more interesting optimizations (and actually publishing papers on them), but still pretty conservative all things considered. They appear content with LLMs as modest coding assistants and document processors, basically; you don’t hear anything about AGI in their presentations.
Architectural improvements could help, but the big guys can’t even get away from “basic” problems like high temperature sampling. Corporate development is way more conservative than you’d think.
And it’s not getting better. See the “all star” ego Tech Bro teams at Meta, OpenAI and such vs. researchers that quit.
The Chinese are testing some more interesting optimizations (and actually publishing papers on them), but still pretty conservative all things considered. They appear content with LLMs as modest coding assistants and document processors, basically; you don’t hear anything about AGI in their presentations.