This financial feature will be initially available to users in the US who subscribe to ChatGPT’s $200-per-month Pro tier.
Apart from the fact that there’s no way in hell I’d want a hallucinating LLM with privacy and security issues to see my health and banking data, I can’t quite get my head around the concept that there are users who willingly pay this much for access to it…
As someone actually running LLMs locally for testing, unfortunately I’m not sure I agree.
For any passable sort of performance, you want as much as possible running on the GPU. Best bang for buck here is 1088 euros for a 24 GB RX 7900 XTX or 1681 for a 32 GB Radeon AI Pro R9700.
Now, you can fit many models on 24 GB, but they’re so far in output quality compared to frontier models, that they’re not actually good for this task. But add MoE offloading and 256 gigs of RAM and you can get a 4-bit quant of qwen 3.5 397B runnin. That’s about 3 grand for RAM. You’d then also need a decent CPU.
For even better performance, you can get a 256 GB Mac.
The upside is that you never run out of tokens. Even the damn 200$ plans for Claude and OpenAI have 5 hour limits that you can run into and then you have to wait again. The downside is that it won’t output fast enough to actually have to consider running out of tokens lol
Apart from the fact that there’s no way in hell I’d want a hallucinating LLM with privacy and security issues to see my health and banking data, I can’t quite get my head around the concept that there are users who willingly pay this much for access to it…
That’s the best part, Palantir has already sold your health and banking data to ChatGPT.
If you’re paying $200+ a month for a tool that doesn’t work then i’m sorry but you deserve to potentially have your bank account wiped out by it.
I mean you’d use the 200 dollar tier if you keep running into usage limits because of Codex or something. There’s really no other reason for it IMHO.
At that point why not just run the model locally? 4 months of subscription would pay for a powerful enough setup.
As someone actually running LLMs locally for testing, unfortunately I’m not sure I agree.
For any passable sort of performance, you want as much as possible running on the GPU. Best bang for buck here is 1088 euros for a 24 GB RX 7900 XTX or 1681 for a 32 GB Radeon AI Pro R9700.
Now, you can fit many models on 24 GB, but they’re so far in output quality compared to frontier models, that they’re not actually good for this task. But add MoE offloading and 256 gigs of RAM and you can get a 4-bit quant of qwen 3.5 397B runnin. That’s about 3 grand for RAM. You’d then also need a decent CPU.
For even better performance, you can get a 256 GB Mac.
The upside is that you never run out of tokens. Even the damn 200$ plans for Claude and OpenAI have 5 hour limits that you can run into and then you have to wait again. The downside is that it won’t output fast enough to actually have to consider running out of tokens lol
See, if you lay off all your staff, it pales in comparison to their silly wages.