Accenture sells AI transformation to thousands of clients while quietly telling its own employees they get a fraction of what some were already spending. The internal memo reveals a tension at the heart of every major consulting firm racing to profit from AI.
If you’re spending $1K per month on LLMs – putting aside whether this is productive or ethical for a moment – wouldn’t it make more sense to just pony up the $5K-$10K or so that it would cost to provision a private, local model? Might that not actually be superior in terms of corporate security?
We have a few dgx sparks in the office that are gathering dust right now because nobody wants to bother with them (and because IT doesn’t allow them on the network because Linux is scary so they’re a pain to get set up).
Matching the level of productivity of frontier models on the local models is possible to a limited degree (depending on the exact task), but takes getting used to. For the local models you have to lean into the fact that tokens are ‘free’ so its fine to burn a lot of them. (Whereas with frontier models the recommendation is to take a higher model for complex tasks because a lower would take more tries to get it right)
Not at the moment. It’s definitely cheaper to buy LLMs as a service than running an in-house server hall. Anthropic and OpenAI are running at a loss, you know.
Now, risking to lose your data to the service provider is another thing.
Look at it this way, if I pay Open AI $1k a month, their people make sure it works they month. If I let Joe from finance spend $10k on computer parts, Joe needs to ensure that it runs or I need to hire someone who is knowledgeable on local models. If I’m a simple firm, $12k a year might be cheaper because someone else makes sure it works and I can fire Joe when the fancy autocorrect loses a bunch of clients by falsifying work. Then I don’t have to spend even bigger bucks hiring a new Joe because no one knows how to work the damn server he replaced the break room with.
Definitely. We do this at my job. But a server big enough to provide a comparable service would cost at least $30k. And you likely need more than one. And you still need someone to maintain it.
I’ve heard that GPU’s for this work are exceedingly hard to get ahold of as well. Glm 5, which i think is near frontier model capabilities, takes around 1.5 Tb of vram un-quantized.
Not hard. Just really fucking expensive and might have a long lead time.
So hard, then.
It depends, sometimes the private API models are better at a specific task to the point it wouldn’t be worth using one that is worse. (Really it’s the harnesses around them tbh)
Overall sovereign AI is where it’s at. If you have good enough tooling in place