LLMs predict the next best “token”. So what’s a token? It can be a word, a simple punctuation mark, or sometimes even a phrase.

So basically, here’s what an LLM works like:

1. An
2. An apple
3. An apple is
4. An apple is red

Basically, it feeds in tokens to itself to predict the next best token. Then, that entire chunk of tokens is fed back in to predict the next token.

As you can see, tokens kinda correlate well with energy consumption, hardware wear and tear and so on.

So, a low token request will consume less resources than a high token request. Hence, it makes sense to charge per token.