• iamthetot@piefed.ca
    link
    fedilink
    English
    arrow-up
    14
    ·
    edit-2
    8 hours ago

    Worth noting that a token is not necessarily a word, though can be. One word could also take multiple tokens. It can also vary from LLM to LLM and their tokenization methods.