it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
the linked paper is very good: https://arxiv.org/pdf/2309.10668
That’s like saying the neural network in your head has compressed your knowledge of language and just expands the relevant parts when needed.
yup