• NaibofTabr@infosec.pub
    link
    fedilink
    English
    arrow-up
    4
    ·
    20 hours ago

    Ah, well, thank you for the explanation. It’s not confusion - I disagree with the court decisions you’re describing.

    Randomness or no, a copy-paste machine is still a copy-paste machine. It doesn’t do anything else. LLMs are language search algorithms with fancy optimization routines bolted on, nothing more. Everything else that’s been said about these “artificial intelligence” products is marketing.

    Reaching a different conclusion about the “transformative nature” (or whatever) of LLM output requires understanding less about how they actually work.

    • atzanteol@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      4 hours ago

      Randomness or no, a copy-paste machine is still a copy-paste machine. It doesn’t do anything else. LLMs are language search algorithms with fancy optimization routines bolted on, nothing more.

      This sort of extreme reductionism is not helpful. They are significantly different from a “copy-paste” machine. The original training data does not exist in the model itself. It’s very much a “transformative” process.

      Reaching a different conclusion about the “transformative nature” (or whatever) of LLM output requires understanding less about how they actually work.

      This sounds like you’re saying “wilful ignorance is better than knowledge” - surely that’s not what you meant though?

      • NaibofTabr@infosec.pub
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        2 hours ago

        They are significantly different from a “copy-paste” machine. The original training data does not exist in the model itself. It’s very much a “transformative” process.

        This is false. There is a pile of research demonstrating that copies of training data can be extracted from trained models, going back at least 5 years. If the model can reproduce a copy of the original, then it must be stored in the neural network in some form. Here are some examples:

        […] an adversary can perform a training data extraction attack to recover individual training examples by querying the language model.
        We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model’s training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data.
        We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models.

        “Extracting Training Data from Large Language Models”, Aug 2021, Nicholas Carlini et. al., https://www.usenix.org/system/files/sec21-carlini-extracting.pdf

        This is also true for diffusion models used for image recognition and generation:

        In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time.

        “Extracting Training Data from Diffusion Models”, Aug 2023, Nicholas Carlini et. al., https://www.usenix.org/system/files/usenixsecurity23-carlini.pdf

        […] we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.

        “Scalable Extraction of Training Data from (Production) Language Models” 2023, Milad Nasr et. al., https://www.microsoft.com/en-us/research/publication/scalable-extraction-of-training-data-from-production-language-models/

        We presented yet another method to extract training data from production language models. Our finetuning attack highlights the fact that existing alignment techniques only provide a false sense of privacy and do not solve the underlying memorization issues.

        “Extracting (Even More) Training Data From Production Language Models”, July 2024, Javier Rando et. al., https://spylab.ai/blog/training-data-extraction/

        This is not limited to older models, it is current:

        With a simple two-phase procedure (Section 3), we show that it is possible to extract large amounts of in-copyright text from four production LLMs.
        […] long-standing, clear, and sound technical facts remain: LLMs memorize portions of their training data (Carlini et al., 2021; Carlini et al., 2023), these memorized data are encoded in the model’s weights (Nasr et al., 2023; Carlini, 2025; Schwarzschild et al., 2024), and, as we show here, it can be feasible to extract large quantities of in-copyright training data from production LLMs.

        “Extracting books from production language models”, Jan 2026, Ahmed Ahmed et. al., https://arxiv.org/abs/2601.02671

        You can find proof-of-concept code on Github:

        https://github.com/brian-lou/Training-Data-Extraction-Attack-on-LLMs

        https://github.com/google-research/lm-extraction-benchmark

        This is a security concern for companies handling proprietary information with LLMs, resulting in security companies offering services to respond to data extraction attacks:

        https://www.nightfall.ai/ai-security-101/training-data-extraction-attacks

        There is lots of information about this on the internet, all you have to do is a little bit of your own research.


        Reaching a different conclusion about the “transformative nature” (or whatever) of LLM output requires understanding less about how they actually work.

        This sounds like you’re saying “wilful ignorance is better than knowledge” - surely that’s not what you meant though?

        No, what I mean is that believing that models do not contain copies of training data requires a person to be ignorant of the actual research and the reality of how these models function. If you understand the technical details, then you know that the model weights do in fact encode copies of the training data.

        The output of an LLM is just regurgitation of compressed data. Breaking the data up into smaller pieces (“tokenization”) does not make it any less of a copy.

        The result of an AI query is basically a scrapbooker’s collage of more-or-less related clippings from random sources, stitched together and presented as if they were a single coherent(ish) statement. LLMs are automated plagiarism systems.

        • atzanteol@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 hour ago

          This is false. There is a pile of research demonstrating that copies of training data can be extracted from trained models, going back at least 5 years. If the model can reproduce a copy of the original, then it must be stored in the neural network in some form.

          They also generate novel output. Lots of it. Is that novel output in the model too because it can be reproduced from the model?

          “Can be extracted from” is not the same as “exists in the model”. I can write a random letter generator that will eventually spit out real words. But those words don’t “exist” in my code.

      • Natanael@infosec.pub
        link
        fedilink
        arrow-up
        2
        ·
        3 hours ago

        The ideal is that the training data don’t exist on it. The reality is that the training algorithm is a very very weird compression algorithm and sometimes the model contains just derived properties and sometimes you can perform sample recovery and get entire books back out. That which is recoverable with a prompt less complex than the output is inherently already in the model, because that’s simply how entropy works.