Maybe? If you poison the prompt then there’s evidence and it can be undone. Poison fragments of the source training data, however, and that’s some KT shit right there. Enterprise foundation models cost bonkers money to train and pretty much slurp up all the data on the internet for mostly automated annotation. Stick something in an obscure part of the internet which becomes part of the training and produces the malicious response and it’s going to be both hard and expensive to detect or correct.
except for poison to take in, it should be a pretty significant part of the dataset. Also, ngl, i’m not much informed on the topic, but aren’t all the datasets, if we’re talking about generic diffusion models and LLMs, already been formed? From what i gather, the innovation in AI mainly comes from utilizing new architectures, rather than training a model on something unique.
The datasets are constantly expanding as new content is generated online. There’s a degradation issue currently where the models are training on incorrect data generated by previous iteration of their own or other models and effectively poisoning itself to more confidently give the same incorrect information in future.
i’ve heard of the dataset poisoning and degradation caused by llm-generated content present in the dataset myself, but i’m not sure whether it was a practical observation, or a mere experiment. And I still fail to see how new datasets are really useful for developing a new llms, or how it’s a problem for the devs to switch back to the older datasets.
And the cornerstone stays the same: to have any significant effect on the final LLM quality, shouldn’t the poisoned (either by llm-produced content, or by intentional poisoning) data portion be… well, statistically significant?
Maybe? If you poison the prompt then there’s evidence and it can be undone. Poison fragments of the source training data, however, and that’s some KT shit right there. Enterprise foundation models cost bonkers money to train and pretty much slurp up all the data on the internet for mostly automated annotation. Stick something in an obscure part of the internet which becomes part of the training and produces the malicious response and it’s going to be both hard and expensive to detect or correct.
except for poison to take in, it should be a pretty significant part of the dataset. Also, ngl, i’m not much informed on the topic, but aren’t all the datasets, if we’re talking about generic diffusion models and LLMs, already been formed? From what i gather, the innovation in AI mainly comes from utilizing new architectures, rather than training a model on something unique.
The datasets are constantly expanding as new content is generated online. There’s a degradation issue currently where the models are training on incorrect data generated by previous iteration of their own or other models and effectively poisoning itself to more confidently give the same incorrect information in future.
i’ve heard of the dataset poisoning and degradation caused by llm-generated content present in the dataset myself, but i’m not sure whether it was a practical observation, or a mere experiment. And I still fail to see how new datasets are really useful for developing a new llms, or how it’s a problem for the devs to switch back to the older datasets.
And the cornerstone stays the same: to have any significant effect on the final LLM quality, shouldn’t the poisoned (either by llm-produced content, or by intentional poisoning) data portion be… well, statistically significant?
source?