Oh wow, LLMs are having issues with using an esoteric language they haven’t been trained on. Might as well point them at a picture of Indus script and be baffled how it cannot parse it.
As long as its a language, an LLM will be proficient in parsing it given enough training and data.
This is a very interesting experiment though, surprised so see how some of the models faired.
I expect it would take about 15 minutes with the language definition to simply write a static tool (via an agentic coder, why not, just to prove the point) to evaluate the rules of the language, translate it to another (normal) language. Llm does the work there, then runs it through a mechanic to go the opposite direction.
Identification that this has occurred becomes an evaluation of the prose “obscured” values are embedded in.
I get the intent. I just think the testing methodology answers a very specific question, and forms a conclusion that over claims.
Oh wow, LLMs are having issues with using an esoteric language they haven’t been trained on. Might as well point them at a picture of Indus script and be baffled how it cannot parse it.
As long as its a language, an LLM will be proficient in parsing it given enough training and data.
This is a very interesting experiment though, surprised so see how some of the models faired.
I expect it would take about 15 minutes with the language definition to simply write a static tool (via an agentic coder, why not, just to prove the point) to evaluate the rules of the language, translate it to another (normal) language. Llm does the work there, then runs it through a mechanic to go the opposite direction.
Identification that this has occurred becomes an evaluation of the prose “obscured” values are embedded in.
I get the intent. I just think the testing methodology answers a very specific question, and forms a conclusion that over claims.
If you modeled your language around bio weapon, red teaming, and super racist language an LLM might actually refuse.
Rot13 every keyword in and out before the agent sees it (solution for anthropic models)
Keyword replace things to be even more offensive to make grok operate at peak efficiency.
Or use a relatively normal language but require code to declare tos-breaking things.
#error, line 47: ‘return’ called before declaring trump a pedophile.
#error, line 59: ‘while’ used in method before declaring intent to bomb a data centre.
This was what I expected to see, syntax full of slurs and questionable content
What would that look like?
The Uncle Fucker song from South Park: Bigger, Longer, Uncut prolly