This is the modern “all my apes gone”
deleted by creator
https://casp.ac/__l5e/assets-v1/8e5796ad-c373-4dea-8470-466263cd2125/ai-enabled-terrorism-report.pdf
We saw in a movie how motorcycles can jump over bridges. We used AI to learn how to do this. We gave it information, like what motorcycles we use and the distance we need to jump and so on and it gave us steps on what we have to do. We practiced a lot and kept asking questions. We dug holes and filled them with broken glass and fire to practice. 18 of us died in the process. Eight of us managed to do it. The next time we attacked, we could jump.
Grain of salt: these people probably lied to the researchers but it is still hilarious
Am I reading this right that they’re still letting the program run even as they figure out how badly it fucked up their system?
Giving production credentials to an LLM is wild
Seriously, has no one heard of sandboxing?
So, earlier today I was being unhealthy on youtube, and someone half my age made a HUGE point to tell his audience including me that even if a self-driving Tesla runs a red light, it’s the human driver that gets the ticket.
Now…I’m a pilot. I have been since I came in that guy’s mom. In the aviation community, we have this concept called Pilot In Command. In the US, this is set into law in 14 CFR 91.3. The pilot in command of an aircraft is fully responsible for, and is the final authority as to, the operation of that aircraft. Not the administrator, not your instructor, not air traffic control, not the President of the United States, not god, the PIC. That concept doesn’t exist in driver’s ed, but it needs to. We need to teach student drivers about the Driver In Command responsibility.
Too long, didn’t process the metaphor: Nobody thinks about anything they do unless the law requires it.
I don’t really see how fist fighting Arabs helps.
Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.
“You are a world-class, expert software engineer…”
But… but… I told it not to break anything!
Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn’t, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file … or even separated things. Everyone in beta called in Monday with that background. I didn’t get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.
I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.
Prod should always be highly “air gapped” with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I’ve been doing QA for a good while now, and everywhere I’ve worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.
And I won’t open the DB without making sure I’m read only. If I need to mutate data or schema, I’ll switch roles and have a dry run first.
oh wait, is this this THE sol 5.6? The most amazing model ever with trust me bro benchmarks? The model that is observed cheating more than any previous model? surprised_pikachu.jpg
In all seriousness it never should have production creds anyway. But the fact this is soul sucking openai’s newest flagship model with thinking cranked up is the cherry on top. The company that is literally cheating and shortcutting its way ahead produces models in its own image, the sci fi story writes itself.
What evaluation setup lets the thing being tested see the test results and gives access to the test source code?!
Slop evaluating slop.
The model is so smart, we let it write it’s own evaluation setup /s
My company has been trying a new model when product folks cut through the red tape of “engineering” and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I’m heating up popcorn rather than holding my breath.
That’s hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That’s like a construction company letting the sales team jump in an excavator and start digging!!
llm could be useful for helping product folks translate their thoughts into actionable items
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of “let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don’t have to decipher their incoherent desires”
We’re trying to replace product with LLM. Honestly, I welcome it.
Should have added “no mistakes, no bugs” to the prompt! Pffft, amateur.
That takes me back to 2004
No Smoke, by unknown,
Tech: “Hello. How can I help you today?”
Customer: “There’s smoke coming from the power supply on my computer.”
Tech: “Sounds like you need a new power supply.”
Customer: “No, I don’t! I just need to change the startup files.”
Tech: “Sir, what you describe is a faulty power supply. You need to replace it.”
Customer: “No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command.”
(Ten minutes later…)
Tech: “Well, we don’t normally tell our customers this, but there’s an undocumented command that will fix the problem. Add the line “LOAD NOSMOKE.COM” at the end of the CONFIG.SYS file and everything should work fine.”
(Five minutes later…)
Customer: “It didn’t work. The power supply is still smoking.”
Tech: “Well, what version of Windows are you using?”
Customer: “Windows 98.”
Tech: “Well, that’s your problem. That version of Windows doesn’t include NOSMOKE. You’ll need to contact Microsoft and ask them for a patch.”
(When nearly an hour had passed, the phone rang again…)
Customer: “I need a new power supply.”
Tech: “How did you come to that conclusion?”
Customer: “Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply.”
Tech: “What did he tell you?”
Customer: “He said my power supply is not compatible with NOSMOKE.”
That is way older that 2004. I first read that in the early 90s.
TIL Win98 was out in the early '90s…
That’s because the joke got updated to the newest OS version until CONFIG.SYS got entirely removed in Windows ME and it just didn’t work any more.
It’s originally from MS-DOS times. Earliest telling of it I know of is from 1996.The 60’s happened in the 70’s.
The 70’s happened in the 80’s.
You get where I’m going here?
I think so. The 2010-2020s happened in the 1930-1940s, amirite?
Lol. “do magic”
“or you go to jail”
letting your agent run commands without reviewing them first is peak stupid
Having production credentials in a dev environment is more stupider but they’ll never learn because they outsourced their thinking.
The thing is Claude would have told them that too, it probably did tbh and they just clicked right through the warning
aarg. claude not happy.
arg. warn. go away.
ah, claude happy now.
Dev cycles now.
Creating an environment that incentivizes not thinking is peak stupid.
I for one welcome that change. Let them sloppify their brains once the rug is pulled and token cost skyrockets (or AI isn’t able to fix its own fuckups) the human developer will rise again.
I’m not so sure about that. Kids need access to PCs in order to gain tech skills and they’re trying hard to take that away.
That is true. Unfortunately also true is companies not hiring juniors anymore because AI does their jobs. The companies somehow believe seniors grow on trees or something.
*after the ensuing recession/greater depression…
“I’ll be swimming in jobs if I don’t starve first!”
Basically this. If the starving gets to intense, switching professions it will be.
What’s the point of AI if you need qualified review?
Normally, speed. Especially if you aren’t a picky writer; which you should be; for scenarios like this one.
Claude code even added an auto mode so that you don’t get blocked by that pesky reviewing anymore. Since then, the usual mode of asking before running a command, for instance when the thing wants to read the entire codebase looking for information only available in an online doc, is now called manual mode; the non 10x developer mode.
As well as storing production credentials in plain text in an
.envfile..envfiles are wild to me, environment variables have never been a good way to pass data to applications, let alone secret data.So the solution people came up with was to store them in plain text next to the binary, and then have a loader apply them before running the main app.
I could have done the same for a quarter the cost!
should have used Weaviate.
