I promise this blog will have more than an engineer yelling at the clouds about LLM pain. I was actually working on a retrospective on X-Men Legends, or my experiences building an NES emulator in Rust. However slop PRs started hitting a codebase I maintain at work. And like any healthy individual I wish to commiserate about this with others on the internet.
I’d like to think I have some ability for introspection. Is it me that’s the problem? Am I just not with it anymore? It’s one thing to see the spam come in and go “man this sucks”. But it’s another thing to have to explain to misguided individuals why their behavior is destructive. This is my attempt at delving into the “why”. Why do these slop PRs bother me so much? Why do they feel like such a drain?
Surely we can come up with networks of trust for this sort of thing, so that you don’t have to deal with PRs from people with no references.
Everyone starts off without references, and there’s already less of a pipeline from user to helpful contributor to fellow maintainer than most projects want without having to add more chokepoints. There isn’t a solution without downsides while there are people using LLMs.
That’s true, but as a maintainer you could encourage those helpful maintainers to triage issues from regular users.
I think the real benefit would come from taking a user’s reputation into account across projects.
At the end of the day you can’t have low effort pull requests, and expect maintainers to look at everything. It’s the same spam problem as in any other domain.
This works if you have the luxury to select the people whose PRs you review, but in a corporate environment you just don’t have that option. I would love to just reject obvious LLM code, but it’s not going to keep me employed. Instead I’m stuck at figuring out how to meaningfully review LLM changes and how to manage the mental model with these rapid changes.