133
StoryScope: Investigating idiosyncrasies in AI fiction
arxiv.orgAs AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.
Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction
This isn’t what I’d call “reliable”.
I’m also not impressed with their methodology, which is heavily based on Gemini.
That’s too many steps removed from anything you’d encounter in the wild. They’re not even testing against human-prompted output.
And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.
Next up: AI adapts to evade detection by last week’s researchers’ detectors.
only 7% of human work got flagged for ai it is reliable though
Percentages are somewhat misleading here. That’s around 2 in 25, enough to falsely accuse a student or 2 in a fairly standard classroom for example.
I remember somebody mentioning a quote along the lines of “A 95% success rate sounds good, until you become the 1 in 20 that it kills”
In matters of life and death, the alternative choices are always important to consider…
5% chance of dying vs a 99% chance of living a horrible life of immobility and pain?
We all die eventually.
Getting your degree or being kicked out and banned from your college and still requires to pay back the loans. IS life or death.
It literally can determine the quality of life of a person for the next 30-40 years. If not forever. This is NOT something to take lightly.
I lived that perspective roughly from high school through sophomore year in University… after sophomore year I took a different view: “I am good enough, regardless of what arbitrary ratings these sad little men put on my transcripts.”
My reality is: I got a teaching assistantship - the D I received in Calc II didn’t weigh on that decision one iota - in contradiction of the admonishments of several “advisors” I was required to see while selecting course schedules for the coming semesters; then, I got a good job out of school and nobody ever read my transcripts to see that I had a 3.45 GPA undergrad and a 3.75 GPA in grad school. Nobody outside my review committee ever looked at my Masters’ Thesis, and I think one of them never really read the whole thing.
Once I “let go” about grades and ratings and all that bullshit, I got another C in Statistics I from a sad little man whose “son hates him too, like me.” Any amount of stressing about that grade and my performance in the class would not have changed the outcome - we were an Honors class of 3.3 GPAs and up, many 3.9s and up in there, and he gave all but one of us C or lower. Elsewhere - and overall - I believe I performed better academically due to not stressing about it.
The parking gestapo started patrolling a week earlier than their announced end of “free parking for summer” period and wrote me a ticket, I appealed, they denied, I decided to see where this led without me paying. For a $20 fine, they sent me about 100 collection letters (postage far higher than $0.20 on each one) over the following years, and told me that “my transcripts are FROZEN and will not be released to anyone until the fine is paid.” Welp, here we are, 40ish years later, and nobody has denied me employment or promotion or any other thing because they can’t get a copy of my transcripts. One new employer 12 years after graduation asked me for a copy “to have on file” and I gave them an earlier copy I had pulled before the parking ticket, didn’t show degree conferred, they didn’t care - gave me Masters’ degree pay rate anyway. Nobody else, anywhere at any of the dozen employers I have had in the last 40 years, ever asked me for transcripts or other proof of my education.
By the time you’ve had something impact your quality of life for 10 years, that is an impact to your entire quality of life. If the University actually expells you over an AI infraction, my take is that they probably wanted to expell you for other reasons already, and your academic career at that institution would have been an unpleasant uphill battle even without the AI flap.
One other thing I learned during my Teaching Assistantship: a couple of my students were demonstrating absolutely zero understanding of the material being taught and making zero effort to improve their understanding, resisting my offers to help. I gave them a failing grade. My advisor was called to the Dean’s office, when he returned he sat me down for a lesson: “These are paying customers, if they show up to class they get at least a C.” I suspect that is true of almost every private university in the world.
Okay so what I’m taking away from this is that you were lucky to find gainful employment without proof of a degree and didn’t have any debt to pay off? Because the context here is people having all the debt and none of the credentials required for jobs where they make enough to pay off that debt. If your card is declined, how you feel about it doesn’t matter.
I had student loans, luckily they only amounted to 4-5 months’ gross income - lucky because scholarships stepped up and I only had to take a loan for one year instead of all six.
I got news for you: you can have all the credentials in the world available from the best Universities, and when a place isn’t hiring, they aren’t hiring. I’ve gone job hunting in Melbourne Florida after Columbia burnt up on re-entry, the entire town wasn’t hiring - that’s not about me, they weren’t hiring anyone.
I interviewed in Manhattan with a fresh Honors degree from a good school, the employment agents there dumped all over me about how worthless I was, until… I wandered into an office where the interviewer had a glass table, she looked down through it and saw my shoes were freshly shined, my silk tie was bright red ($2 from a street vendor), and I had a goofy-trendy $50 haircut that was in-style, she gushed: “oh, I know exactly who will hire you: Kidder Peabody, you’ll so fit right in.” She barely glanced at my resume’ “you do computer stuff, right? Yeah, they’re always looking for more computer guys, they’re just gonna love you!” Transcripts were never mentioned.
After .com hit, any remotely computer related degree, straight out of school was a ticket to more than double what I made starting out with the best computer degree for .com, those kids left school making more than my boss with 25 years experience was making 5 years’ earlier.
Market gets saturated, hiring stops again. It’s different in different fields, but still much the same: cycles. Personal referrals beat the HR screening circus. The place I work today, I had applied to 4 times in the previous 15 years, HR never even called me back. Personal referral got me a work from home spot at a company this company ended up buying, since they bought me I bypassed the HR filters. Once I was past the HR filters they offered my an 8 month salary signing bonus to stay with them at least 1 year, my base pay has increased faster than inflation for 12 years straight now, and the annual bonus runs about 2 months salary.
I’m not revealing any useful “secrets” here… truth is: the system sucks, when it’s not blowing chunks. But, that’s the world - and trust me when I tell you: academics A) don’t know everything about how the rest of the world works, and B) what they do know they tend to distort for their students in various ways, for various reasons.
Any rant against academia this long deserves this link: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf
Holy the ignorant self-centered anecdote. Too long, no relevance, do not recommend. I‘d probably give it a failing grade.
And after age 19 or so, I wouldn’t care.
Traits of the ignorant, as one would expect.
Dude if my code had an error rate of 7%, my boss would not say that this “good enough”.
That’s not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly
this is just spherical cow fallacy combined with purity culture slop
no methodology is perfect, it just needs to be practically valid, using ai != killing people for using ai
unless you are for killing people who use AI, which wouldn’t surprise me one bit about Anti-AI horde
Tell that to the student who gets expelled because they’re writing was falsely identified as AI and then takes their own life.
Tell that to the person falsely identified with facial recognition and is arrested and loses their job.
Using AI to detect AI is as bad as using AI for any important decision.
This isn’t hypothetical. It happens. People will lose their livelihood or die because of it. 7 in 100 is not accurate enough when any penalty is applied to detection.
The key is for the detector to be the screener, innocent until proven guilty, and a screener tool with 7% failure rate is not proof.
Back when I was in Uni poor grades were considered a lifelong sentence to poverty…
To be fair it still is. It’s even worse if your expelled and have to pay back the loan any ways.
As I just wrote elsewhere - that “lifelong sentence to poverty” was, largely, a misconception of the world held by children who haven’t experienced how things actually work in larger society - only listened to what the academic keepers of paper titles have told them. Ivan Illich isn’t far wrong: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf