Hey, it’s Ian.

My wife asked me the other day, “Why haven’t you written about Substack’s AI thing yet? Everyone’s talking about it.”

I said, “I want to play with it first.”

My Write Sports edition from two weeks ago? 100% human. My Sunday Back Page issue from the same stretch? 86% human; 14% AI-assisted.

That gap is what made Substack’s new AI detector fun for me instead of annoying. I spent the last two weeks running the scanner on myself, on writers I follow, on writers I pay for, and on a stack of total strangers. The results say less about how much AI is out there and more about what nobody wants to admit about using it.

🕵️ Testing Substack’s AI detector on me and total strangers

🤔 Why 5% of writers turn detection off entirely

📰 Substack isn’t alone in cracking down on AI slop

👀 The punctuation mark gives away AI-assisted writing, and emmm, it’s not exactly what you think.

Substack turned on an AI detector on July 21, built in partnership with Pangram. Scan any post over 100 words and get an estimate of how much of it is human, AI-assisted, or fully AI-generated. Substack CEO Chris Best said he doesn’t want Substack to turn into a version of LinkedIn buried in content nobody actually wrote. He calls it Claudefishing. We’ll get back to that term later.

More than half of what I tested the scanner on, from strangers to people I follow to pubs I pay for, came back as 100% human. You didn’t see that number from all the hot takes during launch week. A lot of independent writing is still independent.

Weird fact: Some of the posts that broke that pattern were ones about the tool itself. More than one of the articles I read from Substack gurus about Substack’s AI detector showed at least a 30% mix of AI-generated or AI-assisted text. Irony is a fickle thing.

Close to 5% of the Substack content I tested had the detector turned off. You can do that. Interpreting that makes my head spin. Mind your own business, maybe. Don’t want me to know how much you’re leaning on it, maybe. I’ll bet for some, it’s something more philosophical I haven’t figured out yet.

With that in mind, you also can’t test paid articles on Substack unless you pay to subscribe to them.

Between 5% and 10% of the publishers left a note explaining their process. It pops up when you run the scan. The wording varies, but it clusters around two ideas:

When I ran the scanner on this article, it gave me the following score and the option to create a statement:

About 2% of what I scanned came back 100% AI-generated:

Pangram CEO, Max Spero, says the tool’s false-positive rate is about one in 10,000.

That number sent some publishers on a rant. Ghostwriter Alice Lemee argues the detectors are wildly inaccurate. She says a single false-positive can wreck a reputation nobody gets back. Sam Illingworth, a professor at Edinburgh Napier University, hit more sharply: Substack built a second machine, pointed it at human writing, and asked that machine to vouch for the person behind it.

Illingworth’s worry isn’t abstract. False flags tend to land hardest on writers working outside their first language, and on neurodivergent writers, the ones least equipped to argue their way out of a percentage. Writers are already calling the whole thing a witch hunt.

I edited stories at Newsday about an Adelphi University student who was accused of using AI to produce his work. He sued and won.

I use AI to help me.

Almost every word you read from me, I wrote first.

Three days a week, I walk 2.5 miles to my 9-to-5 at Newsday, and I spend a good chunk of that time talking through ideas out loud with AI, Claude to be specific. Is that also considered Claudefishing?

When I sit down to write something like this, I’ll use my rougher talking points from that dialogue and ask for an outline before I start typing. Other times, like this piece, I write until I stop, then ask AI to help me cut it down, suggest edits or even a better structure.

After that, it goes through my own AI-prevention checklist before anyone else sees it.

Also, Claude’s image generation is… underwhelming. I use ChatGPT to help in that area when I can.

The more you let AI shape your creative output, the less of you ends up in it. That’s even if you spend hours training it to sound like you. Use it enough, and eventually you’re editing a very polished draft of somebody else’s voice.

That’s a bigger risk for a sports beat writer than for almost anyone else on Substack. Nobody pays to read a summary they could pull off a wire report. The Associated Press has been doing this for years with small Division I college sports and other things, and it is bland.

Readers pay for the sourcing, the voice, the thing only that writer saw from the press box. A newsletter that quietly lets AI take over the sentences risks more than a bad Pangram score. It risks the one thing that made someone subscribe in the first place.

If you want a second opinion on whether your own newsletter still sounds like you, growth-wise and content-wise, the Write Sports Audit runs $127, and it’s included free with an annual subscription and the Creator Lab.

Substack isn’t the only platform drawing a line on AI content this month.

📰 Snap pulled fully AI-generated videos from Spotlight recommendations, and LinkedIn shipped a dedicated AI-slop reporting button. Every platform hosting your work is quietly deciding what counts as real.

I have a premium LinkedIn subscription, but I haven’t been given access to the AI slop button yet. This is different from Substack’s scanner. This is about reporting someone else you suspect of using AI. It isn’t something I can play with on a whim. There are bigger consequences at stake in that process.

When I first let AI help with my writing, it kept sneaking em dashes into spots where a comma would work. Or one of my favorite phrases, “period, end of sentence.” My biggest pet peeve is it kept using the em dash’s dastardly cousin, the en dash, between number ranges (you know the difference, right?). A few weeks after I spotted this pattern, I noticed that same en dash creeping into newsletters from writers I respect. Ah ha. Substack and Pangram got nothing on me! Constant vigilance, as Mad-Eye Moody says.

Thinking about trying beehiiv? Sign up through my link and get 20% off your first three months.

Full disclosure: that’s my partner link, and I earn a commission if you sign up through it.