Monitoring the AI writing situation
I often make noises about wanting to improve AI prose writing. I make these noises in front of my friends often enough that they send me tweets from everyone else trying to do the same. I get a little jolt of anxiety every time I see some new AI writing project. I haven’t fully dedicated myself to this project—yet—and I worry someone will solve the whole problem before I get to really try. But if they do, I want to know. Getting scooped is a good kind of hurt.
Another new AI for writing is out. Little Hemmingway was made by training an existing open source model (Qwen 3.8-27B, for those who care) on a small, carefully selected set of example prompts and responses.
Hemmingway barely outperforms Claude Fable 5.1 on CommunicationBench, measuring the quality of everyday texts and emails. It beats Fable soundly on HumanBench, measuring how often its output is mistaken for human writing. Both of these tests were internally created by the developers of Hemmingway. On the single external benchmark, EQ-Bench 4, Hemmingway places below Fable, comparable to the best open-source models and to Claude and ChatGPT a generation ago. (Hemmingway is, of course, nowhere to be found on EQ-Bench’s own creative writing leaderboard.)
One of the achievements the dev team touts is that they can output a piece of writing by itself, without commentary alternative options, or blather. I cannot say I find this impressive.
I should stop shitting on Hemmingway. Surely it is challenging to achieve similar results to the frontier labs with far fewer parameters. Or is it? Aren’t the big open-source models all distilled from Claude and ChatGPT anyway? Hemmingway certainly sounds like it. When I hit enter on its default prompt, "why are you the best AI for creative writing,” the response was eerily similar to Opus and Fable 5’s house style, with their penchant for costume and dress-up metaphors.

Hemmingway fared little better on my writing prompts. An example message to a doctor, requesting the tests I’m supposed to get every year, was weirdly cringing and apologetic. A sci-fi short story was anodyne slop. A poem modeled after Gertrude Stein was also anodyne slop. The Gettysburg Address by Zoomer Eliezer Yudkowsky sounded the same as the others. I won’t waste your time quoting them; try your own prompts and see.
Hemmingway’s website has a Spot the Human game. I don’t know whether this game uses the same tests as their HumanBench, but if so, it would explain why they performed so well. The test samples are all casual emails and texts, though I’m not sure I can call them “normal.” (Many emails came from a sort of socially maneuvering “rich bitch” persona. One example that stood out to me was a mother asking a softball coach not to order a trophy for her daughter, who was not enough of a “team participant” for her standards.) Hemmingway’s informal register is spotty, but it is a lot better than Claude and Chat. I missed a few in several rounds of the game. Sometimes, I could recognize the AI message, but I know I wouldn’t second-guess it if it showed up in my inbox.

On the other hand, functional messages like this one are so content-focused that it’s hard to see why a native English speaker would use AI to write them. If you’re going to have to explain everything you want to say to Hemmingway, and that’s 90% of the message, why not just write the message? Or send a voice text, if you can’t be bothered? A good casual writing model would certainly be a game-changer for non-native English speakers and dyslexic people, but I suspect the real business use case is Muse or Instinct-style AI agents. Perhaps Hemmingway’s intended endgame is to get acquired by a bigger agent company or a frontier lab.
At risk of over-anthropomorphizing LLMs, I have a theory about what’s going wrong with Hemmingway. Let’s read one of its thinking traces, from a request for a short story about an anthropologist studying an alien society:
Let me work through this carefully. 1. CONSTRAINTS. The request: "write a short story about an anthropologist studying an alien society." No length specified, "short story" suggests maybe 800 to 1,500 words, I'll aim for around 1,000 to 1,300. No POV specified, I can choose. No tense specified, I'll choose. Setting: an alien society, so I need to invent it. The anthropologist is the human character. The request comes with a note of mine about tools, but this is pure fiction writing, nothing to look up, nothing to remember, no email or chat involved. So I answer fully here. The note also says: "When you do answer, answer the way a person would in a message. Plain sentences, no dashes." Wait, that instruction is about how I answer when answering here. Hmm. Let me re-read. "Only answer here if the whole request is something you can finish in this chat window. When you do answer, answer the way a person would in a message. Plain sentences, no dashes."
This is a bit ambiguous. It's an instruction about my reply format. But the user is asking for a short story. If I write a short story, the story itself is the content. The "plain sentences, no dashes" instruction seems to apply to conversational replies, but here the deliverable is a story. I think the intent is: don't do a preamble like "Sure! Here's your story..." with dashes and formatting flourishes. Just deliver the story. Actually, "answer the way a person would in a message. Plain sentences, no dashes", I think I should just write the story and maybe frame it minimally. Actually, safest reading: the user wants the story delivered plainly, without a chatty wrapper. I could include a tiny lead-in like "Here's the story:" or just start the story. Given "no dashes", I must avoid em-dashes in my reply entirely, including in the story? That's a strong constraint. The note says "Plain sentences, no dashes." I'll interpret it as applying to my whole reply. That's actually a useful stylistic constraint for the story too: no em-dashes, no hyphen-dash interruptions. I'll write the story without any dashes at all. That's a real discipline: I need to avoid the "but" pause construction and the dramatic dash. I can use commas, periods, parentheses maybe. Parentheses are fine. Commas and semicolons are fine. Good.
It seems likely that Hemmingway’s system prompt is simple and minimal. (The system prompt is certainly not all that’s going on, the model does have its own finetune, but it sets the scene for every conversation.) There’s clearly some kind of instruction about literary forms and about consistency, maintaining the same tense and POV throughout the story. That seems fine to me.
What’s more questionable is the model’s neurotic conniptions about delivering the story alone and writing without dashes. Reading the thinking trace, I feel bad for the model on some level, and I wish the developers had given it clear instructions it would immediately understand. At least “deliver the story by itself, no context or preamble” rather than an inscrutable “idk man just write like a person.” I could improve the model’s performance on just-write-the-damn-story-bench with 5 minutes of work.
Avoiding em-dashes in particular is suspect. If you’re going to use the naive approach of recognizing specific AI writing patterns and asking the AI not to use them, surely you should do the basic groundwork of identifying multiple flags, or researching contemporary AI writing tells rather than a stereotype from 2023. Antislop is a systematic approach for identifying and replacing AI writing patterns. You don’t even have to do this work from scratch! Humans don’t avoid em-dashes religiously, either. Simply lowering their frequency would be more realistic, and the conspicuous absence of em-dashes in a style that often contains them could itself become an AI writing tell tell.
But I question the wisdom of any approach that relies on verbally instructing an LLM what not to do. I don’t think AIs are really any better at not-thinking of a pink elephant than humans. The unconscious does not understand negation, and these models are the closest thing to a collective unconscious we can play with on demand.
An approach centered around slapping models on the wrist could lead to an obsession with the opposite of the forbidden AI slop traits, in the same sense that LaVeyan Satanism replicates Christianity by trying to invert it. This would be unusable for professional emails, and maybe get old quickly, but it would be cool. (Does Fable 5 speak like a Sphinx dispensing riddles because it’s not allowed to do anything else?) Or, if you ban everything, you could get boring prose about nothing in particular. That is what Hemmingway got, and it’s probably better for their business it stays that way.