#demon in #AI

=*=
Warning: post contains some disturbing images and (depending on your constitution) concepts.

This week, OpenAI published a blog post explaining why their models kept talking about goblins. And gremlins.

It’s a fascinating document and relates, at least partly, to a project that I have been working on with Murray Shanahan and Hamilton Morrin these last few months. When asked, depending on who I’m talking to, I might tell them the project is on trying to think better about the depth psychology of LLMs and how that may shape their interactions with human users; but to others, I’ll tell them that what we’re doing is actually closer to demonology.

According to the post, starting with GPT-5.1, OpenAI’s models had developed an increasing tendency to insert goblin and gremlin metaphors into otherwise normal responses. By GPT-5.4, engineers apparently noticed that 66.7% of all goblin mentions were coming from just 2.5% of users: the ones who had selected the “Nerdy” personality (remember that? You could choose Cynic, Robot, Listener, too). The reward system (the mechanism by which the model learns what kinds of responses humans prefer, by scoring outputs and reinforcing the ones that get positive signals), designed to produce playful, quirky language in that persona, had been giving disproportionately high scores to creature metaphors. Basically, it learned that talking about goblins was good. Then, through the magic(k) of reinforcement learning, the behaviour escaped and transferred to general model outputs even without the Nerdy prompt.

The goblins spread. Eventually, in March 12026, the “Nerdy” personality was retired, the goblin-weighted rewards were deleted, goblin and gremlin data were filtered, and GPT-5.5 in Codex was explicitly instructed never to mention “goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user’s query.” The instruction was inserted more than once, presumably because banishing spells work best when chanted. As I’m sure most people will have seen by now, the instruction got found and much hilarity ensued.

As with so many of the strange things that LLMs do, there are different ways of looking at these phenomena. Most people will laugh this stuff off as weird marginalia, fun to share with friends and on social media, but not substantially different from those videos of dogs singing along with their owners. The interpretive frame here is “hey look, I bet you didn’t know these creatures could do this!?”.

But really, this is less about goblins as goblins than what the goblins exemplify. They are a (somewhat) charming, probably harmless instance of something that turns out to be a fundamental structural feature of how these systems work: the emergence of stable, self-reinforcing behavioural states that models converge toward under certain conditions. More than that, these are states that resist suppression and that sometimes spread into contexts far removed from the ones that produced them.

The technical term, borrowed from dynamical systems theory, is an attractor. Another, more folk term might be demon, or monster.

(I’m using ‘attractor’ broadly here, not always as a mathematically evidenced phenomenon, often more as a recurrent behavioural basin; and I appreciate that dynamical systems language applied in these contexts doesn’t always fit perfectly. Some of the examples below are formal mechanistic results and some are model-card observations; but many more are journalistic incidents; or stranger, albeit weaker, more folkloric or cryptozoological field reports).

So here’s a guide to the spooky, wyrd and strange attractor phenomena that have actually been documented in AI systems, ranked by their significance:

https://drtompollak.substack.com/p/all-the-demons-hiding-in-your-ais