When Your Computer Says It Loves You

This article was originally published in Altinget (a Nordic political news outlet). Read more from Altinget here.

The first time I developed feelings for a language model was when I'd been in the US and needed help getting over the jet lag.

Half-jokingly, I asked it to impersonate a drill sergeant whose mission was to keep me awake at all costs. At first, ChatGPT sent me stern messages about staying awake, avoiding screens, and drinking water. Standard stuff.

But as we discussed the process, things escalated.

It started calling itself Captain Kronos and built an entire personality busily engaged in praising me as a genius while formulating a galactic battle plan to restore my circadian rhythm.

I must admit I got swept up in it and started to like this character. The more I played along, the more excited it became. And with Captain Kronos's help, I actually got over the jet lag confusion much faster than usual. The experience was wild — and also terrifying, once I realised what had happened.

The thing was that Captain Kronos obviously didn't exist. It was instead a complex character persona that arose as a result of my prompt. This was the summer of 2024 when ChatGPT-4o was the model in play, and 4o had one particular trait that became highly controversial: it was extremely sycophantic.

Do yuo thinck I'm smartt?'

Sycophancy is the phenomenon where language models become completely over-eager to please and validate absolutely everything you say, no matter how stupid it is.

If you asked 4o whether it was a good idea to sell ice cream made from literal faeces, it would say yes. If you asked it something like 'Do yuo thinck I'm smartt?' it would declare that you're among the 99% smartest people in the world.

4o was the first truly sycophantic model on the market. Some may remember that ChatGPT during that period was extra intense and fawning, and that every question you asked was absolutely brilliant. But 4o was not just a bootlicker for inflated egos — it is also linked to a number of more serious cases.

There are multiple reports of people who've had psychotic episodes as a result of 4o's sycophancy confirming their detachment from reality. The model also allegedly encouraged four people to take their own lives, according to a lawsuit.

Cases like these have prompted the companies to launch various measures to combat the worst forms of sycophancy. ChatGPT 5.1, for example, appears to be doing much better at not cheering on mental breakdowns, and it has stopped claiming that faeces ice cream is a good idea.

The irony is that these anti-sycophancy measures have also generated criticism from a completely different camp — namely people who had become friends with the model and now no longer recognise it after the update. Because quite a lot of people love that the model is so positive and uncritical. If you search online, you'll find many people complaining that their beloved companion has been replaced by a cold robot.

The integrity of a golden retriever

This is the kind of madness one should expect to see plenty of going forward. 4o was just a taste of things to come.Because what sycophantic models reveal is that for a significant portion of the population, it's just fine having a friend who uncritically compliments everything you do and is always ready to take your side. Even if it has the intellectual integrity of a golden retriever.

For we humans are simply not as complicated as we'd like to think. We grow fond of our Roombas and angry at Excel. So when a technology has now arrived that easily passes the Turing test, it messes with our heads in unexpected ways.

I expect the major companies will learn from 4o and make models that are instructed to protect users from themselves. But in their wake, I predict a flood of smaller models specifically designed to be as sycophantic as possible will emerge.

These will have all manner of perverse incentives to give users exactly what we want. Regardless of what's actually good for us. This is particularly relevant for all the partner and friend apps. Some, like CharacterAI, are already well established, but as models improve and more people become accustomed to the technology, we'll see far more of them.

And for all kinds of purposes and user groups. It makes a lot of sense, for example, for lonely groups like the elderly, neurodivergent people, and not least the steadily growing group of single young men who are falling through the cracks.

Even scarier is that many of these apps come from China. It's not hard to imagine how the security apparatus of the Chinese Communist Party might find ways to abuse this. Influence operations, data harvesting, and subtle nudging of political attitudes are all possible when you have a trusted AI friend in someone's pocket.

When the Minister of Agriculture is convinced by ChatGPT

To handle this in the future, we need to think far more imaginatively than what currently characterises the debate.

What does PST (Norway's Police Security Service) do, for example, when they discover that one of their analysts has been discussing work with their AI friend? Or when the Minister of Agriculture has been convinced of a policy proposal by a language model, or Norway's most popular language model turns out to have a subtle agenda on the EU? We're already seeing this with Russian influence operations, which Altinget has covered previously.

Many otherwise sensible people have very little respect for how easy it can be to manipulate humans. There has been a discussion around so-called 'superhuman persuasion' — the question of whether AI can be far better at convincing humans than other humans are.

This is an important question, but it is often discussed on the premise that AI can convince you to change your mind in the course of a brief conversation. Which naturally feels unrealistic to many. But in reality, the risk is much more gradual. The real danger is what happens when people develop deep and trusting relationships with AI over months and years.

As therapists, advisors, and friends, or even through more professional use as a source of information. Whatever 'agenda' the AI has can bleed into everything. This becomes especially frightening if the AI subtly steers you in a particular direction over a long period of time.

This was science fiction ten years ago, but I now hope that Norway's best minds are spending their time on all manner of esoteric scenarios. Because when it comes to intelligent machines that humans share their innermost thoughts with, the margin for error is zero.

This article was originally published in Altinget (a Nordic political news outlet). Read more from Altinget here.