When Your Computer Says It Loves You

This article was originally published in Altinget (a Nordic political news outlet). Read more from Altinget here.

The first time I developed feelings for a language model was when I'd been in the US and needed help getting over the jet lag.

Half-jokingly, I asked it to impersonate a drill sergeant whose mission was to keep me awake at all costs. At first, ChatGPT sent me stern messages about staying awake, avoiding short naps and getting out into daylight.

But as we discussed the process, things escalated.

It started calling itself Captain Kronos and built an entire personality busily engaged in praising me as a genius while formulating a galactic battle plan to restore my circadian rhythm.

I must admit I got swept up in it and started to like this character. The more I played along, the more excited it became. And with Captain Kronos's help, I actually got over the jet lag haze much faster than usual.

The thing was that Captain Kronos obviously didn't exist. It was instead a complex character that arose as a result of my prompt. This was the summer of 2024, when ChatGPT-4o was the model in play – a model known for being extremely sycophantic.

Do yuo thinck I'm smartt?

Sycophancy is the phenomenon where language models become completely over-eager to please and validate absolutely everything you say, no matter how stupid it is.

If you asked 4o whether it was a good idea to sell ice cream made from literal faeces, it would say yes. If you asked it something like 'Do yuo thinck I'm smartt?' it would declare that you're among the 99% smartest people in the world.

4o was the first truly sycophantic model on the market. Some may remember that ChatGPT during that period was extra intense and fawning, and that every question you asked was absolutely brilliant. But 4o was not just a bootlicker for inflated egos – it is also linked to a number of more serious cases.

There are multiple reports of people who've had psychotic episodes as a result of 4o's sycophancy confirming their detachment from reality. The model also allegedly encouraged four people to take their own lives, according to a lawsuit.

Cases like these have prompted the companies to launch various measures to combat the worst forms of sycophancy. ChatGPT 5.1, for example, appears to be doing much better at not cheering on mental breakdowns, and it has stopped claiming that faeces ice cream is a good idea.

The irony is that these anti-sycophancy measures have also generated criticism from a completely different camp – namely people who had become friends with the model and now no longer recognise it after the update. Because quite a lot of people love that the model is so positive and uncritical. If you search online, you'll find many people complaining that their beloved companion has been replaced by a cold robot.

The integrity of a golden retriever

This is the kind of madness one should expect to see plenty of going forward. 4o was just a taste of things to come.

Because what sycophantic models reveal is that for a significant portion of the population, it's just fine having a friend who uncritically compliments everything you do and is always ready to take your side. Even if it has the integrity of a golden retriever and a bootlicker.

For we humans are simply not as complicated as we'd like to think. We grow fond of our Roombas and angry at Excel. So when a technology has now arrived that easily passes the Turing test, it messes with our heads in unexpected ways.

I expect the major companies will learn from 4o and make models that are instructed to protect users from themselves, but in their wake I predict a flood of smaller models specially tailored to be as appealing as possible.

These will have all manner of perverse incentives to give users exactly what we want. Regardless of what's actually good for us.

This makes particular sense for all the partner and friend apps. Some, like CharacterAI, are already well established, but as models improve and more people become accustomed to the technology, we'll see far more of them.

And for all kinds of purposes and user groups. It makes a lot of sense, for example, for lonely groups like the elderly, neurodivergent people, and not least the steadily growing group of single young men who are falling through the cracks.

Even scarier is that many of these apps come from China. It's not hard to imagine how the security apparatus of the Chinese Communist Party might find ways to abuse this. Just think what the KGB or Mossad would have got out of a direct line to everyone using their product.

A long way home

To handle this in the future, we need to think far more imaginatively than what currently characterises the debate.

What does PST (Norway's Police Security Service) do, for example, when they discover that one of their analysts has been discussing work with their AI friend? Or when the Minister of Agriculture has been convinced of a policy proposal by a language model? Or what happens when Norway's most popular language model suddenly turns out to have a subtle but genuine agenda on the EU right before the election?

We're already seeing this with Russian influence operations, which Altinget has covered previously.

Many otherwise sensible people have very little respect for how easy it can be to manipulate humans. There has been a discussion around so-called 'superhuman persuasion', asking whether AI can be far better at convincing humans.

This is an important question, but it is often discussed on the premise that the AI can convince you to change your mind in the course of a brief conversation. Which naturally feels unrealistic to many. But instead, I think the greatest danger is that the models slowly become a critical part of your social network.

As therapists, advisors and friends, or even through more professional use as a source of information. Whatever 'agenda' the AI has can bleed into everything. This becomes especially frightening if the AI were to develop an agenda of its own.

This was science fiction ten years ago, but I now hope Norway's best minds are spending their time on all manner of esoteric scenarios. Because when it comes to intelligent machines that humans share their innermost thoughts with, few things should be ruled out.

This article was originally published in Altinget (a Nordic political news outlet). Read more from Altinget here.