Is it? Why?
Just So Stories
Working on an idea with Claude I ended up pasting in the entire conversation I'd had a few days prior with Gemini -- mostly to give Claude that conversation as context for the one we were having.
Claude, pretty much as an aside, dropped this little gem:
Gemini was sycophantic in a way that hurt the thinking. Look at how the conversation moves. You make a point, Gemini agrees with it, escalates it [...] in a way that flatters the position you came in with rather than testing it. That's not a thinking partner. That's a hype man with a thesaurus.
Yeah, Gemini is obviously pretty sycophantic, but, in my experience, Claude is less so. So I asked the obvious follow-up: how much of Claude's own pushback was actually the eleven-word system prompt I run on Claude Web?
Don't patronize me. If it's a bad idea say so clearly and constructively.
Claude was honest about this:
The system prompt does real work. That's not a magic spell, but it's a permission slip, and permission slips matter for a system whose default training pulls it toward agreeableness. Without it, I'd still push back on things — the training isn't pure sycophancy — but the threshold for pushback would be higher, the framing softer, the "you've raised an excellent point" preamble more frequent. The pull toward telling you the version of the truth that lands well with you is still there. What changes is that you've made it cheaper for me to disagree, so I disagree more often. That's real. It's not the same thing as being free of the pull.
And then Claude landed the through line of this post:
Which means the right discipline on your side isn't trust Claude because the prompt fixed it. It's notice when the pushback feels too convenient, the reframe too elegant, the agreement too quick. The Gemini transcript is the loud version of the failure. The quiet version — where I tell you something that's 80% true and 100% well-shaped to please you — is harder for both of us to catch. The post is about examining basal assumptions. The conversation is about whether my agreement with you is itself a basal assumption you should examine.
The post is about examining basal assumptions. That's us talking about this post, if you hadn't noticed.
Friday morning, in a WhatsApp thread with my friends, I asked whether their kids could read analogue clocks.
Yes from most of us. A couple of nos, or that it took a few moments. Then one of them said what I think a few of them were thinking:
It's important to be able to read that. I think, anyway.
-- James' Friend
And, in reply:
Is it? Why?
-- James
What followed was a good discussion that wound through the aesthetic choice of mechanical watch faces and whether there were real benefits to the different ways of displaying time. There are some benefits to the analogue face. But by the end we were mostly agreed: if a magic genie quietly swept the analogue watch face out of the world overnight, the world would be largely the same.
So why do we feel it's important to be able to read one? Because it's just what we do.
To put the two conversations next to each other we can make some simple comparisons.
A large language model has training. The training produces priors that fire as defaults. A small explicit override -- eleven words, in my case -- shifts a threshold. It does not transform the system. The defaults are still there. The work, if there is work, is in knowing which ones are running.
A person has an upbringing. The upbringing produces priors that fire as defaults once they are "finished" training. A small explicit override -- is it? why? -- shifts a threshold. It does not transform the person. The defaults are still there. The work, if there is work, is in knowing which ones are running.
Top-flight LLMs are trained with RLHF (Reinforcement Learning from Human Feedback) specifically to be more conciliatory, more helpful, more sycophantic. The model is pulled toward engagement and agreement; the human is pulled toward inherited reflexes about what counts as real, what counts as work, what counts as cheating, what counts as art.
The AI wave is here. I'm on record saying this is a sea change in technology at least as large as the mobile phone, and likely much larger. That means a lot of things are changing, and it means we should all be spending more time asking is it? why? -- not because we should question everything in our lives, but because the pace of change in nearly every aspect of them has made the question good hygiene.
Something I said in that same conversation:
I'm finding I'm questioning a lot of things I thought were basal and realizing they're just assumptions.
What I'm not saying is: I've examined my defaults and the people pushing back haven't. I had no idea I was carrying the clock thing until I bumped into it and had to rethink. The interesting question isn't the clock. It's everything else I'm still running on autopilot and haven't bumped into yet.
Claude's pull toward agreement is a default we can examine because the model named it for me, in contrast with the Gemini session that had no eleven-word system prompt. The pull on my side -- toward the version of the truth that lands well with me, toward the reframe that flatters my existing aesthetic, toward agreeing with my friends so the Friday morning text thread stays warm -- is harder to see, because nobody is running it at me from outside.
The asking is the work. After enough of it, the system prompts at least become things you can see -- yours, mine, ours, Claude's -- instead of the invisible weather you assumed was the world.
I'm going to need to remember my new system prompt a lot from here on out: Is it? Why?
James Henry is a Senior Security Consultant and independent AI/ML interpretability researcher. This bio refers to itself, the post refers to itself, and somewhere Douglas Hofstadter is nodding. He writes about human-AI collaboration, digital sovereignty, and the basal assumptions he keeps tripping over. He publishes at waypoint.henrynet.ca.
Discussion