The AGI debate is all wrong: AI is Intelligence Upside Down

There's a lot of talk of insanely-powerful AI models at the moment, but there isn't enough talk about just how weird artificial intelligence is:

It can win a gold medal at the International Mathematical Olympiad but can't count the letters in a word.
It can synthesise 30,000 news articles into a coherent narrative but lose the thread of a three-message conversation.
It can produce strategic analysis that would impress a CEO and then misspell their name.

There are hundreds more examples like this - and all of them illustrate the problem with the AGI as a benchmark.

On Monday a paper in Nature argued AGI has already arrived. Within hours, the man who actually coined the term pushed back. They'll go back and forth for years.

But both sides are asking looking at AI in the wrong way.

The interesting thing isn't whether AI has reached human-level intelligence. It's that AI's intelligence is upside down. The hard stuff is easy. The easy stuff is hard.

And as soon as you realise that, it unlocks a cheat code for how you should actually work with it.

Intelligence built in the wrong order

Here's what's happening. Four researchers at UC San Diego just published in Nature arguing that we already have AGI.

Their evidence? GPT-4.5 passes the Turing test 73% of the time. LLMs are winning olympiad gold medals, proving theorems and diagnosing diseases. By any reasonable standard, they wrote, this is artificial general intelligence.

Shane Legg, co-founder of Google DeepMind, the man who actually coined "AGI" over twenty years ago, disagrees.

His argument? If a system still fails at tasks any human can do trivially, it hasn't crossed the line. "Minimal AGI" requires a machine that can do all the cognitive things people can typically do. Not some of them brilliantly and others not at all. All of them.

They're both right about the evidence. They're just looking at different floors of a building that was constructed upside down.

Every intelligence we've ever observed builds from the bottom up:

Babies learn to count before they learn calculus.
Students learn grammar before they write novels.
Junior analysts learn to check their numbers before they produce strategy memos.

The assumption, so deep it's almost invisible, is that simple cognitive tasks are foundational. You master them first, then build upward.

Then LLMs arrived and did it backwards.

They didn't start with routine cognitive tasks and work up.

They started at the top.

Synthesis, analysis, creative problem-solving, cross-domain reasoning, all turn out to be easier for LLMs than counting, basic consistency, sequential logic, or simple arithmetic.

There's an old observation in AI called Moravec's Paradox: what's hard for humans is easy for computers, and what's easy for humans is hard for computers.

Chess, calculus, data analysis? Easy.

Recognising a cat in a photograph, walking across a room, catching a ball? Incredibly hard.

A famous xkcd comic from 2014 that captures this perfectly:

xkcd tasks.png

That comic made sense for years, and Moravec's Paradox drew the line between cognitive tasks and physical/sensory tasks.

But now with LLMs, that line has moved inside cognition itself.

That's what makes this moment genuinely new.

Not that AI is getting smarter, that's to be expected. What's weird is that it got smart in the wrong order.

Not just autocomplete

So how did this happen? The most common explanation for how LLMs work is that they "predict the next word."

It's the line everyone reaches for when they want to sound informed. And it's technically true. That is what LLMs do. They take a sequence of text and predict what comes next.

For a couple of years, that felt like the best explanation I had. Not a dismissal or trivialisation of these tools, just a genuine attempt to understand the mechanism.

Then I started noticing how badly it failed as a mental model for actually working with these tools.

The thing that really shifted my thinking was a piece of research Anthropic published last October. One of their experiments forced Claude to output a word it hadn't planned, like "bread." The model treated it as a glitch. Something had gone wrong.

But when the researchers injected internal evidence that "bread" had been intended earlier in the process, it accepted the word and built a plausible reason around it.

It was checking some kind of internal record of its own prior processing to work out whether it had actually meant to say something. That's not autocomplete. That's something closer to a colleague catching themselves mid-sentence.

That sent me down a rabbit hole. And the thing I found that made it all click was a line from Geoffrey Hinton: if you want to predict the next word well, you have to understand the sentences.

His argument is simple. Language is compressed knowledge about the world. To predict it accurately, you need to decompress it back into the world that generated it.

And the world's highest-level patterns, the conceptual relationships, the causal reasoning, the strategic logic, are exactly what's richest in training data. That's what humans write about.

Nobody writes down how to count letters in a word. Billions of people write down how to think about strategy, how to diagnose problems, how to weigh competing arguments.

So the model learned the top of the cognitive hierarchy first, because that's where the signal was strongest.

That's why "just autocomplete" is technically correct but practically useless. It's like saying human thinking is just neurons firing: while true at the implementation level, it doesn't help you understand what humans actually do.

And if you're using that as your mental model, you'll treat AI like a fancy search engine.

You'll miss what it can actually do. And you'll be blindsided by failure modes you never saw coming.

The search for a better analogy

So if "just autocomplete" doesn't cut it as a mental model, what does?

People have tried.

Ethan Mollick called AI a "jagged frontier," a map of uneven capabilities across different tasks, peaks here, valleys there.

Andrej Karpathy called it "alien intelligence," shaped by completely different pressures to our own.

Both useful, but both trying to categorise AI as a type of thing. And you can't build a working relationship with a thing. You can't learn when to trust it and when to check behind it.

So people have tried describing AI as various types of person.

"Brilliant intern." "Savant."

And I get why people reach for savant in particular - it seems like the best description we have for why AI is freakishly good at some things, hopeless at others.

But it's not quite right either.

A savant has specific areas of brilliance, and lacks general intelligence. What makes AI strange is that the pattern isn't domain-limited. It's inverted.

The stuff that should be hardest is easiest. The stuff that should be trivial is where it falls apart. That's not a savant. That's something more familiar.

That's someone who operates at the top of the cognitive hierarchy and falls apart at the bottom.

And actually, we've all worked with someone like this.

We already know this person

We have all worked with them:

The visionary creative director who produces the most brilliant campaign you've ever seen but can't book a meeting room.
The born networker who knows everyone, remembers every detail about everyone, yet their email inbox is a disaster zone.
The strategist who writes a memo so sharp it changes the direction of the entire company, but it's got basic factual errors and major typos.

We don't call these people "not intelligent."

We don't refuse to work with them until they can manage their calendar.

We learn their shape.

We figure out what to hand them and what to keep away from them.

We develop a feel for when they're in the zone and when they need someone to check behind them.

And the people who learn to work with that kind of person, rather than trying to force them into a standard performance framework, get disproportionate results.

That's the relationship most of us need to build with AI.

Not categorising it. Not endlessly debating whether it's "really" intelligent. Learning its shape.

What that looks like

I'm no expert on how these models work under the hood. But I've been spending a lot more time working with AI lately than the average person, and a few things have stood out to me.

The turning point was when Anthropic released Claude Opus 4.5.

Before that, I was using AI to make little pieces of the puzzle. A draft here, an analysis there, help fixing a specific coding bug .

With Opus 4.5, I could suddenly build the whole thing at once. I could give it the full picture of what I wanted, and it could visualise the end result and take me there:

Data analysis that would have taken a team days
Complex web builds that would have taken weeks
Research synthesis across thousands of sources.

All done in hours.

And then it would hallucinate a source that doesn't exist. Or slap !important on a CSS rule instead of taking the time to find the underlying formatting conflict. Lazy fixes. The kind of thing you'd pull a junior developer aside for.

Once you stop being frustrated by this and start treating it as useful information, everything changes.

Because the pattern is consistent: trust it more on the hard stuff, check it more on the easy stuff. Which is the exact opposite of what your instincts tell you to do.

You develop a feel for that over time, the same way a creative director develops an eye for good work.

And the gap is widening fast between people who use AI once a week and people who use it all day, every day. They're developing completely different levels of calibration.

Build the right team

Some of the best advice I ever got came from an executive coach.

He'd done a deep psychological profile with me, the kind that maps out your strengths and your blind spots in uncomfortable detail. And his advice wasn't what I expected.

The most common mistake, he said, is that people see their blind spots and immediately try to plug the gaps.

They pour energy into fixing weaknesses, and in the process they sacrifice the strengths that made them effective in the first place.

The better move is to lean into what you're genuinely good at, and then find people whose strengths are your blind spots. Build a team of complementary capabilities rather than trying to become well-rounded on your own.

I think about that advice every time I talk to someone who's tried AI a couple of times, found it unreliable, and quietly moved on.

They're waiting for the perfect colleague.

They're going to be waiting a long time.

The answer isn't to wait for a single AI to be perfect at everything. It's to build a team. In practice, that means a few things.

Make upside-down intelligence work for you

Flip your instincts. Your natural impulse is to trust AI on simple tasks and double-check the complex ones. Do the opposite. Trust it more on the hard stuff. Check it more on the easy stuff. This single mental shift will save you more time than any prompt trick.
Use different models for different jobs. Different AIs have different strength profiles. The best practitioners I know use Codex for one part of the engineering workflow, Claude Code for another. One model for generation, a different one for review. You wouldn't hire one person to do every role. Same logic.
Set up adversarial checks. Sometimes the most effective setup is deliberately pitting one AI against another. A second model whose whole job is to catch the blind spots of the first. It sounds redundant. It's not.
Invest in the harness. The raw model is like the raw person. Talented but inconsistent. The harness is everything you put around them: custom instructions, reference files, structured workflows, guardrails. A few hours of setup upfront will completely transform the quality of what you get back. Most people skip this step and then blame the model.
Give it real feedback. Most people ask a question, take the answer, move on. That's using AI like a search engine. The practitioners who get the best results push back. "That's not quite right, here's why." They give the model context it's missing. They iterate. They treat it like briefing a colleague, not querying a database.
Give it real context. The quality of your output is directly proportional to the quality of your input. If you give the model a vague brief, you'll get a vague answer. Give it the full picture of what you're trying to achieve, who it's for, what good looks like, and the output transforms. The people who complain AI produces generic work are usually giving it generic briefs.
Develop your taste. This is the human part that doesn't get automated. The ability to look at AI output and feel whether it's good or not. Whether it's understood the brief or just produced something plausible. That's a judgment you build over time, the same way a creative director develops an eye. There is no shortcut. You have to put in the reps.

The bottom line

The AGI debate will keep going. People will keep arguing about whether AI has crossed some threshold of general intelligence.

But for anyone actually using these tools every day, the more useful insight is simpler.

AI didn't learn intelligence the way we did. It learned the top of the cognitive hierarchy first and the bottom last.

It's brilliant where you'd expect it to struggle and unreliable where you'd expect it to cruise.

Once you internalise that, you stop asking "is it intelligent?" and start asking the same question you'd ask about any brilliant, frustrating colleague:

What are they actually good at?
How do I get the best out of them?
What do I need to catch before it goes out the door?

You wouldn't ask the visionary creative director to proofread their own deck.

You wouldn't expect the born networker to keep their own CRM up to date.

You'd build a team around them.

Someone to catch the typos, manage the details, handle the stuff that's beneath their operating level but still needs to get done.

That's the move.

Not waiting for a perfect AI. Building a system of complementary strengths, human and machine, that covers the blind spots without sacrificing what makes each part brilliant.

The people who figure that out first will have an extraordinary advantage.

Not because they understand the theory better. Because they've learned the shape of the people they're working with, and built a team accordingly.