In my “LoRA Is All You Need” post, I argued that modern artificial intelligence has drifted away from our traditional notion of intelligence. Here, I want to develop that idea further and ask whether the path LLMs are taking is actually the right path toward real intelligence—or what we often call AGI (Artificial General Intelligence).
What does intelligence even mean? How should we measure it? When would we know we’ve reached AGI? Is an image generator intelligent? A music model? Even a large language model that writes coherent paragraphs—does that qualify?
Back in 11th grade, when I gave a presentation on AI, I used a simple definition borrowed from Max Tegmark’s book Life 3.0: intelligence is the ability to accomplish tasks autonomously. It wasn’t academic, but it captured something intuitive: the independence. Using that definition, on for example image and music generators that are highly capable tools, but narrow ones. They solve specific tasks and barley on their own. So I wouldn’t consider them as intelligent, rather than advanced algorithms, even when they understand context and their underlying principles resemble LLMs like the transformer architecture , it feels wrong to call them truly intelligent.
LLMs are different. When they reason — or at least appear to reason — when they use agents to extend their capabilities, it can look as they are truly accomplishing tasks on their own. This leads to the tempting idea that if we simply scale them up, they’ll continue getting smarter until eventually they can perform any task a human can, or even surpass us. In other words: AGI.
This is also where the idea of technological singularity comes in — the moment when models become able to improve themselves, triggering exponential progress and transforming human history forever. But I believe the road to AGI is far less straightforward. Two critical obstacles stand out: the ungrounded nature of LLMs and the scalability limits of human feedback.
Before diving into those, I want to refine the definition of intelligence I used years ago. A more complete version would be:
Intelligence is the ability to construct, update, and use internal models of the world to plan, reason, and achieve goals in novel situations.
If we evaluate today’s token-based systems against this definition, the gaps are obvious. LLMs do not maintain persistent world models. They cannot reliably simulate counterfactuals. They often wander through reasoning, contradict themselves, or fall into pattern-based dead ends. Their “planning” is just the description of a plan — not the internal execution of one.
To me, AGI would be a system that can learn and generalize like a human, starting with minimal sensory input and without being preloaded with task-specific knowledge.
Modern LLMs are not built this way. They are trained on massive datasets using unsupervised learning to predict the next word. Then they are fine-tuned and aligned via human feedback, and once deployed, their weights are frozen and stay so. This leaves them unable to continuously learn or form lasting world models. They don’t understand the world — they imitate the linguistic patterns of beings who do understand it. Even “reasoning models” are ultimately generating more structured token sequences, not performing internal reasoning. Agents may look more autonomous because they can use tools and extend their context, but they inherit the same limitations.
Their generalization is emergent, not principled. They can solve many tasks, but struggle on others — as shown by the ARC-AGI benchmark, which tests simple abstract transformations that humans grasp with ease but current models still struggle to internalize. This is a signal: real intelligence requires rule formation, internal modeling, and robust adaptation to novelty.
This brings us to the first major barrier: LLMs are fundamentally ungrounded. They lack connection to physical, contextual, or sensory reality. Their “understanding” is abstract, detached from the world. This makes them brittle: they struggle with ambiguous inputs, contradictions, or situations that differ from the patterns in their training data.
The second barrier is that human feedback does not scale. Teaching new concepts via labeled data demands enormous time, cost, and effort, yet yields only incremental gains. RLHF and similar techniques are great at refining behavior but poor at enabling fundamentally new capabilities. Relying heavily on human feedback risks stagnation. Reaching human-level generalization will require new architectures or paradigms that learn from fewer examples and can infer abstract principles.
Scaling LLMs and improving alignment will continue to matter, but overcoming these barriers will require foundational innovation: – integrating multimodal sensory inputs or physical interaction so models gain grounded understanding – developing self-supervised or synthetic data methods to reduce dependence on humans – building architectures that learn from sparse examples and adapt autonomously
These challenges — grounding and scalability — aren’t just engineering problems. They are foundational obstacles to building truly intelligent systems.
Some research directions point toward what might come next. SimuRA, for example, is built around simulative reasoning: instead of treating reasoning as a token sequence, it constructs a latent internal “world” where the agent can imagine outcomes before acting. Other work takes world modeling literally. DeepMind’s Genie 3 can generate entire interactive environments from a single image and step them forward in time, functioning almost like a compact physics engine. It isn’t human understanding, but it gestures toward systems capable of internal simulation of world models — capabilities necessary for robust intelligence.
Yet even with these advances, benchmarks like ARC-AGI keep reminding us of the current limits. Scale alone won’t solve the problem.
Real intelligence won’t emerge from ever-bigger next-token predictors. It will come from building systems with an internal sense of reality — machines that simulate, adapt, and make sense of novelty. Only then will AI cross the boundary from sophisticated mimicry into something that genuinely deserves the name AGI.