Artificial Intelligence has evolved rapidly, reshaping industries and redefining how we interact with technology. Central to this evolution are Foundation Models — advanced AI systems trained on vast datasets, capable of adapting to numerous tasks without extensive retraining. In this article we dive into the heart of Foundation Models, examining their current limitations, future potentials, and the transformative concept of Software 3.0.

Companion deck — covers the same material as the sections below. Use the arrow keys to navigate, or expand to full screen.

The Problem Landscape

Humans vs LLMs: The Sample Efficiency Gap

Humans learn incredibly efficiently. A four-year-old child processes roughly 1 petabyte ($10^{15}$ bytes) of visual information — about fifty times more than GPT-4 has ever seen — yet generalises effortlessly and robustly. Current AI models, by contrast, exhibit significant inefficiencies, requiring massive amounts of data and compute for every new capability they acquire.

Data Availability Ceiling

We are quickly approaching a ceiling of available high-quality text data, predicted to be exhausted within this decade. This limitation pushes research toward multimodal models — systems that integrate diverse data sources (images, audio, sensor streams) into a shared representation, promising richer learning signal per token.

Autoregressive Limits: Error Cascades

Traditional autoregressive models predict sequentially, token by token, which lets small errors compound exponentially over long horizons. Yann LeCun has argued repeatedly that this approach is inherently limited for AGI, and that future models must integrate predictive world-modeling — simulating outcomes before committing to actions.

Predictive World Models: The New Standard

Future AI must adopt internal predictive world models, continuously simulating outcomes before acting. This enables strategic planning, counterfactual reasoning, and the ability to imagine before deciding — moving beyond reactive prediction to deliberate cognition.

Variable Computation: Adaptive “Thinking Time”

Today’s one-size-fits-all inference is inefficient: easy prompts and hard prompts get the same compute. New models will employ adaptive computation, allocating effort dynamically — like humans spending more time on difficult problems and dispatching trivial ones quickly.

Memory & Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation enhances reliability by dynamically integrating external knowledge into the model’s responses, sharply reducing hallucinations and keeping answers fresh as the underlying corpus evolves. RAG is fast becoming the standard production deployment pattern for LLMs.

The Power of Tokenization

Tokenisation is fundamental: it converts raw data into the standardised discrete symbols an AI can manipulate. Effective tokenisation is what enables multimodal fusion — text, images, audio and sensor data sharing a common vocabulary — and is one of the quietly load-bearing pieces of the whole foundation-model stack.

Software 3.0: The Evolution of Development

Andrej Karpathy’s framing is the cleanest summary of where we are:

• Software 1.0 — developers explicitly write algorithms.
• Software 2.0 — developers curate data and train models; weights replace hand-coded logic.
• Software 3.0 — large language models are general-purpose platforms, programmed via natural language.

In the 3.0 paradigm, development cycles accelerate dramatically through rapid iteration driven by conversational prompts and AI-driven code refactoring. Developers evolve into orchestrators, shaping and guiding intelligent models rather than writing detailed procedural code.

LLM as Operating System

Foundation models are becoming akin to operating systems, providing memory management, scheduling, and API access for entire application stacks. Applications themselves become thinner, leaning on these AI backbones. This paradigm shift raises new challenges:

• Reliability — preventing hallucinations through structured grounding and sandboxed execution.
• Security & Privacy — protecting against prompt injection and data leakage.
• Governance & Compliance — establishing audit trails, licensing, and attribution.
• Sustainability — reducing computational cost and environmental impact.
• Ethical Alignment — ensuring models produce safe, unbiased, ethically sound outputs.

The Hopeful Future

Foundation models promise immense opportunities for collaboration between humans and AI — new roles, enhanced creativity, multiplied productivity. But responsible, ethical stewardship is what determines whether these tools become a positive force at scale.

Foundation models and Software 3.0 together represent an exciting AI frontier, blending technological innovation with human ingenuity to shape a future where technology complements and enhances what people can do.

References & Further Reading

1. R. Bommasani et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM. arXiv:2108.07258
2. OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774
3. A. Karpathy. Software 2.0. Medium (2017).
4. Y. LeCun (2022). A Path Towards Autonomous Machine Intelligence. OpenReview
5. 3Blue1Brown. Neural Networks series (YouTube).