World model and Maze World simulator. The upper half shows the ground truth, and the lower half shows the model’s prediction.
What is general-purpose in-context learning, and why emphasize it?
We believe that the core capabilities behind emergent intelligence in large models include two aspects:
- Zero-shot generalization: A model stores the data and knowledge acquired during training in its parameter space, then uses that knowledge to reason, generalize, and solve problems.
- Few-shot in-context learning: A model goes further and uses information in a prompt to acquire data patterns or skills that were entirely absent from training.
In-context learning usually has several valuable properties. It can learn a new skill from very few examples, far fewer than gradient descent typically requires. It is relatively insensitive to the form of supervision: often, positive examples are unnecessary, and prompt feedback that merely explains what is wrong can still support learning. It can therefore encompass what are traditionally called supervised, reinforcement, and unsupervised learning. It is also robust to noise and can execute a task correctly even when the prompt contains ambiguous information. Together, these properties give a model the potential to generalize across an extremely broad range of tasks.
Current in-context learning, however, has clear limitations. It usually operates on only a small number of examples and is often confined to “instruction following.” It cannot carry out complex processes that require repeated attempts and optimization inside the context, as reinforcement learning does. Moreover, although the context lengths supported by recent large models continue to grow, much of this context processing is not genuine “learning.” It often consists of simple copying, retrieval, or recall of particular content.
The definition of general-purpose in-context learning and how it differs from other learning methods.
We argue that achieving genuine artificial general intelligence requires extending in-context learning into General-Purpose In-Context Learning (GPICL). Compared with other methods, GPICL has three defining characteristics:
- Minimal inductive bias and low zero-shot generalization.
- A long in-context learning horizon.
- High in-context learning potential.
Improving the upper bound and horizon of in-context learning has become a broadly shared goal. Deliberately reducing zero-shot capability, however, sounds counterintuitive. Why would this help? We believe zero-shot generalization is inversely related to data diversity. Stronger zero-shot generalization can indicate that the samples and tasks used to train a model were insufficiently diverse, allowing the model to retain too much knowledge tied to particular tasks or a limited task family. That retained knowledge can then restrict its ability to generalize beyond the training task distribution.
The right task set is therefore crucial for teaching AI a general-purpose in-context learning capability. The objective of lower zero-shot generalization does not mean preventing a model from learning anything from a fixed training set. It means that we should use a training set so diverse that the model cannot form any useful prior assumptions about the tasks themselves.
Requirements for a general-purpose in-context learning benchmark
For these reasons, building training and evaluation task sets suitable for GPICL is essential. As its name suggests, general-purpose in-context learning should generalize to a broader collection of unseen tasks, potentially across different modalities, without requiring those tasks or modalities to appear during pretraining or meta-training. More specifically, GPICL requires the following:
- A sufficiently large number and diversity of tasks. The task collection must be so broad that no model can extract useful zero-shot knowledge from it, encouraging minimal inductive bias. If a task set contains many tasks but too little diversity, a model can easily develop strong zero-shot generalization, leaving little need or potential for in-context learning and potentially weakening generalization beyond the task family. If diversity is high but the number of tasks is too small, the model can fall into the trap of “multitask learning plus task identification.” It memorizes how every task works during pretraining, then needs only a small amount of context at inference time to identify the task and retrieve the memorized solution. Neither case elicits genuine GPICL.
- Lifelong in-context learning. A model should require a truly long context to master a task: not two or three examples, nor twenty or thirty, but potentially millions or even billions of contextual observations. Here, context length means the minimum context required under an ideal learning method.
- Continual generation and interaction. Context should consist of an ongoing stream of examples produced through real-time generation and interaction.
A meta-language generator
We believe the process of learning a new language fits the needs of GPICL better than learning any particular natural language. Yet only a few hundred languages are commonly used around the world, far too few if each natural language is treated as one training example. An ideal alternative is to create an almost unlimited number of “languages.” Each language can be viewed as one extremely long sequence, and a model can learn a new language from its context.
We propose a way to “create” such languages: use networks with random parameters to generate random sequences. A short segment of one of these sequences has no apparent meaning, but as the sequence grows, an external learner can begin to capture the underlying rule that generates it. This unlimited supply of “new language” sequences provides an ideal testbed for GPICL.
We find that Transformer models can capture these random sequences and develop in-context learning. We further show that the resulting capability is general-purpose. The model can learn a new random sequence from context, but when the same model, pretrained only on random sequences, is applied to real natural-language tasks, it still demonstrates an ability to learn. Importantly, the model never sees any natural-language sequence during pretraining.
Evaluation of a meta-language model on meta-languages of different complexity. PG-19, shown last, is a natural-language benchmark. Performance improves as context length increases, even on natural language absent from training, demonstrating the broad applicability of meta-language training. Increasing the scale of a model with tens of millions of parameters provides no further benefit, showing that GPICL does not always require an enormous parameter count.
A meta-language model can use context to learn completely unfamiliar data. Here, the model learns English words and captures their patterns.
A meta-language model can use context to learn completely unfamiliar data. Here, the model learns patterns in mathematical equations.
Generating and simulating Maze World
Maze World is a lightweight simulation environment with clear rules and substantial diversity. We can easily vary the size, structure, and walls of each maze. The environment contains two task families: navigation and survival. In a navigation task, for example, we randomly place a number of distinct “landmarks” at different locations in a randomly generated maze. The environment gives the agent a random instruction asking it to reach a landmark of a particular color as quickly as possible. The task requires exploration, memory, localization, and path planning. Because both the maze and landmarks are entirely random, the agent has no source of prior information about any task beyond its own exploration.
Reinforcement learning is often difficult to scale, so in addition to the environment itself, we provide “oracle agents” that support imitation learning and rapid warm starts. These oracle agents can access a global map, although a location must first be observed before it can be recorded. They generate large quantities of high-quality data that a model can imitate directly.
Our experiments show that Transformer-based in-context learning can simultaneously learn a world model and a policy model. To our knowledge, this is the first work to observe in-context learning of a world model. The results also show that current models remain far from the ideal represented by a strong oracle agent, leaving substantial room for improvement.
The Maze World environment and the map memory used by the oracle agent.
A maze-navigation model that integrates a world model and a policy model.
During evaluation, both the world model and policy model adapt as the available context grows.
An in-context world model becomes increasingly accurate at future prediction, case 1. The upper row shows real observations. The lower row shows the world model’s imagined prediction for the next nine time steps under the same action sequence. The variable t denotes the context length available to the model.
An in-context world model becomes increasingly accurate at future prediction, case 2.
An in-context world model becomes increasingly accurate at future prediction, case 3.
More parameters, or more memory and context?
In both environments, our current implementation uses in-context learning horizons of only 2K to 4K steps. A configuration change, however, can raise the minimum learning horizon to millions or even billions of steps. We believe these training and evaluation sets are valuable for both long-context learning and general-purpose in-context learning. Although the two environments are relatively simple and lightweight, they address important research goals. Meta-language trains a general agent that can learn and adapt to languages. Maze World develops a general embodied agent that combines exploration, localization, and navigation in an unknown environment.
Our work also confirms a finding from many earlier studies: in-context learning capability has only a weak relationship with model parameter count. After the parameter count exceeds a certain threshold, increasing it no longer improves in-context learning. Capability is instead strongly and positively related to the scale of context and memory state. This points to a path different from brute-force parameter scaling: sufficiently many parameters, combined with extensive memory and contextual knowledge.
Based on this insight, we propose a training method distinct from those used for current large language models. First, use massive synthetic datasets to train a model’s general-purpose in-context learning capability. Then use a smaller amount of high-quality data to align the model with human preferences. This process more closely resembles the emergence of natural intelligence: the first stage corresponds to billions of years of biological evolution, while the second corresponds to the lifetime of an individual human.
A large-model training pipeline centered on general-purpose in-context learning.