What is a foundation model in AI?

Where the term comes from

Researchers at Stanford University introduced the term in an August 2021 paper. They described models trained on broad data at scale that can be adapted to a wide range of downstream tasks. Their examples included BERT, DALL-E and GPT-3. The authors said the name was meant to stress that such models are central yet incomplete.

The paper also flagged a risk. When many applications are adapted from the same model, they all inherit its flaws.

How it relates to other terms

A foundation model is often a large language model, an AI system trained on text. The two terms are not identical. A 2023 explainer from Georgetown University's Center for Security and Emerging Technology (CSET) says the terms are often used roughly as synonyms. It says "foundation model" is the broader idea, defined by what a model does. It can include systems that handle more than language. CSET adds that none of these labels has a sharp boundary.

The UK's Competition and Markets Authority described foundation models as large machine learning models trained on vast amounts of data. In a September 2023 summary of its review, it said OpenAI released the first public one in 2018. It estimated that around 160 had been developed and released since then.

The European Union's AI Act uses a related legal term instead. Article 3 defines a general-purpose AI model as one that shows significant generality and can be built into many downstream systems.

Why some countries build their own

Germany's SOOFI project shows why governments care. Its name stands for Sovereign Open Source Foundation Models. Fraunhofer IIS, one of its members, says the federal economy ministry funds it with about 20 million euros. It says the aim is to give companies and public bodies independent alternatives to non-European models.

The project's first model, Soofi S, is a base model. Fraunhofer explains that a base model has been pre-trained and serves as a starting point for custom versions. Users can then build their own models for a specific field or task.