Pretraining
Pretraining is the first stage of training a language model, in which the model learns from a large amount of unlabeled text.
On this page
Source content checked . English translation published 2026-10-27.
In business automation
When reading about language models, you will encounter several different training stages. Pretraining is the first stage of training a language model, in which the model learns from a large amount of unlabeled text. According to Anthropic, a model that has only undergone pretraining is not yet suited to serve as an assistant, so further stages follow, such as fine-tuning and RLHF.
Examples
These examples illustrate a possible workflow. Adapt the connections and rules to your own systems.
Why a base model does not respond like an assistant
A developer tries a base model that has only undergone pretraining and gets a text continuation instead of an answer to a question. They switch to a version that has also been fine-tuned for conversation, which is suited to a service bot.
Relevant tools
- Anthropic
- This definition is from Anthropic's glossary.
What to check
- Assuming that pretraining teaches the model your business's information. RAG or fine-tuning is generally used for that.