Quantization
Quantization reduces the computation and memory cost of running a model, by representing the weights and activations in lower precision data types, for example int8 instead of float32. The common cases are moving from float32 to float16 and from float32 to int8. This makes it possible to run open models on a small server or an edge device, but the drop in precision may harm quality, so it should be tested on real data.
On this page
Source content checked . English translation published 2026-10-27.
In business automation
Running a model requires a lot of computation and memory. Quantization reduces this cost by representing the weights and activations in lower precision data types, for example int8 instead of float32. The common cases are moving from float32 to float16 and from float32 to int8, and the price is a possible drop in precision.
Examples
These examples illustrate a possible workflow. Adapt the connections and rules to your own systems.
An open model on a small server
A team wants to run an open model on a small server that does not have enough memory for the original version. It runs a quantized version of the model, which requires less memory and computation, and checks whether the answers are still good enough for the task.
Testing on real data
Before replacing a model with a quantized one in a real process, both are run on the same set of examples from the business and the answers are compared. A drop in precision may harm quality, so the comparison is made on real data and not on a feeling.
Relevant tools
- Hugging Face
- The definition comes from the Hugging Face Optimum quantization guide.
- Ollama
- Open models shrunk by quantization are the way to run them on a small server or an edge device.
What to check
- Assuming the quantized version is as good as the original without testing. The drop in precision may harm quality.
- Testing only on general examples and not on real data from the process in which the model will be used.