GGUF Quantization with Imatrix and Okay-Quantization to Run LLMs on Your CPU

Quick and correct GGUF fashions on your CPU

GGUF is a binary file format designed for environment friendly storage and quick massive language mannequin (LLM) loading with GGML, a C-based tensor library for machine studying.

GGUF encapsulates all needed parts for inference, together with the tokenizer and code, inside a single file. It helps the conversion of varied language fashions, resembling Llama 3, Phi, and Qwen2. Moreover, it facilitates mannequin quantization to decrease precisions to enhance velocity and reminiscence effectivity on CPUs.

We frequently write “GGUF quantization” however GGUF itself is simply a file format, not a quantization methodology. There are a number of quantization algorithms carried out in llama.cpp to scale back the mannequin measurement and serialize the ensuing mannequin within the GGUF format.

On this article, we are going to see find out how to precisely quantize an LLM and convert it to GGUF, utilizing an significance matrix (imatrix) and the Okay-Quantization methodology. I present the GGUF conversion code for Gemma 2 Instruct, utilizing an imatrix. It really works the identical with different fashions supported by llama.cpp: Qwen2, Llama 3, Phi-3, and so forth. We may even see find out how to consider the accuracy of the quantization and inference throughput of the ensuing fashions.