What's in a GGUF, besides the weights – and what's still missing?
GGUF (GPT-Generated Unified Format) is a binary format designed to package quantized large language model weights along with minimal metadata. However, as the article details, critical information such as tokenizer type, vocabulary size, model architecture details (e.g., number of layers, attention heads), and inference parameters (e.g., context length, rope scaling) are often missing or inconsistently stored. This forces developers to manually specify these settings when loading models, leading to compatibility issues between different inference engines like llama.cpp, Ollama, and LM Studio. The lack of standardized metadata also complicates model sharing and reproducibility, as users must rely on external documentation or trial-and-error to run a model correctly. The author proposes extending GGUF with a more comprehensive metadata schema, including fields for tokenizer configuration, model architecture, and inference defaults, to improve interoperability and ease of use.
Missing metadata in GGUF forces developers to manually configure models, reducing portability and reproducibility.