ggml-org/llama.cpp
- Stars
- 134,525
- Language
- C++
- License
- MIT
- Daily
- rank 13 · +287 stars gained
Repository metrics were observed on this edition date and may have changed.
Key Features Runs LLM and VLM inference in C/C++ and provides a CLI, an OpenAI-compatible API server, and a built-in web UI. Notable features include support for various quantization formats, mixed CPU/GPU inference, and multiple backends such as Metal, CUDA, HIP, and Vulkan. Use Cases The README presents workflows for downloading a GGUF model from Hugging Face and running it through the CLI, or launching an OpenAI-compatible server with the same model. The server also includes a built-in web UI for browser use. Costs and Limitations The code is licensed under MIT, and a local inference option is available. Each model's terms of use and cost conditions need to be confirmed. Self-hosting incurs hardware, storage, and electricity costs, while cloud execution incurs separate infrastructure costs. Adoption Considerations Check the binary or build configuration appropriate to the selected model format, size, and execution backend. The supplied information alone cannot establish whether a particular device has sufficient memory or what processing performance it will deliver. Why It Stands Out It was observed at No. 13 on daily Trending, with a star increase of 287. Build b12332, listed on 2026-11-12, connects operations previously limited to the scalar CPU path to SpacemiT RVV kernels, illustrating continued refinement of backend-specific implementations.
View on GitHub