llama.cpp Windows Setup
Windows Setup Experience for llama.cpp
This documentation details the process of installing llama.cpp on a Windows environment, operating the llama-cli, and utilizing the llama-bench for performance assessment.
The text covers pre-requisites, installation steps and examples of command-line interaction with models. It also covers methods for using the benchmarking tool to evaluate system speed and efficiency.
This resource is intended for developers seeking to llama.cpp on Windows systems.
Llama.cpp
llama.cpp is an open-source C/C++ library that allows you to run large language models (LLMs) like Llama-3, qwen and other similar models consumer-grade hardware. It’s designed to be efficient, portable, and easy to integrate into different projects.
Environment
- Dell Precision 5680
- CPU 13th Generation Intel Core i7-13800H vPro Enterprise, 14 Cores, 20 Threads
Get-CimInstance Win32_Processor | Select-Object -Property NumberOfCores, NumberOfLogicalProcessors
- GPU Embedded RTX3500 Ada dGPU
- Windows 11
Pre-requirement
Installation Guide
Clone the llama.cpp repository from GitHub
1
git clone https://github.com/ggerganov/llama.cpp
Generate build file
1
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler"
Note: If CMake fails to find your CUDA installation (often indicated by an error saying CMake cannot find CUDA 13.x.props or similar), you may need to manually copy the integration files from your CUDA installation to the Visual Studio MSBuild directory.
Manual CUDA-VS Integration Fix (CUDA v13.1 Specific)
If you encounter an error related to missing CUDA build customizations in MSBuild (happened to me at CUDA 13.1), perform the following manual copy:
Locate CUDA MSBuild Extensions: Navigate to your CUDA Toolkit installation. Example for CUDA:
- C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\extras\visual_studio_integration\MSBuildExtensions*
- Copied CUDA files in this folder.
- Locate Visual Studio Build Customizations: Navigate to your Visual Studio installation. Example for VS 2026 Community:
- C:\Program Files\Microsoft Visual Studio\18\Community\MSBuild\Microsoft\VC\v180\BuildCustomizations\
- Paste the copied CUDA files into this folder.
- Note: The 18 (Visual Studio major version) and v180 (internal MSBuild version) in the example path above will change according to your specific Visual Studio IDE version. Ensure you are copying to the path corresponding to the Visual Studio installation you intend to use.
- Retry the Configuration: Return to your llama.cpp folder and rerun the cmake configuration command above.
Build llama.cpp
Once configuration is successful, compile the project in Release mode using all available CPU threads for a faster build:
1
cmake --build build --config Release -j
llama-cli
Once compilation is complete, you can use the llama-cli to interact with a GGUF model.
1
./build/bin/release/llama-cli.exe -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -ngl 999
User prompt is read to themodel and generate output. TTFT (Time to first token) and TPOT (time per token output is generated)
llama-bench
The llama-bench tool is used to measure the performance of your system with specific models and workloads. This is essential for determining prompt processing and token generation speeds.
1
./build/bin/release/llama-bench -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -p 1024 -n 128 -ngl 999
- Model Path: -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf
- Meta-Llama-3, 8B parameters, Quantization 4bits, gguf format
- Input length: 1024 tokens
- Output length: 128 tokens
- GPU offload: -ngl 999 all layers offload to GPU
Performance Statistics
There are two key metrics essential for an LLM system:
- Prompt Processing (TTFT - Time To First Token): Measures the initial responsiveness—how long it takes for the model to “think” about your prompt and start writing its answer.
- Token Generation (t/s - Tokens Per Second): Measures the average typing speed of the model after it has started generating.