Installation¶

This guide covers installing cyllama on different platforms.

Requirements¶

Python 3.12 or later

From v0.3.0, cyllama ships only abi3 (CPython stable ABI) wheels, which require Python 3.12+. For Python 3.10 / 3.11, install an older release (pip install "cyllama<0.3.0") or build from source.

C++ compiler (clang or gcc)
CMake 3.21+
Git

Platform-Specific Requirements¶

macOS:

xcode-select --install  # Xcode Command Line Tools

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install -y build-essential cmake git python3-dev

Fedora/RHEL:

sudo dnf install -y gcc-c++ cmake git python3-devel

Install from PyPI¶

pip install cyllama

GPU-Accelerated Variants¶

GPU variants are available on PyPI as separate packages (dynamically linked, Linux x86_64 only):

pip install cyllama-cuda12   # NVIDIA GPU (CUDA 12.4)
pip install cyllama-rocm     # AMD GPU (ROCm 6.3, requires glibc >= 2.35)
pip install cyllama-sycl     # Intel GPU (oneAPI SYCL 2025.3)
pip install cyllama-vulkan   # Cross-platform GPU (Vulkan)

All GPU variants install the same cyllama Python package -- only the compiled backend differs. Install one at a time (they replace each other). GPU variants require the corresponding driver/runtime installed on your system.

`cyllama-sycl` host prerequisites¶

The SYCL wheel deliberately does not vendor the Intel oneAPI runtime (libsycl, MKL, TBB, libiomp5, the Intel compiler runtimes). The full vendored stack exceeds PyPI's 100 MB per-file cap, and the GPU-coupled pieces must match the host driver anyway. The host therefore has to supply two things, and they live in separate registers.

1. oneAPI userspace runtimes (required for import cyllama to succeed).

These provide the libraries the wheel's DT_NEEDED entries point at -- libsycl.so.8, libmkl_*.so*, libtbb.so.12, libiomp5.so, libsvml.so, libimf.so, libintlc.so.5, libirng.so. On Debian/Ubuntu after adding the Intel oneAPI APT repo:

sudo apt install \
  intel-oneapi-compiler-dpcpp-cpp-runtime-2025.3 \
  intel-oneapi-openmp-2025.3 \
  intel-oneapi-mkl-core-2025.3 \
  intel-oneapi-mkl-sycl-blas-2025.3
source /opt/intel/oneapi/setvars.sh
python -c "import cyllama"   # should succeed

Keep the -2025.3 suffix. The wheel is built against oneAPI 2025.3 and links libsycl.so.8. The unversioned intel-oneapi-runtime-* packages now install 2026.x, which ships libsycl.so.9 only. The GPU wheel CI smoke test installs this same package set.

For RPM-based distros, use Intel's DNF/Yum repo with the same package names. An existing Intel oneAPI base toolkit install works if it is 2025.x; intel-oneapi-base-toolkit-2025.3 pins it. Source its setvars.sh. Without these libraries on the loader path, import fails with libsycl.so.8: cannot open shared object file (or a similar message naming one of the other runtimes).

2. An Intel GPU visible to SYCL.

ggml's SYCL backend requires a GPU device. It has no CPU fallback. Install the driver for one of:

Intel GPU via OpenCL: install the Intel compute-runtime package providing libOpenCL.so.1 and the Intel GPU ICD (intel-opencl-icd on recent Ubuntu, or the upstream intel-compute-runtime packages). Follow Intel's GPU driver install guide for your distro and GPU family (Arc, Iris Xe, Data Center GPU Max/Flex).
Intel GPU via Level Zero: install level-zero and intel-level-zero-gpu. Same install guide.

Package names and recommended install paths drift across distro versions and Intel releases, so we link to Intel's authoritative install pages rather than hard-coding an apt install line we can't keep current. Without a GPU, import cyllama succeeds, but registering backends aborts the process with can not find preferred GPU platform. import cyllama.sd, cyllama info and loading a model all register backends.

You can verify which backend is active after installation:

cyllama info

Build from Source¶

# Clone repository
git clone https://github.com/shakfu/cyllama.git
cd cyllama

# Build everything (downloads llama.cpp, whisper.cpp, builds cyllama)
make

# Download a test model
make download

# Verify installation
python -c "from cyllama import complete; print('OK')"

Build Options¶

Default Build¶

The default build enables GPU acceleration appropriate for your platform:

macOS: Metal (Apple GPU)
Linux: CPU-only (GPU backends optional)

make build

GPU Backends¶

Build with specific GPU support (static or dynamic):

# Static builds (all libs compiled into the extension)
make build-cpu       # CPU only
make build-cuda      # NVIDIA CUDA
make build-vulkan    # Vulkan (cross-platform)

# Dynamic builds (shared libs installed alongside extension)
make build-cpu-dynamic
make build-cuda-dynamic
make build-vulkan-dynamic

# Multiple backends
GGML_CUDA=1 GGML_VULKAN=1 make build

See Building with Different Backends for detailed GPU setup instructions.

Optional Components¶

Stable Diffusion support:

WITH_STABLEDIFFUSION=1 make build

# SD shares llama.cpp's ggml by default; opt into SD's vendored copy instead
SD_USE_VENDORED_GGML=1 make build

SD_USE_VENDORED_GGML is the only ggml switch for SD. See Stable Diffusion ggml selection.

Whisper support (included by default):

make build  # Whisper is built automatically

Build System¶

Cyllama uses scikit-build-core with CMake for building the Cython extensions. The build process:

Dependencies: make downloads and builds llama.cpp, whisper.cpp (and optionally stable-diffusion.cpp)
Cython compilation: CMake compiles .pyx files to C++ using Cython
Extension linking: C++ extensions are linked against the static libraries
Installation: Extensions are installed in editable mode

Build Commands¶

Command	Description
`make`	Full build (dependencies + editable install)
`make build-<backend>`	Static build for a specific backend (e.g. `build-cuda`)
`make build-<backend>-dynamic`	Dynamic build for a specific backend (e.g. `build-cuda-dynamic`)
`make wheel`	Build wheel for distribution
`make wheel-<backend>`	Static wheel for a specific backend
`make wheel-<backend>-dynamic`	Dynamic wheel for a specific backend
`make clean`	Remove build artifacts and dynamic libs
`make reset`	Full reset including thirdparty and .venv
`make remake`	Clean rebuild with tests
`make leaks`	RSS-growth memory leak detection

Wheel Distribution¶

To build a distributable wheel:

make wheel
# Creates wheel in dist/

The wheel includes all compiled extensions and can be installed on systems with matching platform/Python version.

Installing Models¶

LLM Models (GGUF format)¶

Download the default test model:

make download
# Downloads: models/Llama-3.2-1B-Instruct-Q8_0.gguf

Or download manually from Hugging Face:

# Example: Download a model
curl -L -o models/llama.gguf \
  "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q8_0.gguf"

Whisper Models¶

Download from ggerganov/whisper.cpp:

curl -L -o models/ggml-base.en.bin \
  "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin"

Stable Diffusion Models¶

Download SDXL Turbo or other SD models in GGUF or safetensors format.

Verification¶

Test Installation¶

# Run test suite
make test

# Quick smoke test
python -c "
from cyllama import complete
print(complete('Hello', model_path='models/Llama-3.2-1B-Instruct-Q8_0.gguf', max_tokens=10))
"

Check GPU Support¶

from cyllama.llama.llama_cpp import ggml_backend_load_all

# Load all available backends
ggml_backend_load_all()

# Check what's available
from cyllama.llama.llama_cpp import LlamaModel, LlamaModelParams
params = LlamaModelParams()
params.n_gpu_layers = -1  # Offload all layers to GPU
# If GPU is available, layers will be offloaded

Troubleshooting¶

"No module named 'cyllama'"¶

Make sure you're in the project directory or have installed cyllama:

cd cyllama
make  # or: uv pip install -e .

Build Errors¶

Clean and rebuild:

make reset  # Full clean
make build

Metal Not Working (macOS)¶

Ensure Xcode Command Line Tools are installed:

xcode-select --install

CUDA Not Found (Linux)¶

Add CUDA to your PATH:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Development Install¶

For development with editable install:

git clone https://github.com/shakfu/cyllama.git
cd cyllama
make  # Builds dependencies and installs in editable mode

For manual editable install (after dependencies are built):

uv pip install -e .

Next Steps¶

User Guide - Learn the API
Cookbook - Common patterns and recipes
Building with Different Backends - GPU setup details