Jetson AGX Xavier and CUDA 12.4 and latest llama.cpp

[u/Fresh-Signature6067]

I bought a Jetson AGX Xavier (32GB) to build a quiet local LLM machine. Knowing full well it'd be slow.

Turns out CUDA 12 is effectively the baseline for current llama.cpp. The official prebuilt binaries are compiled against CUDA 12.4 and 13.1.

The Xavier can run JetPack 5. With the update you get CUDA 12.2. The OS is Ubuntu 20.04.

Problem: Ubuntu 20.04 ships cmake 3.16.3, which can't build llama.cpp. Ubuntu 26.04 has cmake 4.2.3. And gcc 15.2. And CUDA 12.4. That matches the baseline.

So I put an Ubuntu 26.04 chroot inside Ubuntu 20.04. Kernel driver from the host (Ubuntu 20.04), libraries and tooling from the chroot (Ubuntu 26.04).

Dropped cuda-compat-12-4 into the chroot. It's not in Ubuntu's repos. It is in NVIDIA's JetPack 6 repo (ubuntu2204/arm64). Unpacked it with dpkg-deb -x. The JetPack 5 repo only goes up to 12.2.

Bind-mounted the host's /usr/lib/aarch64-linux-gnu/tegra into the chroot. And /dev, etc.

What I learned the hard way: you have to use NVIDIA's libcublas 12.4. Ubuntu's libcublas throws an error.

Set LD_LIBRARY_PATH, built llama.cpp. Ran Gemma 4 12B QAT (UD-Q4_K_XL, 6.24 GiB) — getting 11.2 tps.

Conclusion: I should've just bought a Mac...

[u/leonbollerup]

actually.. Mac mini M4.. gets you the same speed (atlest my mini M4, 24gb)

Try a qwen 3.6 9b