Overview & Backend Selection
InterGenOS runs local AI inference on the hardware in front of it. The local assistant, InterGen, detects the host’s RAM and GPU at startup and selects an appropriately sized model, then serves it through a llama-server subprocess from llama.cpp over a local HTTP API. The compute backend that subprocess uses to talk to your GPU is the subject of this page.
The guiding principle is the same one that runs through the rest of the system: a machine you understand, can modify, and can trust. The default backend is chosen to work on the widest range of hardware without a vendor-specific driver stack. Vendor-tuned backends are available, but they are opt-in.
What ships today: Vulkan GPU inference, with a CPU fallback
The shipping llama.cpp build is compiled with a Vulkan compute backend and a portable CPU fallback (an AVX2-class instruction-set floor, with non-native codegen so the binary runs on every supported target rather than only the build host). Whether InterGen’s inference actually uses the GPU is decided by the detected hardware tier, not by GPU presence alone: a Tier 1 machine serves on the CPU only, and Tier 2 and Tier 3 machines offload every layer through Vulkan. Selection is automatic — there is no vendor driver stack to install for the default path.
Vulkan is the default because it is vendor-neutral: one API that runs across AMD, Intel, and NVIDIA GPUs through each vendor’s installable client driver, rather than a vendor-specific SDK baked in at build time. For AMD and Intel the open Mesa drivers provide the Vulkan driver in the default image; on NVIDIA the Vulkan driver arrives with the proprietary-driver opt-in. This keeps the default install small, inspectable, and portable while still using the GPU where one is present.
Two different “Vulkans.” The Vulkan graphics stack —
radv(AMD) andanv(Intel), used for the desktop and rendering — and the Vulkan compute backend that accelerates LLM inference are related but distinct. Both ship and work on InterGenOS today. See Vulkan for the details.
Opt-in vendor backends
Beyond the vendor-neutral GPU path, two vendor-tuned backends are available as opt-in choices for users who want maximum throughput on specific hardware. Both are in-tree today; neither is on the installed image:
- ROCm for AMD GPUs. See ROCm (AMD).
- CUDA for NVIDIA GPUs. See CUDA (NVIDIA).
Each typically delivers higher throughput than the vendor-neutral path on its hardware, at the cost of pulling in a vendor-specific runtime that enlarges the trusted surface — so neither is enabled by default. Unlike the vendor-neutral Vulkan backend, which ships enabled, ROCm ships from source as an opt-in, mirror-only compute tier; CUDA is provided via the NVIDIA driver runtime (the CUDA Toolkit is not packaged). The per-backend pages track each one’s status.
How a backend is selected
Backend selection follows a local-first, hardware-detected philosophy. The shipping llama.cpp is built with a Vulkan GPU backend and a CPU fallback (a portable AVX2 instruction-set floor). Which of the two serves a request is decided by the detected tier: Tier 1 serves on the CPU only, even when the machine has a GPU with a working Vulkan driver, and Tier 2 and Tier 3 offload every layer through the vendor-neutral Vulkan path. This happens automatically, with no configuration and no vendor lock-in; an explicit llama_server.gpu_layers integer in your configuration overrides it verbatim on every tier.
The vendor-specific ROCm backend ships as an explicit opt-in (mirror-only), used only where enabled and where AMD hardware is present; the CUDA path is the NVIDIA driver runtime, likewise opt-in. The out-of-the-box path stays vendor-neutral.
Relationship to model tiers
The tier decides both which model runs and how it is served. The compute backend is what makes acceleration possible; the tier is what decides whether it is used. On Tier 2 and Tier 3 the model is offloaded to the GPU through Vulkan and sized for that card. On Tier 1 the model is served on the CPU even when a GPU is present, because the 2B model is sized for CPU serving and a Tier-1 GPU is not treated as an inference device. The tier keys off the presence and size of dedicated VRAM; system RAM is not part of the decision.
For the tier rules and thresholds, see the GPU section overview. For per-GPU notes, see Per-GPU guidance.
What this is not
This page covers the local inference backend only. It is unrelated to optional cloud features:
- InterGen Sentinel, the pluggable security scanner, runs entirely on-device by default (a fast local-rules pass plus an optional local Qwen classifier). Cloud scanning, backed by one of six opt-in providers, is configured by you and off by default.
- Phone-A-Friend (Frontier/Cloud Escalation) is an optional, consent-first path that hands a request to a frontier model in the cloud. It is off by default; no provider is configured out of the box.
Neither feature depends on the GPU backend, and neither is enabled out of the box. The default install runs and reasons entirely on your own hardware.