Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Language-Runtime Foundations

InterGenOS ships a built-from-source toolchain and a set of language runtimes that the AI subsystem and other workloads run on. This page explains what those foundations are, where they come from, and how the local AI assistant uses them. The goal is the same one that runs through the whole system: a machine you understand, can modify, and can trust.

This page describes InterGenOS as it ships. Package counts below are derived from the live package tree and drift over time; treat them as a snapshot, not a fixed contract.

Where runtimes come from

Everything is built from source through a staged pipeline rather than assembled from prebuilt binaries. The build runs in 21 phases:

validate → verify-sources → setup → toolchain → chroot-prep → chroot-tools → core → config → core-extra → base → kernel → desktop → extra → compute → ai → bootloader → image → manifest → squashfs → ukis-verity → iso

A publish phase is optional and runs after the others.

The runtime foundations are laid early. The toolchain phase builds the compiler and core build tooling that every later phase depends on; the core and base phases bring in the system libraries and language runtimes that user-facing software links against; and the ai phase builds the assistant on top of those runtimes.

Packages are organized into seven tiers. The live tree holds over 1,100 package definitions — desktop is the largest at 460+, core follows at 310+, and toolchain is the smallest at under three dozen. These numbers move as packages are added and consolidated. Derive the current counts from the package tree rather than quoting any figure as permanent.

Not every definition ships on the installed image: roughly three quarters do, and the rest are mirror-only — present in the repository, installed only when you ask for them. Several language toolchains are in that mirror-only set rather than on the image, so a default installation does not carry them.

The runtime the AI subsystem uses

The local assistant, InterGen, is a tiered, hardware-detected, offline-first local assistant with zero telemetry. It runs entirely on the local machine by default, with no cloud dependency, and selects a model sized to the hardware it detects.

InterGen has two distinct runtime layers, and separating them helps because they place different demands on the system:

  1. The model-serving layer. InterGen serves its selected model through llama-server, the HTTP server from llama.cpp, managed as a subprocess (intergen/llama_manager.py). This is the native, compiled inference runtime that does the heavy numeric work. It is built with a Vulkan GPU compute backend and a CPU fallback — on a supported GPU with a working Vulkan driver it accelerates through Vulkan, otherwise it runs on the CPU (see Overview & Backend Selection). It exposes the model over a local HTTP API.

  2. The orchestration layer. The router, safety classifier, hardware probe, model manager, memory store, D-Bus service, and MCP client are organized as modules (intergen/router.py, intergen/safety.py, intergen/hardware.py, intergen/model_manager.py, intergen/memory.py, intergen/state_cache.py, intergen/dbus_daemon.py, intergen/mcp_client.py). These coordinate the assistant: deciding how to answer a request, classifying its safety, and managing state.

These orchestration modules run on the shipping Python runtime, which is Python 3.14 (3.14.3 in the current release).

Rust lives under /opt, and the package puts it on your PATH

Rust 1.95.0 is a core-tier package. Its binaries live under /opt/rustc-1.95.0/bin, reached through the /opt/rustc symlink, rather than in /usr/bin. The package itself carries /etc/profile.d/rustc.sh, which prepends /opt/rustc/bin to PATH, so cargo and rustc are on the path of any login shell started after the package is installed.

If which cargo comes back empty, the usual cause is a shell session that predates the install — profile scripts are read at login. Start a new login shell, or add the directory for the current one:

export PATH=/opt/rustc/bin:$PATH      # this session only

Earlier development images placed that profile script through a post-install step that could abort before writing it, which left Rust installed but off the path. In R001 the script is part of the package payload, so it is written when the package is installed.

The split matters for runtime planning. The orchestration layer is light and predictable. The model-serving layer is where available GPU memory decides which model can run at all; on a machine falling back to the CPU, throughput on the larger tiers is CPU-bound, which is exactly why such a machine is assigned the small model instead.

Hardware tiers and what they need

InterGen probes the GPU and its dedicated VRAM at startup (intergen/hardware.py) and assigns a tier; the model catalog and selection live in intergen/model_manager.py. System RAM is detected and reported but is never an input to the tier decision — a machine with no discrete GPU serves the small model regardless of how much RAM it has, because the 9B on CPU runs at roughly 50 seconds a query and is unusable. Capability that cannot be determined always fails down. A card counts as discrete only if it exposes at least 3 GB of dedicated VRAM, which is what separates a real card from an integrated GPU borrowing system memory.

The tiers are:

  • Tier 1 (2B, the universal floor) — InternVL3.5-2B Q4_K_M, about 1.2 GB. Selected when there is no discrete GPU, when the card has less VRAM than Tier 2 requires, or when VRAM cannot be read. It handles semantic matching, system queries, keyword extraction, and everyday explanation, and its tool dispatch is code-owned. It is not aimed at large-scale code generation.
  • Tier 2 (9B, shipping) — Qwen3.5-9B Q4_K_M, about 5.5 GB, selected for a discrete GPU with at least ~7 GB of VRAM, sized so the model, its vision projector and the load-time buffers all fit. An 8 GB card qualifies; a 6 GB card does not. On such a machine InterGen runs the 9B natively and unlocks native tool-calling, gated by the safety classifier and Sentinel.
  • Tier 3 (35B) — Qwen3.5-35B-A3B Q4_K_M, a Mixture-of-Experts model, about 21 GB, selected for a discrete GPU with at least ~22 GB of VRAM so the model stays resident. A 24 GB card qualifies; a 20 GB card deliberately does not. The model is catalogued but ships without a pin, and an unpinned model can be neither downloaded nor load-verified — so rather than dead-ending the install, the recommendation is capped to the highest pinned tier and a Tier-3 machine runs the 9B today. The cap is logged, not silent.

A small embedding model (nomic-ai/nomic-embed-text-v1.5, Q8_0, about 139 MB) ships alongside every tier to power the router’s semantic-matching layer.

GPU presence and VRAM size are the signals InterGen uses to select a tier, and the tier then decides the serving path as well as the model. Inference is served by a bundled llama.cpp built with a Vulkan GPU backend and a CPU fallback (a portable AVX2 instruction-set floor). On Tier 1 the model is served on the CPU only, whether or not the machine has a GPU: the 2B is sized for CPU serving and a Tier-1 GPU is not treated as an inference device. On Tier 2 and Tier 3 every layer is offloaded to the GPU through the Vulkan loader and the installed driver. A machine whose tier cannot be determined offloads, matching the assume-Tier-2 posture used for absent detection. An explicit llama_server.gpu_layers integer in your configuration is honoured verbatim on every tier — it is the last word, so 0 pins CPU-only serving and any other number is used as written. See the Overview & Backend Selection page.

How the runtime resolves a request

InterGen does not hand every prompt to the model. A priority-ordered router (intergen/router.py) tries to satisfy each request with the cheapest, most predictable method first and only escalates the runtime cost when it has to:

  • Priority 0 (Decomposition) — splits compound requests into sub-tasks and routes each in turn.
  • Priority 1 (Keyword/Regex Match) — fast pattern matches for common system commands, dispatched to a built-in tool without invoking the model.
  • Priority 2 (Semantic Embedding Match) — lightweight embedding search against a precomputed capability catalog; a high-confidence match dispatches to a built-in tool.
  • Priority 3 (LLM Tool Calling) — a gated path in which the query plus a tool schema go to the model, which selects a tool and arguments for the router to execute. On the 2B floor this path is disabled: tool dispatch is code-owned — the deterministic Priority 1/2 matchers select the tool and extract the arguments, and the model never decides a tool call. Native tool-calling unlocks on the shipped 9B tier (intergen/dispatch_policy.py), where a capable machine lets the model decide tool calls under the same safety-classifier and Sentinel gates; the 2B floor stays code-owned.
  • Priority 4 (LLM Free Response) — the fallback: the model answers conversationally from its own knowledge and the conversation context.

The practical effect is that most interactions never touch the model-serving runtime at all, which keeps the system responsive even on modest hardware.

Safety classification at the runtime boundary

Every action the router proposes passes through the classifier in intergen/safety.py, which sorts it into one of three tiers:

  • AUTO — read-only or harmless operations (for example ls, grep, systemctl status). Run immediately.
  • CONFIRM — state-changing operations (for example systemctl restart, pkm install, or editing a config file). The assistant pauses and asks for approval first.
  • BLOCKED — destructive or security-bypassing operations (for example rm -rf /). Refused outright, with an explanation.

Nothing that changes the system runs without explicit approval, and the most dangerous commands cannot run at all.

These three classes are the assistant-facing view of a larger trust model. For how AUTO/CONFIRM/BLOCKED relate to the system’s gating outcomes and trust zones, see Registry & Trust Boundary.

Crossing the network boundary

Two features can take a request off-device, and both run only when you ask:

  • InterGen Sentinel is a pluggable security scanner that inspects content crossing the ingress and egress boundaries. Its default configuration runs two local stages: a fast local-rules pass and an optional deep pass backed by a small local Qwen classifier. For deeper analysis you may opt in to one of six named cloud providers — Claude (Anthropic), Gemini (Google), Copilot (Microsoft), ChatGPT (OpenAI), Grok (xAI), or DeepSeek — or to any custom OpenAI-compatible endpoint you supply. None is configured by default, so a default install scans entirely on-device.
  • Phone-A-Friend (Frontier/Cloud Escalation) is an optional, consent-first path that hands a request to a more capable frontier model in the cloud when the local assistant cannot satisfy it. It is off by default; the same provider set can be configured. An API key is written to the system keyring and never into the configuration file, which records only the keyring identifier. Every outbound payload is scanned by Sentinel’s egress policy before it leaves the machine.

The rest of the platform

Beyond the AI subsystem, the same source-built foundations support the shipping system: pkm (the package manager), Forge (the installer), a signed Secure Boot chain, dm-verity integrity, and UKI signing. The shipping desktop is GNOME 49 on Wayland.

See also