GitHub Launches Code‑Offline: Private, Containerized AI Coding Agent Using Llama.cpp and

GitHub’s new Code‑Offline stack stitches together the Pi coding agent with the open‑source llama.cpp inference engine, delivering a fully containerized AI assistant that runs entirely on‑premises. The project, hosted on GitHub under the opensecurity/code‑offline repository, ships a Docker‑Compose environment that can be toggled between pure‑CPU mode and NVIDIA‑GPU acceleration with a single `MODE=gpu` flag in the Makefile commands [GitHub repo]. By default it pulls the Qwen 3.5 models from Hugging Face, using the UD‑Q4_K_XL quantization to balance speed and precision, and stores them in a persistent `models/` volume so subsequent runs avoid re‑download [GitHub repo].

Configuration is driven by a simple `.env` file; developers can swap the Hugging Face repository or model identifier without rebuilding the containers, enabling rapid experimentation with alternatives such as the 4‑billion‑parameter or the 35‑billion‑parameter Qwen 3.5 variants listed in `agent_data/agent/models.json` [GitHub repo]. The agent’s state—including chat history, authentication tokens, and runtime settings—is persisted in `agent_data/`, allowing a seamless hand‑off between sessions while keeping all data confined to the host machine’s filesystem [GitHub repo].

The Makefile abstracts the lifecycle: `make build` constructs the images, `make start` launches a background llama.cpp server that automatically downloads the selected model on first run, and `make agent` drops the user into an interactive terminal attached to a temporary container. When the session ends, the container self‑cleans, leaving only the workspace code, model cache, and agent state on disk [GitHub repo]. For teams that need GPU throughput, the same commands accept `MODE=gpu`, which activates the NVIDIA Container Toolkit integration and routes inference through the GPU‑enabled llama.cpp binary [GitHub repo].

GitHub positions Code‑Offline as a privacy‑first alternative to cloud‑based code‑completion services. Because the entire stack runs behind the developer’s firewall, no API keys or telemetry are sent to external endpoints; the llama.cpp backend advertises an “openai‑completions” API surface but requires no `apiKey`—the `.env` defaults to `none` [GitHub repo]. This design aligns with a broader industry trend toward on‑prem LLM deployment, exemplified by Alibaba’s recent release of the Qwen 3 series, which has been touted as “state‑of‑the‑art” among open models [VentureBeat]. By leveraging the same Qwen 3.5 weights, GitHub’s offering inherits the performance gains reported for those models while avoiding the licensing and cost constraints of proprietary APIs.

Early adopters can test the stack on any machine that supports Docker, with optional GPU acceleration for faster token generation. The repository includes a `make upgrade` target to pull fresh base images and rebuild without cache, ensuring that security patches and upstream llama.cpp updates can be incorporated with minimal friction [GitHub repo]. While the solution is still community‑maintained rather than an official GitHub product, its open‑source nature permits enterprises to audit the code, extend the agent’s capabilities, or integrate it into existing CI/CD pipelines. In an ecosystem where developers increasingly demand both powerful AI assistance and strict data sovereignty, Code‑Offline offers a pragmatic, self‑hosted bridge between the two.