Local AI control plane

A small FastAPI service for starting, stopping and observing local language-model services and the GPU they share.

automation · in use

Designer and builder · Updated Aug 2026

  • Local AI
  • Operations
  • Observability
  • GPU

01 · Problem

Multiple local models compete for a finite GPU, while their runtime state, throughput and failures are otherwise split across service and command-line tools.

02 · System

A single service reads systemd state, model metrics and GPU telemetry, exposes a control dashboard and applies explicit start/stop actions to each model service.

03 · Outcome

The control surface reports model uptime, token throughput, active sessions, VRAM use, GPU utilization and temperature from one local endpoint.

Context

The system is intentionally small and host-native. It wraps existing service managers and metrics rather than introducing a second orchestration stack.

Architecture

  1. 01FastAPI control and health endpoints
  2. 02systemd service inspection and lifecycle commands
  3. 03Prometheus-compatible llama.cpp metric parsing
  4. 04nvidia-smi GPU telemetry
  5. 05Self-contained browser control panel

Decisions & trade-offs

Wrap the host's source of truth

Systemd remains responsible for process state; the application provides a safer, more legible interface over it.

Evidence

  • One view for two local model services and shared GPU state
  • Health endpoint and systemd-managed runtime
  • Loading and error states for unavailable model metrics

What this taught me

  • A control plane should expose the underlying state clearly instead of pretending to own it.