Blog›Engineering
Engineering · 16 min read

How We Built a Multi-Tenant Cloud Platform on Incus

LM
Layla Mansour ·April 15, 2025·16 min read
tenant-a: project
tenant-b: project
Incus cluster — shared hardware
OVN
ZFS
cgroups v2

This is an engineering post about the decisions that went into VeloxaHost's infrastructure layer. We chose Incus as our virtualization backbone and built a multi-tenant platform on top of it. Here's what that means in practice — the good, the hard parts, and the patterns that work.

Why Incus, Not Kubernetes?

The obvious question when building a cloud platform in 2025 is: "why not Kubernetes?" We evaluated it carefully and decided against it for our use case. Here's why:

  • Kubernetes is for application containers. Our customers deploy system-level workloads — they SSH in, install packages, run databases, configure cron jobs. Kubernetes doesn't model that at all. System containers vs app containers →
  • The operational overhead is enormous. K8s requires significant expertise to run securely in multi-tenant mode. Incus multi-tenancy is an explicit design feature, not an afterthought
  • Resource efficiency. System containers share the host kernel — far lower overhead per workload than Pods running in dedicated VMs

We use Kubernetes inside our own infrastructure (for our control plane services). We do not use it as the compute layer for customer workloads.

Tenant Isolation: Incus Projects

The fundamental isolation primitive in Incus is the project. Every VeloxaHost tenant gets their own Incus project. Here's what that gives us:

# Create a project for a new tenant
incus project create tenant-a8f2c3 \
  --config features.networks=true \
  --config features.storage=true \
  --config features.profiles=true \
  --config limits.containers=50 \
  --config limits.memory=128GB \
  --config limits.cpu=256

Within a project:

  • Containers, VMs, networks, and storage pools are scoped to the project — a tenant cannot see or interact with another tenant's resources in any way
  • Resource limits (limits.containers, limits.memory, limits.cpu) are enforced at the project level — one tenant cannot starve another
  • Profiles (instance configuration templates) are per-project — tenant A's default profile doesn't affect tenant B

Networking: OVN

Every tenant project gets its own OVN logical switch. OVN (Open Virtual Network) is a software-defined networking layer built on Open vSwitch — the same technology used in OpenStack and many hyperscalers.

# Each tenant gets an isolated OVN network
incus network create tenant-a8f2c3-net \
  --project tenant-a8f2c3 \
  --type ovn \
  --config network=UPLINK \
  --config ipv4.address=10.100.0.1/24 \
  --config ipv4.nat=true \
  --config ipv6.address=none

What this gives us:

  • Per-tenant private networks — 10.x.x.x ranges are tenant-specific and fully isolated at the OVN switch level
  • NAT and floating IPs — tenants can assign public IPs to specific instances without exposing the whole network
  • Firewall rules at the logical network level — ingress/egress filtering before packets reach the host kernel
  • No cross-tenant routing — OVN logical routers are per-tenant; there is no mechanism for traffic to leak between tenant networks

Storage: ZFS

Each tenant gets a ZFS dataset within a shared pool. ZFS provides:

# Tenant storage pool allocation
incus storage create tenant-a8f2c3-pool zfs \
  --project tenant-a8f2c3 \
  --config source=pool0/tenants/a8f2c3 \
  --config size=2TB \
  --config zfs.pool_name=pool0
  • Instant snapshots — ZFS copy-on-write means snapshots are taken in milliseconds, regardless of dataset size
  • Storage quotas — ZFS enforces per-dataset quotas; a tenant consuming 2 TB cannot consume more, even if the underlying pool has space
  • Efficient clones — When a tenant deploys from a marketplace image, we clone the base ZFS dataset rather than copying it — near-instant, regardless of image size
  • Compression — zfs.compression=lz4 gives us ~30% effective storage savings across typical workloads

Resource Limits: cgroups v2

Every instance gets resource limits enforced by Linux cgroups v2, set at the Incus instance level:

# Example: a 2 vCPU / 4 GB instance
incus config set instance-name \
  limits.cpu=2 \
  limits.memory=4GB \
  limits.memory.enforce=hard \   # Hard limit — no overcommit
  limits.cpu.priority=50 \       # CPU share for fair scheduling
  limits.disk=/: 50GB

Key decisions:

  • Hard memory limits — we don't overcommit memory. When an instance hits its memory limit, the kernel OOM-kills processes inside the container — it does not affect other instances or the host
  • CPU shares, not pinning — CPUs are shared across instances using cgroups CPU weight. A burst to higher usage is allowed when the host has spare capacity, but sustained usage is limited by the purchased plan
  • Disk I/O limits — limits.disk.io.read and limits.disk.io.write to prevent a single tenant from saturating the storage pool's IOPS

The Control Plane

VeloxaHost's console is a Python/Flask application that talks to Incus via its REST API. The Incus API is the source of truth for all compute operations:

# Python SDK example — how we provision an instance
import pylxd

client = pylxd.Client(endpoint="https://incus.internal:8443", cert=("client.crt", "client.key"))

instance = client.instances.create({
    "name": f"gh-{tenant_id}-{instance_id}",
    "source": {"type": "image", "fingerprint": image_fingerprint},
    "profiles": ["default", f"plan-{plan_slug}"],
    "project": f"tenant-{tenant_id}",
    "config": {
        "limits.cpu": str(plan.vcpu),
        "limits.memory": f"{plan.memory_gb}GB",
        "user.data": cloud_init_config,
    },
    "devices": {
        "root": {"path": "/", "pool": f"tenant-{tenant_id}-pool", "size": f"{plan.disk_gb}GB", "type": "disk"},
        "eth0": {"name": "eth0", "network": f"tenant-{tenant_id}-net", "type": "nic"},
    },
}, wait=True, project=f"tenant-{tenant_id}")

Lessons Learned

1. cgroups v2 is non-negotiable. cgroups v1 had partial isolation — v2 adds proper memory accounting for all subsystems including anonymous memory, shared memory, and kernel pages. Upgrade your kernel if you haven't.

2. ZFS recordsize matters for databases. Default ZFS recordsize (128K) is great for sequential I/O but hurts database random access. We set recordsize=8K for instances running PostgreSQL and MySQL.

3. OVN uplink bandwidth shaping is essential. Without it, one tenant's traffic spike affects everyone. We use OVN's limits.egress and limits.ingress per logical network port.

4. The Incus REST API is stable and excellent. We've never had an API breakage between minor versions. The project was clearly designed with automation in mind, unlike some hypervisor APIs we've dealt with.

Running infrastructure built on this stack — deploy your first instance →