GPU Cluster
Status: Maintained
I brought standalone servers into a 9-node, 67-GPU Slurm fleet serving 30+ researchers. It combines RTX 3090s, A5000s, A6000s, A100 80GBs, and Blackwell RTX PRO 6000s. A single Proxmox host runs login, identity, the Slurm controller, and Docker; the GPU nodes stay on bare metal.
The system
- Identity. FreeIPA provides one login across the fleet. Authentication does not depend on the storage server, and the login path is designed to stay responsive during reconnect bursts.
- Scheduling. Slurm uses cgroups v2 for strict per-job GPU isolation. NUMA-aware pinning places single-GPU jobs on CPUs local to their GPU, without extra flags from the user. GPU time is accounted for in GPU-seconds, with fairshare scheduling, short interactive and longer batch tiers, and reservations around submission deadlines.
- Storage. TrueNAS provides shared storage over NFS, while home directories stay local to each node. NFS delegations are disabled, and reads are cached on local SSD to keep shared storage responsive under load from dozens of concurrent users.
- Network. A 10 GbE MikroTik LAN carries NAS and control-plane traffic, and a self-hosted Netbird WireGuard overlay connects remote nodes and storage.
Much of the upkeep is automated: compute nodes pull their configuration from the controller, and a nightly job reconciles the user roster against the identity system.
The first cluster
From 2023 to 2025 I maintained its predecessor at the XFACT lab: six nodes behind a control plane on a single VMware host, with FreeIPA identity, TrueNAS storage, and Slurm running a fair queue. This cluster grows out of that one: the same identity and storage design, Proxmox in place of VMware, and the fair queue extended into GPU-second fairshare with strict per-job GPU isolation.