GPU Cluster
Status: Active
Unified a set of standalone servers into a 9-node, 67-GPU Slurm fleet for 50+ researchers. The hardware is deliberately heterogeneous: RTX 3090s, A5000s and A6000s, A100 80GBs, and Blackwell RTX PRO 6000s. A single Proxmox host runs the control plane (login, identity, the Slurm controller, and Docker), while the GPU nodes stay bare metal.
The system
- Identity. FreeIPA, one login across the fleet. The login path stays responsive under reconnect bursts and never depends on the storage server.
- Scheduling. Slurm on cgroups v2, with NUMA-aware pinning that lands a single-GPU job on NUMA-local CPUs with no flags to set. Fairshare bills GPU-seconds, with short interactive and longer batch tiers, plus reservations around submission deadlines.
- Storage. TrueNAS shared over NFS, home directories local to each node. Delegations are off and reads cache to local SSD, so it stays fast under dozens of concurrent users.
- Network. A 10 GbE MikroTik LAN for the NAS and control plane, and a self-hosted Netbird WireGuard overlay that makes remote nodes and their storage feel local.
The fleet runs itself: compute nodes pull their config from the controller, and a nightly job reconciles the user roster from the identity system.
The first cluster
From 2023 to 2025 I maintained its predecessor at the XFACT lab: six nodes behind a control plane on a single VMware host, with FreeIPA identity, TrueNAS storage, and Slurm running a fair queue. This cluster scales up what that one proved: the same identity and storage design, the hypervisor swapped for Proxmox, and the fair queue grown into fairshare billing with strict per-job GPU isolation.