GPU Cluster: OSI Lab
Status: Active
Unified a set of standalone servers into a 9-node, 67-GPU Slurm fleet for 50+ researchers. The hardware is deliberately heterogeneous: RTX 3090s, A5000s and A6000s, A100 80GBs, and Blackwell RTX PRO 6000s. A single Proxmox host runs the control plane (login, identity, the Slurm controller, and Docker), while the GPU nodes stay bare metal.
The platform:
- Identity: FreeIPA, one unified login across the fleet
- Scheduling: Slurm with cgroups-v2 and NUMA-aware per-job GPU isolation
- Network: a 10 GbE MikroTik LAN for the NAS and control plane, with a self-hosted Netbird WireGuard overlay linking remote nodes and their storage
- Storage: TrueNAS shared storage, home local to each node
Some of the engineering:
- Correct by default. NUMA-aware pinning lands a single-GPU job on NUMA-local CPUs with no flags to set.
- A filesystem that holds up. NFS delegations are off and reads are cached to local SSD, so it stays fast under dozens of concurrent users.
- Resilient identity. The login path stays responsive under reconnect bursts and never depends on the storage server.
- Self-running. Compute nodes pull their config from the controller, and a nightly job reconciles the user roster from the identity system.
- Fair by design. Fairshare bills GPU-seconds, with short interactive and longer batch tiers, plus reservations around submission deadlines.