Auto repair for Together GPU Clusters: from GPU fault to healthy node

Together AI describes its automatic node remediation system for GPU clusters, which detects faults from both inside the VM (Xid errors, DCGM ECC and thermal signals, Node Problem Detector) and on the host (InfiniBand rail state, host reachability), then recommends and optionally executes repairs. Repairs follow a cordon-wait-drain-act-rejoin lifecycle with an escalation ladder from reboot to quick reprovision to migration, governed by per-fault consent, wait-for-idle controls, concurrency budgets, cooldowns, and a circuit breaker. Migrated or removed machines are automatically re-validated with GPU, NCCL, InfiniBand, and storage checks, and only an explicit failure files a hardware ticket.