What GPU clusters ask of a platform team

Platform teams should name the pools. A training pool can be preemptible. A serving pool should not be. The scheduler, whether it is Kubernetes or a vendor console, is only as good as that policy.

What has to be observed

GPU hardware is expensive and easy to waste. A job can hold devices while it waits on data. A service can reserve a whole device for a trickle of requests. The useful graphs are utilization over time, queue time, failure rate, and cost per successful job. A screenshot of a dashboard during a demo is not an operating practice.

Checkpointing and restart matter because the job will fail. Disks fill. A node disappears. A training run that cannot resume is a process that only works on the days nothing breaks.

Isolation is a security control

Accelerators often sit beside sensitive data: proprietary text, customer content, model weights. The cluster needs the same identity and network rules as the rest of the platform. Who can submit a job? Can a job in one namespace read another team’s dataset? Are the weights stored with access logs? Kubernetes gives you namespaces and RBAC as building blocks. It does not apply them for you.

A sketch that is intentionally generic:

Researchers -> submission service (authenticated)
  -> scheduler
    -> training pool (preemptible, checkpointed)
    -> serving pool (protected, autoscaled)
Data and weights -> access-controlled storage
Telemetry -> central logs, including who submitted the job

Analysis

The mature question is not “how many GPUs.” It is which jobs may consume them, what happens when a job dies, and whether a team can explain the bill. Those are platform questions. The model vendor will not answer them inside your cluster.

What this does not settle

This piece does not recommend a chip, a cloud, or a cluster size. Those numbers go stale, and inventing them would be a different kind of article. Size the pool from your job lengths and your latency target, then measure.