Contents
In brief
How Ray’s core primitives, Train, and Serve enable teams to orchestrate large‑scale training and low‑latency inference on GPU clusters without replacing existing infrastructure.
Executive summary
Ray provides an open‑source execution layer that lets platform teams schedule Python tasks, stateful actors, and distributed objects across GPU‑enabled nodes managed by Kubernetes, Slurm, or other cluster managers. Ray Train distributes user‑written training loops, while Ray Serve offers programmable, auto‑scaling inference deployments that can share GPUs and batch requests. The framework does not replace drivers, cluster managers, or model registries; instead it sits between them, handling placement, fault tolerance, and data movement.
Introduction
Modern AI workloads place heavy demands on compute, networking, and operational processes. Training large language models or vision transformers often requires dozens of GPUs that must communicate efficiently, while inference services need to serve thousands of requests per second with low latency. These requirements push the limits of traditional cluster management tools, which were built for batch jobs rather than fine‑grained, Python‑centric AI pipelines. Ray emerges as a unifying framework that abstracts the complexity of distributed execution while leaving the underlying infrastructure—drivers, cluster managers, storage, and networking—under the control of platform teams.
Technical fundamentals
Ray is organized into three logical layers. The top layer, Ray AI Libraries, includes Ray Data, Ray Train, Ray Tune, Ray Serve, and RLlib. Beneath that sits Ray Core, the distributed execution engine that implements tasks, actors, objects, and placement groups. The bottom layer consists of Ray Clusters, which are collections of workers plus a head node that run on a user‑provided cluster manager such as Kubernetes, Slurm, or YARN.
- Tasks – Stateless remote functions submitted by a caller. Ray schedules a task on a worker that satisfies the requested resources and returns an object reference immediately.
- Actors – Stateful workers instantiated from a user class. Method calls are routed back to the same process, preserving in‑memory state such as model weights or open sockets.
- Objects – Values stored in a distributed object store. A reference is passed between tasks and actors; the actual data is fetched on demand.
- Placement groups – Collections of resource bundles that enforce co‑location policies, for example a gang of GPU workers that must start together.
These primitives give developers explicit control over where computation happens and what data moves, while Ray handles the low‑level details of scheduling, serialization, and fault handling.
Architecture and components
The following diagram visualizes the separation of responsibilities:
References
- What is Ray? · Ray
- Key Concepts · Ray
- Ray Train Overview · Ray
- Ray Serve: Scalable and Programmable Serving · Ray



