# Multi-node serving on Dynamo

Serve a model too large for one GPU across two nodes, gang-scheduled by the Dynamo stack.

Source: /guides/serving-multi-node-on-dynamo/

<!-- vale write-good.Passive = NO -->
Qwen2.5-14B's FP16 weights are about 29 GB, larger than one NVIDIA L4's 23 GB, so
it serves across two nodes as a gang: a `Leader` and a `Worker`, one L4 each,
pipeline-parallel across the pair. On a
[Dynamo cluster]({{< ref "/platform/inference-cluster.md#serving-stack" >}}) Grove
and the KAI Scheduler gang-schedule the two pods together, and Modelplane composes
them as a Grove `PodCliqueSet`.

This is the [getting started tour]({{< ref "/getting-started" >}}) scaled to two
nodes: the same `vllm serve`, one larger model, on a `spec.stack: Dynamo` cluster.
[Set up the platform]({{< ref "/getting-started/build-the-platform.md" >}}) first,
for the gateway and cloud credentials, then apply the manifests below.

## Register a Dynamo cluster

The `InferenceClass` describes a single-L4 node, like the getting started tour's
but with more memory and disk for the larger model. The `InferenceCluster` runs
two of them and sets `spec.stack: Dynamo`, so Modelplane installs Grove and the
KAI Scheduler on the cluster.

{{< manifests "guides/serving-multi-node-on-dynamo/inference-class.yaml" >}}

{{< manifests "guides/serving-multi-node-on-dynamo/inference-cluster.yaml" >}}

Provisioning the pool and installing the stack takes about 15 minutes:

```bash
kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m
```

## Cache the weights

A gang reads its weights from a shared cache, so pods don't each pull a copy.
Create the namespace and the cache:

```bash
kubectl create namespace ml-team
```

{{< manifests "guides/serving-multi-node-on-dynamo/model-cache.yaml" >}}

## Deploy the gang

The `Leader` and `Worker` run the same `vllm serve`, differing only in node rank.
`$(MODELPLANE_LEADER_ADDRESS)` resolves to the leader on either stack, but
`$(MODELPLANE_RANK)` isn't injected on Dynamo yet, so the worker derives its rank
from Grove's `GROVE_PCLQ_POD_INDEX`.
[Multi-node deployments]({{< ref "/models/model-deployment.md#multi-node" >}})
covers this.

{{< manifests "guides/serving-multi-node-on-dynamo/model-deployment.yaml" >}}

Wait until `READY` shows `True`. The first start hydrates the cache, so it's
slower than later ones:

```bash
kubectl get md -n ml-team --watch
```

On the workload cluster the gang is a Grove `PodCliqueSet`, the Dynamo stack's
multi-node workload in place of a LeaderWorkerSet:

```bash
kubectl get podcliquesets.grove.io -A   # workload cluster
```

## Expose and query

{{< manifests "guides/serving-multi-node-on-dynamo/model-service.yaml" >}}

Read the endpoint's address and send it a request. The `model` field is the
`--served-model-name` the deployment sets:

```bash
ADDRESS=$(kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}')
kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -s "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"qwen2.5-14b\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'
```

The request routes through the gateway to the leader, which serves the gang's one
endpoint.
<!-- vale write-good.Passive = YES -->