# Deploy Ray Serve on K8s Cluster

**URL:** <https://discuss.ray.io/t/deploy-ray-serve-on-k8s-cluster/8221>\
**Category:** Ray Serve\
**Created:** [November 10, 2022, 3:20pm UTC](https://discuss.ray.io/t/deploy-ray-serve-on-k8s-cluster/8221 "2022-11-10T15:20:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kasper\_Kooijman](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kasper_kooijman/32/3409_2.png) [@Kasper\_Kooijman](https://discuss.ray.io/u/Kasper_Kooijman)\
**Post date:** [November 10, 2022, 3:20pm UTC](https://discuss.ray.io/t/deploy-ray-serve-on-k8s-cluster/8221/1 "2022-11-10T15:20:48Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi there!

I’m trying to deploy a transformer model using Ray Serve on my Kubernetes cluster, but I’m failing to follow the documentation, so I hope someone can help me over here.

- I’m trying to deploy it on my local docker-desktop K8s cluster
- I have deployed the KubeRay operator
- I have deployed the Ray Cluster

I followed the [getting started guide](https://docs.ray.io/en/latest/serve/getting_started.html) to create the `serve.py` code. I omitted some code for brevity:

```auto
import numpy as np
import torch

from ray import serve
from starlette.requests import Request
from typing import Dict

@serve.deployment
class Predictor:
    def __init__ (self):
        device = "cuda" if torch.cuda.is_available() else "cpu"
        print("device: ", device)
        self.model = torch.load(
            "sbert_v0_epoch2_totaln40000.torch", map_location=device
        ).eval()

    async def __call__ (self, starlette_request: Request) -> Dict:
        request = await starlette_request.json()
        text = request["text"]
        return text

 
predictor = Predictor.bind()

```

I have the following config file, created with `serve build create_vectors:predictor -o predictor_config.yaml`:

```auto
# This file was generated using the `serve build` command on Ray v2.1.0.

import_path: create_vectors:predictor

runtime_env: {}

host: 0.0.0.0

port: 8000

deployments:

- name: Predictor

```

My Ray Cluster is defined as follows:

```auto
kind: RayCluster
metadata:
  name: raycluster-complete
spec:
  rayVersion: "2.0.0"
  enableInTreeAutoscaling: true
  headGroupSpec:
    serviceType: LoadBalancer # Options are ClusterIP, NodePort, and LoadBalancer
    enableIngress: false # Optional
    rayStartParams:
      block: "true"
      dashboard-host: "0.0.0.0"
    template: # Pod template
        metadata: # Pod metadata
        spec: # Pod spec
            containers:
            - name: ray-head
              image: rayproject/ray-ml
              # Keep this preStop hook in each Ray container config.
              lifecycle:
                preStop:
                  exec:
                    command: ["/bin/sh","-c","ray stop"]
              ports: # Optional service port overrides
              - containerPort: 6379
                name: gcs
              - containerPort: 8265
                name: dashboard
              - containerPort: 10001
                name: client
              - containerPort: 8000
                name: serve
  workerGroupSpecs:
  - groupName: ray-worker
    replicas: 0
    minReplicas: 0
    maxReplicas: 5
    rayStartParams:
        block: "true"
        num-gpus: "1"
    template: # Pod template
      spec:
        # Keep this initContainer in each workerGroup template.
        initContainers:
        - name: init-myservice
          image: busybox:1.28
          command: ['sh', '-c', "until nslookup $RAY_IP.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"]
        containers:
          - name: ray-node
            image: rayproject/ray-ml:2.0.0-gpu
            # Keep this preStop hook in each Ray container config.
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh","-c","ray stop"]

```

I think I should somehow incorporate my config file into the Ray Cluster definition, but I’m not sure how and whether it’s correct. How do I go continue from here?

Thanks!

---

<div class="post-metadata">

**Author:** ![Sihan\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sihan_wang/32/2570_2.png) [@Sihan\_Wang](https://discuss.ray.io/u/Sihan_Wang)\
**Post date:** [November 10, 2022, 5:28pm UTC](https://discuss.ray.io/t/deploy-ray-serve-on-k8s-cluster/8221/2 "2022-11-10T17:28:54Z")

</div>

Hi @Kasper_Kooijman , thank you for asking questions!

For k8s deployment guide: [Deploying on Kubernetes — Ray 2.1.0](https://docs.ray.io/en/latest/serve/production-guide/kubernetes.html#serve-in-production-kubernetes)

For your question, yes, you need to merge two configs together. ([Deploying on Kubernetes — Ray 2.1.0](https://docs.ray.io/en/latest/serve/production-guide/kubernetes.html#deploying-a-serve-application) check the `tips`) but it will look like [kuberay/ray\_v1alpha1\_rayservice.yaml at release-0.3 · ray-project/kuberay · GitHub](https://github.com/ray-project/kuberay/blob/release-0.3/ray-operator/config/samples/ray_v1alpha1_rayservice.yaml)

Hope this can help you!