Ingress not routing to Ray Dashboard

Ray Clusters Kubernetes
1

I am wondering if anyone has been successful in using an ingress to access the Ray dashboard. When I access the browser using the ingress rule, I get a 404 error:

192.168.65.3 - BA [15/Mar/2022:21:38:15 +0000] "GET /c3/ray HTTP/1.1" 404 14 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" 1520 0.002 [khk-local-env-app-k8sray-local-ray-head-8265] [] 10.1.2.168:8265 14 0.001 404 89ad533747ec91f3e7b1e96dd343fa7f

My ingress controller is using the following:

registry: k8s.gcr.io
image: ingress-nginx/controller
tag: v.1.1.0

Output of kubectl -o yaml get ingress ... is:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: app
    meta.helm.sh/release-namespace: khk
    nginx.ingress.kubernetes.io/proxy-body-size: 250m
  creationTimestamp: "2022-03-15T18:26:00Z"
  generation: 7
  labels:
    app: local-env-app
    release: app
  name: local-env-app-k8singr-ray
  namespace: khk
  resourceVersion: "555169"
  uid: e0349a39-c7db-4bff-8cb5-ae8d3bacada9
spec:
  rules:
  - host: dev.platform.com
    http:
      paths:
      - backend:
          service:
            name: local-env-app-k8sray-local-ray-head
            port:
              number: 8265
        path: /app/ray
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - hostname: localhost

My Ray head service looks like:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-03-15T18:43:12Z"
  name: local-env-app-k8sray-local-ray-head
  namespace: khk
  ownerReferences:
  - apiVersion: cluster.ray.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: RayCluster
    name: local-env-app-k8sray-local
    uid: cf43f6da-f05b-459e-96d5-b213e174b2fb
  resourceVersion: "543577"
  uid: 2b0423df-b8e9-4397-9ba7-441a39dabb88
spec:
  clusterIP: 10.108.179.183
  clusterIPs:
  - 10.108.179.183
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: client
    port: 10001
    protocol: TCP
    targetPort: 10001
  - name: dashboard
    port: 8265
    protocol: TCP
    targetPort: 8265
  - name: ray-serve
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    cluster.ray.io/component: local-env-app-k8sray-local-ray-head
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer:
    ingress:
    - hostname: localhost

Ray head is currently running

ray start --head --no-monitor --include-dashboard true --dashboard-host 0.0.0.0

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
2

Found something interesting. When I set the ingress path to be:

      - backend:
          service:
            name: local-env-app-k8sray-local-ray-head
            port:
              number: 8265
        path: /
        pathType: Prefix

I can access the dashboard from dev.platform.com/#, but not on any other url. Perhaps I missed something here?

3

Got this to work with an nginx rewrite target annotation

4

Thanks @Kun_Hwi_Ko , please let us know if you need any further help.

5

We’ll keep in mind to document network set up for the Ray on K8s on the major cloud providers.

6

Could you please share your ingress.yaml file, please. I have the same problem with GCP using GCE.
Here is my config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ray-external-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - http:
      paths:
        - path: /ray-cluster-1/*
          pathType: ImplementationSpecific
          backend:
            service:
              name: ray-cluster-1-head-svc
              port:
                number: 8265
7

HI @Kun_Hwi_Ko would you able to share the sample code for ingress with rewrite ?

8

Facing a similar issue here. RayServe application via ingress unreachable - Ray Libraries (Data, Train, Tune, Serve) - Ray