How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
I have deployed a Ray Cluster based on this Helm Chart with some slight modifications made to the Ingress to port forward the Dashboard, Client and Serve ports like so:
Assume the Host URL is https://ray.com
- Ray Dashboard is accessible from
https://ray.comas I have set/to port forward port 8265 - Ray Client is accessible from
https://ray.com/clientas I have set/clientto port forward port 10001 - Ray Serve is accessible from
https://ray.com/serveas I have set/serveto port forward port 8000
The dashboard works while the serve and client endpoints return a 404 Not Found error, but I am assuming that they are supposed to do that when you make a curl request to them.
NOTE: I am accessing the Ray Cluster which is deployed within a VPC through a VPN.
Methods to Submit a Job Attempted
- Submitting a job to the cluster to run works when I use the REST API method:
# Output of the jobs I deployed
{'raysubmit_jxQVCQ3LRGHtuw8C': {'status': 'SUCCEEDED', 'entrypoint': 'echo hello', 'message': 'Job finished successfully.', 'error_type': None, 'start_time': 1661220561123, 'end_time': 1661220561801, 'metadata': {'job_submission_id': '1234'}, 'runtime_env': {}}, 'raysubmit_TXw3cZHZuy5s9FaD': {'status': 'SUCCEEDED', 'entrypoint': 'echo hello', 'message': 'Job finished successfully.', 'error_type': None, 'start_time': 1661153417462, 'end_time': 1661153418330, 'metadata': {'job_submission_id': '123'}, 'runtime_env': {}}}
- Submitting a job via the Python SDK does not work, I have tried the following:
client = JobSubmissionClient("http://ray.com") # connection timeout-
client = JobSubmissionClient("http://ray.com/client") # Jobs API is not supported on the Ray cluster. Please ensure the cluster is running Ray 1.9 or higher.(not sure how to upgrade the Ray Cluster as I am using the latest image as it is.
- Submitting a job via the CLI:
I tried to first all the jobs but even that does not work. Why does Ray automatically assume that I need to port-forward the address that is passed in? Is there any way to disable this?
> ray job list --address "http://ray.com"
ConnectionError: Failed to connect to Ray at the address: http://ray.com:8265.
Ray Serve Job Submission Attempt
I also tried to deploy a simple Ray Serve Deployment based on the Ray Serve Deployment example.
import ray
from ray import serve
# Connect to the running Ray cluster.
ray.init(
address="ray://ray.com",
namespace="test-serve",
)
# Bind on 0.0.0.0 to expose the HTTP server on external IPs.
serve.start(detached=True, http_options={"host": "0.0.0.0"})
@serve.deployment(route_prefix="/hello")
def hello(request):
return "hello world"
hello.deploy()
I tried all the following addresses as well:
ray://ray.com/clientray://ray.com/serve
But all of them throw the same error ConnectionError: ray client connection timeout.
Ingress
I have attached the Ingress file below for reference just in case there is some error on my end with the deployment:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "staging-ray-cluster-ingress"
namespace: {{ .Values.operatorNamespace }}
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/security-groups: {{ .Values.sgElb }}
alb.ingress.kubernetes.io/group.name: "ray"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: {{ .Values.certificate }}
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/healthcheck-path: "/"
alb.ingress.kubernetes.io/success-codes: "200"
labels:
app: "staging-ray-cluster"
spec:
rules:
- host: {{ .Values.externalDNS.domainName}} # ray.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ray-head
port:
number: 8265
- path: /
pathType: Prefix
backend:
service:
name: ssl-redirect
port:
name: use-annotation
- path: /client
pathType: Prefix
backend:
service:
name: ray-head
port:
number: 10001
- path: /serve
pathType: Prefix
backend:
service:
name: ray-head
port:
number: 8000