Submitting jobs to a remote cluster via Airflow

What is the best practice for opening up a ray cluster publicly to submit jobs to it from multiple different environments?

We want a centralized ray cluster to submit jobs to from developer computers as well as Airflow running on k8s. It seems the only way to do this is to run ray dashboard <cluster.yml>. However, for containers, is that really the best way?

Is there a best practice for this use case, like using a load balancer in front of ray? If so, what ports would be needed, just 8265?

I’ve read the docs and the section on using a remote cluster is very slim.

Hi jtele2!

Hopefully this will help you out:

  1. Set up an NGINX reverse proxy, AWS ALB, or GCP Load Balancer to route traffic and provide a stable access point. This is especially useful for handling multiple clients like developer machines and Airflow. I do believe you can set up your own ports for it, so 8265 should be fine (let me know if it is different).
  2. Open 8265 (or whatever your load balancing port is) for the Ray Dashboard and 10001 for Ray Client (if needed). Other ports may be required depending on your setup.
  3. Lock things down with authentication (tokens, identity providers) and only allow trusted sources. Don’t leave your Ray cluster wide open to the internet.
  4. KubeRay makes it easier to manage Ray clusters in Kubernetes.
  5. Ray nodes need to talk to each other, so check your firewall and security groups to allow bi-directional traffic.
  6. These integrate well with Ray and give you visibility into cluster performance.

Some more docs that might help you: