Ray Tune + Kuberay + MLFLow and AWS IAM Permissions

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

When using Ray (v1.11.0) with kuberay (0.2.0) on a kubernetes cluster (1.21.4) on AWS EC2 Instances (not EKS), I have to add the IAM role to the Ray head and worker nodes/pods as well as the MLFlow pods even though MLFlow is theoretically the one that needs to write to S3 etc. and Ray Tune just passes the information to be uploaded along.

This does not seem like expected behavior and more like a bug. Is there a reason why it is that way right now?

2

Hey @clizarraga can you elaborate more on what your setup looks like, specifically how you’re using MLFlow with Tune? If you’re using the MLflowLoggerCallback, then the MLflow client should be running directly on the Ray nodes. Let me know if I’m misunderstanding something here!