Ray images should not assume any particular uid

Ray Clusters Kubernetes
1

This is typical error from running community images on OpenShift, where all containers run using an anonymous UID:

WARNING: The directory '/home/ray/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numpy in ./anaconda3/lib/python3.7/site-packages (1.20.0)
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/ray/.local'

Recommend changing all files with:

chown -R 9998:0 /dir
chmod -R g+rwX /dir
USER 9999  # different than 9998 above

In general /home/ray assumes a user ray, which in openshift will definitely never exist.

2

an example of a dockerfile I made recently:

If I can run this using USER 9999 it will probably also work in OpenShift when the user id is 100003034243

3

@Erik_Erlandson
Thanks, noted re: uids.

The offending lines causing the error are these:

You can fix the particular error you got by removing this setupCommands field.

The “setupCommands” field was ported from the Ray cluster launching config set up – but to be honest this doesn’t make too much sense in the context of Ray on Kubernetes.
For that reason, this PR also removes this field from the operator config examples: [autoscaler][kubernetes] Ray client setup, example config simplification, example scripts. by DmitriGekhtman · Pull Request #13920 · ray-project/ray · GitHub

I like how OpenShift’s default behavior is catching these bits of code and configuration that don’t belong in the Ray operator.

Does autoscaling work once the setupCommands are removed?
I don’t think there should be any issues running head and worker ray start commands once that’s done.

4

thanks Dmitri!
I will remove those and test it when I have a chance and let you know.

5

Also cc-ing Ian (Anyscale Docker expert) here :grinning:
@ijrsvt