Private python dependencies and RuntimeEnv

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have some python packages I’m passing to the RuntimeEnv py_modules as wheel files, however some of these packages depend on packages from private registries. I can not figure out the best way to go about giving access to the Worker process via credentials. I tried setting the env vars PIP_EXTRA_INDEX_URL and PIP_INDEX_URL but it doesn’t seem to pass the value to the subprocess call when installing the wheel. I thought about using a requirements.txt with the information but I don’t want to hardcode sensitive information in my distribution, and can’t enforce others to do so as well. Any ideas?

I see a similar question here however from what I can tell they are using the pip option and not py_modules and from the quick skim of the code it looks like the two use different functions when calling pip install command.

Hi @wizrds . Can you share what happened when you tried Adding index option to pip runtime_env ? E.g. adding PIP_INDEX_URL to runtime env vars? Maybe there is some workaround that unblocks you.

For secrets, there’s an open feature request for this. Would be great if you could +1/add a small blurb here [Core]Read working_dir zip from private s3 · Issue #34708 · ray-project/ray · GitHub

Ok I tried a few different things. I’m using package-a which has a dependency to package-b but package-b is in a private index. When I pass the package-a wheel file to py_modules, and set the pip env vars it fails with RuntimeError: Failed to install py_modules wheel. When I pass the package-a wheel, and set the pip param to install package-b, and set the env vars it still fails. It looks like it will successfully install package-b however it fails when trying to install the wheel. I imagine this would be because the pip install is after the py_modules installs? Finally, when I do not pass a wheel at all, and use the pip param to install package-a directly with the pip env vars it correctly works. Unfortunately I can only rely on passing source distributions and wheels via py_modules in my application so setting pip installs isn’t something we can do.


Do you mind filing a github enhancement ticket?

@wizrds Once you have filed an enhancement ticket as suggested by @jjyao, we can close this issue.

I have already filed one [Core] Private dependencies and `py_modules` · Issue #35559 · ray-project/ray · GitHub