Requirements.txt support with Job API

Ray Clusters
1

Hi,

In the documentation for version 1.11.1 (Ray Job Submission: Going from your laptop to production — Ray 1.11.1), you mention that passing in a requirements.txt file is not supported while using the Job API. Is it still the case with later releases? The documentation for the later releases don’t mention such a limitation but the release notes for the later versions don’t mention the restriction being lifted either.

Thanks,
Emre Aydin

2

Hi @emre-aydin,

Yes, passing a requirements.txt file is supported in later Ray releases (starting in 1.12).

3

Hi @shrekris,

Thanks for your answer. It works when I use a local directory as my runtime_env but when I use a ZIP file stored on S3 and a requirements.txt within that ZIP, it fails (complaining about not being able to find the requirements.txt file). Should I create an issue for this on GitHub?

Thanks,
Emre Aydin

4

Hi @emre-aydin , you can pass in a local requirements.txt in the "pip" field of runtime_env, but if there’s just a requirements.txt file in the "working_dir" directory, it won’t automatically get installed. Just to be sure, do you mind pasting your runtime_env?

5

Hi @architkulkarni,

runtime_env={
    "working_dir": "s3://<some-s3-url>.zip",
    "pip": "requirements.txt",
}

I expected that Ray would unpack the ZIP archive located at the S3 URL and use the requirementst.txt that it contained but it fails with file not found errors.

P.S.: I also tried ./requirements.txt.

6

Hi @emre-aydin, thanks for the clarification. Unfortunately the behavior you expected isn’t supported; currently the working_dir is only unpacked and chdir’ed to at Ray worker startup time, not before the pip setup. If you would like to see this as a feature, if you could file an enhancement request, it would be appreciated! Sign in to GitHub · GitHub

7

Thanks @architkulkarni , here it is: [Job Submission] requirements.txt support for runtime_env stored on S3 · Issue #26155 · ray-project/ray · GitHub

8

Great, thanks for creating the issue!