How to increase package size bigger than 250.00MiB for workers?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hello. I’m trying to configure the runtime environment of workers by specifying my Python module, like:

ray.init(runtime_env={"py_modules": ["<my_module>"], "excludes": [ ... ] })

Then, I’ve got an error saying:

ValueError: Package size (319.20MiB) exceeds the maximum size of 250.00MiB. You can exclude large files using the 'excludes' option to the runtime_env.

Is there any way to have a Python module package larger than 250MiB for workers? I excluded unnecessary ones already by using excludes.

Thanks in advance.

2

cc @architkulkarni

I believe in this case, you should use the remote URI (Environment Dependencies — Ray 3.0.0.dev0). But I will also wait for his response

6

Better to write this suggestion to the error message of ValueError above? I created a PR here

1 Like
7

Hi @3rdgenta , unfortunately we don’t have a way to increase the size limit; it’s an internal limitation. Another option for such a large package is to preinstall it on your nodes manually instead of dynamically installing it using runtime_env, especially if your module is not changing frequently.

By the way, is the large size expected for your single module? It seems on the large side so I just wanted to check that there wasn’t some other issue.