Configure runtime_env for multiple local packages

Ray Clusters
1

I work on a project that is distributed among multiple git repos.

For example
ClassificationTrainerApp (main git repo not installed)

  • DataTransformerUtils (installed lib)
  • CustomLoggingUtils (installed lib)

Locally I develop using pip install -e on the dependent libs. Is there a way to configure the runtime env to mimic this.

The documentation I have read to configure the runtime_env using work_dir and pymodules seems to assume a monorepo type setup.

2

can you try this?

import ray
import DataTransformerUtils
import CustomLoggingUtils

ray.init(runtime_env={"py_modules": [DataTransformerUtils, CustomLoggingUtils]})

@ray.remote
def test_my_module():
    # No need to import modules inside this function.
    DataTransformerUtils.f()
    CustomLoggingUtils.g()

ray.get(test_my_module.remote())
3

Thanks! This worked!

1 Like
4

We should update our docs here… @jascase901, since firsthand feedback from the user is most important, any ideas on where in our docs we should update this?

5

I think its already here Environment Dependencies — Ray 2.37.0 I just missed it.