Using Ray Multiprocessing on Docker

Ray Core
1

I am trying to parallelize processes using Ray in a docker container.

from ray.util.multiprocessing import Pool

with Pool(self.n_cores) as pool:
   pool.starmap(func=func, iterable=list_of_parameters)

while it is perfectly working locally, when it gets run in the docker container, the following error occurs:

✗ failed cod_simulation-run Job ‘cod_simulation-run’ failed: | AttributeError: ‘StreamCapture’ object has no attribute ‘fileno’ Write “details cod_simulation-run” to inspect the error. Processed 1 jobs (1 failed).

I was previously performing the same thing with python multiprocessing:

import multiprocessing as mp

with mp.Pool(self.n_cores) as pool:
   pool.starmap(func=func, iterable=list_of_parameters)

and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.

FROM python:3.9

WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

RUN find .

ENV DISABLE_CONTRACTS=1

RUN pipdeptree
RUN python setup.py develop --no-deps

RUN cod-demo --help

CMD ["cod-demo"]

This is my DockerFile and I am installing ray as a requirement.
Ray was added without specifications, requirements.txt:

(...)
ray
(...)

Thanks in advance for any suggestion

2

What’s the version of Ray you are using? Also do you mind creating a Github issue with runnable reproduction script’? Issues · ray-project/ray · GitHub