I am trying to parallelize processes using Ray in a docker container.
from ray.util.multiprocessing import Pool with Pool(self.n_cores) as pool: pool.starmap(func=func, iterable=list_of_parameters)
while it is perfectly working locally, when it gets run in the docker container, the following error occurs:
✗ failed cod_simulation-run Job ‘cod_simulation-run’ failed: | AttributeError: ‘StreamCapture’ object has no attribute ‘fileno’ Write “details cod_simulation-run” to inspect the error. Processed 1 jobs (1 failed).
I was previously performing the same thing with python multiprocessing:
import multiprocessing as mp with mp.Pool(self.n_cores) as pool: pool.starmap(func=func, iterable=list_of_parameters)
and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.
FROM python:3.9 WORKDIR /project_name COPY requirements.txt . RUN pip install -r requirements.txt COPY . . RUN find . ENV DISABLE_CONTRACTS=1 RUN pipdeptree RUN python setup.py develop --no-deps RUN cod-demo --help CMD ["cod-demo"]
This is my DockerFile and I am installing ray as a requirement.
Ray was added without specifications, requirements.txt:
(...) ray (...)
Thanks in advance for any suggestion