Hello!
I have an issue where the working dir specified in ray job submit --working-dir
is not transmitted to the cluster. Interestingly, if using a subdirectory of the same directory instead, it is transmitted successfully.
So, this works fine (note the output contains the working dir printed by pwd
and a filename test.py
listed by ls
:
ray job submit --working-dir /code/ray-test -- bash -c 'pwd; ls'
...
Tailing logs until the job exits (disable with --no-wait):
2025-01-28 19:10:16,287 INFO job_manager.py:530 -- Runtime env is setting up.
/tmp/ray/session_2025-01-28_18-55-45_577283_70/runtime_resources/working_dir_files/_ray_pkg_879177c81ec48c5a
test.py
------------------------------------------
Job 'raysubmit_uHi6hwrMLtNn19RB' succeeded
------------------------------------------
But here ls
lists no files:
ray job submit --working-dir /code -- bash -c 'pwd; ls'
...
Tailing logs until the job exits (disable with --no-wait):
2025-01-28 19:12:17,840 INFO job_manager.py:530 -- Runtime env is setting up.
/tmp/ray/session_2025-01-28_18-55-45_577283_70/runtime_resources/working_dir_files/_ray_pkg_3030303030303030
------------------------------------------
Job 'raysubmit_zAduUSWTJVs35DHh' succeeded
------------------------------------------
I should add that I am running ray job submit
from a Docker container on a local machine and /code
is a directory mounted into that container from the local machine.
Anyone knows why /code
won’t sync to the cluster and how to work around this?