Working_dir files not copied for Tune jobs?

Ray Tune
1

RuntimeEnv(working_dir…) provides the great option to copy necessary files into my working directoy.

However when using a Tuner and Trials these files are not present in the trials local path.
Is that intentional, is that a bug?

The very very ugly way to retrieve the files is to do in my Trainable

ctx = ray.get_runtime_context()
dir_name = ctx.runtime_env["working_dir"].split("/")[-1].split(".zip")[0]  # gcs://file_uuid.zip
working_dir_files = (Path(self._storage.trial_working_directory).parent.parent.parent.parent.parent / "runtime_resources" / "working_dir_files").glob(dir_name + "/" + "*")
# copy into local_path

That can’t be right that I need the runtime context and possibly also the session dir. And I am not 100% sure if this even works on remotes.
Is that really the only way to get the working_dir files in a tune context?

At least as a workaround, could I download it via the gcs address? And how would I do that?

2

It seems likely that when using Ray Tune’s Tuner with a runtime_env(working_dir=…), the files from working_dir are not automatically copied into each trial’s local directory; instead, Ray sets the working directory for the process, but the files themselves are not duplicated into the trial’s log directory. This appears to be intentional: by default, Ray changes the working directory of each worker to a trial-specific directory, and the working_dir contents are available relative to that, but not copied into the trial directory itself. The recommended way to access files from working_dir is to use absolute or relative paths from the working directory, not by copying them into the trial directory. If you need to save trial-specific outputs, you should use tune.get_context().get_trial_dir() for writing, but for reading, access files as they exist in the working_dir, not by copying them manually. The workaround you describe is not the intended usage and may not be robust across environments. Downloading directly from the GCS address is not documented as a standard approach for this use case.

For more detail, see the discussion and code examples in the Ray Tune FAQ, which explains the working directory behavior and how to access files in Tune trials. The test_tuner.py file also demonstrates that the working directory is changed for each trial, and files should be accessed from the original working directory if needed, not by copying them into the trial directory.

Would you like a step-by-step breakdown or code example for accessing files in this context?

Sources:

Hint: Mention @RunLLM in the post for followups.

3

Ok, the answer is to look in os.environ["TUNE_ORIG_WORKING_DIR"].

I remember this path to be the dir where I execute my script. But when I use working_dir in the runtime env it points to the dir in the session path.

A doc update would be nice and having access to both features working_dir without a modified os.environ["TUNE_ORIG_WORKING_DIR"].