Working_dir files not copied for Tune jobs?

Ray Tune
1

RuntimeEnv(working_dir…) provides the great option to copy necessary files into my working directoy.

However when using a Tuner and Trials these files are not present in the trials local path.
Is that intentional, is that a bug?

The very very ugly way to retrieve the files is to do in my Trainable

ctx = ray.get_runtime_context()
dir_name = ctx.runtime_env["working_dir"].split("/")[-1].split(".zip")[0]  # gcs://file_uuid.zip
working_dir_files = (Path(self._storage.trial_working_directory).parent.parent.parent.parent.parent / "runtime_resources" / "working_dir_files").glob(dir_name + "/" + "*")
# copy into local_path

That can’t be right that I need the runtime context and possibly also the session dir. And I am not 100% sure if this even works on remotes.
Is that really the only way to get the working_dir files in a tune context?

At least as a workaround, could I download it via the gcs address? And how would I do that?

2

It seems likely that when using Ray Tune’s Tuner with a runtime_env(working_dir=…), the files from working_dir are not automatically copied into each trial’s local directory; instead, Ray sets the working directory for the process, but the files themselves are not duplicated into the trial’s log directory. This appears to be intentional: by default, Ray changes the working directory of each worker to a trial-specific directory, and the working_dir contents are available relative to that, but not copied into the trial directory itself. The recommended way to access files from working_dir is to use absolute or relative paths from the working directory, not by copying them into the trial directory. If you need to save trial-specific outputs, you should use tune.get_context().get_trial_dir() for writing, but for reading, access files as they exist in the working_dir, not by copying them manually. The workaround you describe is not the intended usage and may not be robust across environments. Downloading directly from the GCS address is not documented as a standard approach for this use case.

For more detail, see the discussion and code examples in the Ray Tune FAQ, which explains the working directory behavior and how to access files in Tune trials. The test_tuner.py file also demonstrates that the working directory is changed for each trial, and files should be accessed from the original working directory if needed, not by copying them into the trial directory.

Would you like a step-by-step breakdown or code example for accessing files in this context?

Sources:

Hint: Mention @RunLLM in the post for followups.

3

Ok, the answer is to look in os.environ["TUNE_ORIG_WORKING_DIR"].

I remember this path to be the dir where I execute my script. But when I use working_dir in the runtime env it points to the dir in the session path.

A doc update would be nice and having access to both features working_dir without a modified os.environ["TUNE_ORIG_WORKING_DIR"].

4

@RunLLM a step-by-step breakdown or code example for accessing files in this context, without coping the workdir into the trial dir

5

To access files from your runtime_env(working_dir=…) in a Ray Tune trial, you do not need to copy them into the trial directory. Instead, you should reference them using paths relative to the current working directory, which is set to the trial’s working directory by default. If you want to access files from the original working directory, you can set the environment variable RAY_CHDIR_TO_TRIAL_DIR=0, so the working directory does not change, and then use absolute or relative paths as needed. For saving trial-specific outputs, use tune.get_context().get_trial_dir().

Here is a code example based on the Ray Tune FAQ:

import os
from pathlib import Path
from ray import tune

def train_func(config):
    # Read from a file in the working_dir
    print(open("./read.txt").read())

    # The working directory shouldn't have changed from the original
    assert os.getcwd() == os.environ["TUNE_ORIG_WORKING_DIR"]

    # Write to the Tune trial directory, not the shared working dir
    tune_trial_dir = Path(tune.get_context().get_trial_dir())
    with open(tune_trial_dir / "write.txt", "w") as f:
        f.write("trial saved artifact")

os.environ["RAY_CHDIR_TO_TRIAL_DIR"] = "0"
tuner = tune.Tuner(train_func)
tuner.fit()

This approach ensures you can access files from the working_dir without copying them into the trial directory. For more details, see the Ray Tune FAQ.

Sources:

Hint: Mention @RunLLM in the post for followups.