Local caches of ray.train working_dirs

Ray Libraries (Data, Train, Tune, Serve)
1

Starting from Ray.2.20, ray.train will produce a local caches of the working_dirs. Is there a way to disable that cache?

Thanks

2

Hi @wxie2013, what do you mean by the working dirs cache?

Are you talking about the working_dirs folder being created in this local staging directory under /tmp/ray/? Configuring Persistent Storage — Ray 2.21.0

1 Like
3

Thanks @justinvyu, that’s what I’m looking for.

4

@wxie2013 There should be nothing in this directory as long as you don’t save files to relative directories in your training function (it is set as the current working directory).

There’s no way to disable the creation of these directories at the moment. Are you running into any problems due to these directories being created?

5

Hi @justinvyu , I indeed have output files from my train function. Those are the weights and other parameters of the model needed for consecutive training. Currently I set RAY_CHDIR_TO_TRIAL_DIR=0 to keep those outputs location unchanged to move on.