I am running a simple modification of the Ray Tune Quick Start example from the docs Tune: Scalable Hyperparameter Tuning — Ray v1.2.0 (taking a uniform distribution for alpha and making it run indefinitely) from ray import tune def objective(step, alpha, beta): return (0.1 + alpha * step / …

Yeah, this is an issue that was also reported on Github: [tune] Logs fill up disk space causing a "No space left on device" error · Issue #15595 · ray-project/ray · GitHub We’ll be taking a look at this soon. I think perhaps consider just setting TUNE_MAX_PENDING_TRIALS_PG=1 for now?

Thanks, that does reduce the log size significantly! However, it also seems to limit the parallel execution of the trials, so I set TUNE_MAX_PENDING_TRIALS_PG to the number of CPUs used (which I also pass to ray.init). How exactly is TUNE_MAX_PENDING_TRIALS_PG related to parallel trial execution? I…

Hi Daniel, TUNE_MAX_PENDING_TRIALS_PG limits the number of Trials currently in the PENDING state. This means that Tune should still be able to leverage the full resources on the cluster, but might take a little bit longer to start these trials. However, in most cases and especially on single machin…

Yeah it should be easy to do it, and I will work on it next week.

Is it the understanding of the maintainers that, per the PR that is linked to being closed, this is an issue that has been fixed on more recent releases and/or on Github? Asking because someone in the group I work with is currently experiencing this issue, much to the misfortune of our SSD’s ongoing…

Hey @Cody_Wild , great to see you here! This should be fixed on 1.4.1 (the latest Ray release). If you cannot upgrade, limiting the env var would be best.

Ray tune log grows extremely large

Ray Tune

rliaw May 8, 2021, 4:47pm 5

BTW @sangcho would it be easy to reduce the amount of logging that is done?

Topic		Replies	Views
Ray Tune run out of disk (/tmp/ray) Ray Tune	3	480	July 2, 2021
[Tune] Ray tune terminates after OSError: [Errno 28] No space left on device Ray Tune	2	2684	May 7, 2021
Ray tune trial log is now showing Ray Tune	1	370	February 27, 2023
RayTune gets stuck after completing all trials Ray Tune	1	715	February 11, 2022
Ray Tune gets stuck for infinity Ray Tune	7	64	May 5, 2025

Ray tune log grows extremely large

Related topics