# Ray tune log grows extremely large
**URL:** https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992
**Category:** Ray Tune
**Created:** [April 30, 2021, 1:38pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992 "2021-04-30T13:38:00Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@daniel-pp](https://discuss.ray.io/u/daniel-pp)
#### Post date: [April 30, 2021, 1:38pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/1 "2021-04-30T13:38:00Z")
I am running a simple modification of the Ray Tune Quick Start example from the docs [Tune: Scalable Hyperparameter Tuning — Ray v1.2.0](https://docs.ray.io/en/latest/tune/index.html)
(taking a uniform distribution for alpha and making it run indefinitely)
```
from ray import tune
def objective(step, alpha, beta):
return (0.1 + alpha * step / 100)**(-1) + beta * 0.1
def training_function(config):
# Hyperparameters
alpha, beta = config["alpha"], config["beta"]
for step in range(10):
# Iterative training function - can be any arbitrary training procedure.
intermediate_score = objective(step, alpha, beta)
# Feed the score back back to Tune.
tune.report(mean_loss=intermediate_score)
analysis = tune.run(
training_function, num_samples=-1,
config={
"alpha": tune.uniform(0.001, 0.1),
"beta": tune.choice([1, 2, 3])
})
print("Best config: ", analysis.get_best_config(
metric="mean_loss", mode="min"))
# Get a dataframe for analyzing trial results.
df = analysis.results_df
```
When I let the tuning run, I notice that the log directory in /tmp/ray/session\_latest/logs grows extremely large. In particular, the file **gcs\_server.out grows to ca. 100 MB in one minute** , and contains just a series of the following messages:
```
[2021-04-30 15:07:58,109 I 16431 16431] gcs_placement_group_manager.cc:292: Registering placement group, placement group id = 8c725267c9a2384dbcd107adc450d63c, name = __tune_a640a6de__ 6faa9d03, strategy = 0
[2021-04-30 15:07:58,109 I 16431 16431] gcs_placement_group_manager.cc:296: Finished registering placement group, placement group id = 8c725267c9a2384dbcd107adc450d63c, name = __tune_a640a6de__ 6faa9d03, strategy = 0
[2021-04-30 15:07:58,109 I 16431 16431] gcs_placement_group_scheduler.cc:141: Scheduling placement group __tune_a640a6de__ a25b2634, id: 8d3433fc1f7eb1df3fb8cde4757b0e8a, bundles size = 1
[2021-04-30 15:07:58,109 I 16431 16431] gcs_placement_group_scheduler.cc:150: Failed to schedule placement group __tune_a640a6de__ a25b2634, id: 8d3433fc1f7eb1df3fb8cde4757b0e8a, because no nodes are available.
[2021-04-30 15:07:58,109 I 16431 16431] gcs_placement_group_manager.cc:215: Failed to create placement group __tune_a640a6de__ a25b2634, id: 8d3433fc1f7eb1df3fb8cde4757b0e8a, try again.
```
When running the tuning for a couple of hours, my disk has no space left due to just this one file.
**Is there a way to a) disable the logging or b) fix the problem reported in the logs?**
I’m running the tuning using ray version 1.3.0 from pip on Ubuntu 16.04.
Please let me know what other information I should provide to help reproduce the problem.
---
### Author:  [@rliaw](https://discuss.ray.io/u/rliaw)
#### Post date: [May 2, 2021, 7:08am UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/2 "2021-05-02T07:08:12Z")
Yeah, this is an issue that was also reported on Github: [[tune] Logs fill up disk space causing a "No space left on device" error · Issue #15595 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/15595)
We’ll be taking a look at this soon. I think perhaps consider just setting `TUNE_MAX_PENDING_TRIALS_PG=1` for now?
---
### Author:  [@daniel-pp](https://discuss.ray.io/u/daniel-pp)
#### Post date: [May 3, 2021, 11:53am UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/3 "2021-05-03T11:53:35Z")
Thanks, that does reduce the log size significantly! However, it also seems to limit the parallel execution of the trials, so I set TUNE\_MAX\_PENDING\_TRIALS\_PG to the number of CPUs used (which I also pass to ray.init).
How exactly is TUNE\_MAX\_PENDING\_TRIALS\_PG related to parallel trial execution? I could not understand it from the documentation.
---
### Author:  [@kai](https://discuss.ray.io/u/kai)
#### Post date: [May 3, 2021, 8:45pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/4 "2021-05-03T20:45:31Z")
Hi Daniel,
`TUNE_MAX_PENDING_TRIALS_PG` limits the number of Trials currently in the PENDING state. This means that Tune should still be able to leverage the full resources on the cluster, but might take a little bit longer to start these trials. However, in most cases and especially on single machines this should only be a couple of seconds - does this prevent parallel execution for you completely?
Setting it to the number of available CPUs is definitely a good choice though.
---
### Author:  [@rliaw](https://discuss.ray.io/u/rliaw)
#### Post date: [May 8, 2021, 4:47pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/5 "2021-05-08T16:47:57Z")
BTW @sangcho would it be easy to reduce the amount of logging that is done?
---
### Author:  [@sangcho](https://discuss.ray.io/u/sangcho)
#### Post date: [May 10, 2021, 8:50pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/6 "2021-05-10T20:50:32Z")
Yeah it should be easy to do it, and I will work on it next week.
---
### Author:  [@Cody\_Wild](https://discuss.ray.io/u/Cody_Wild)
#### Post date: [July 7, 2021, 5:28pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/7 "2021-07-07T17:28:27Z")
Is it the understanding of the maintainers that, per the PR that is linked to being closed, this is an issue that has been fixed on more recent releases and/or on Github? Asking because someone in the group I work with is currently experiencing this issue, much to the misfortune of our SSD’s ongoing available space, and I’m wondering whether the current best solution is still to limit `TUNE_MAX_PENDING_TRIALS_PG`
---
### Author:  [@rliaw](https://discuss.ray.io/u/rliaw)
#### Post date: [July 12, 2021, 7:07pm UTC](https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/8 "2021-07-12T19:07:30Z")
Hey @Cody_Wild, great to see you here! This should be fixed on 1.4.1 (the latest Ray release).
If you cannot upgrade, limiting the env var would be best.