How to disable `object_store_memory` logging?

Ray Libraries (Data, Train, Tune, Serve) Ray Train
1

We’re using Ray + PyTorch Lightning and our runs are flooded with object_store_memory messages. How can it be disabled/filtered out?

[2m[36m(pid=6333, ip=240.120.38.131) [0mRunning: 0/199.0 CPU, 0/16.0 GPU, 133.0GB/159.8GB object_store_memory:  73%|███████▎  | 1215/1654 [03:52<01:09,  6.34it/s]

I’ve tried the following without success:

ray_train_logger = logging.getLogger("ray.train")
ray_train_logger.setLevel(logging.ERROR)

def filter_out_object_store_memory(record):
    return "object_store_memory:" not in record.getMessage()

ray_train_logger.addFilter(filter_out_object_store_memory)
2

Hi!

The log statements you are seeing is actually a Ray Data progress bar, not a Ray Train progress bar. Typically this is useful to know, and the progress bar on the driver should actually update in place i.e without duplication. Are you running on a Jupyter notebook? If so, make sure to install ipywidgets

Even beyond that, sometimes prints coming from user code (i.e within user defined functions), or with Ray Train where computation and data processing overlap, the progress bar logs can get annoying. If you want to turn it off, you can do ray.data.DataContext.get_current().enable_progress_bars = False on the driver at initialization time.

Refer to this guide here for more details: Monitoring Your Workload — Ray 2.40.0

3

Thanks for the update @SumanthRH.

Are you running on a Jupyter notebook

We’re submitting jobs with ray submit. The progress bar can be useful but in our case there were tens of thousands of log entries in GCP logging.

We ended using the following when building the base image but great to know about get_current().enable_progress_bars = False:

ENV RAY_DATA_VERBOSE_PROGRESS=0
ENV RAY_DATA_DISABLE_PROGRESS_BARS=1
1 Like