Ray behavior with deleted log files

saumya-saran · June 7, 2024, 5:01am

How severe does this issue affect your experience of using Ray?

Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I have a long-running instance of Ray on a node with many workers (number of CPUs). Each worker has its own python-core-worker.log file, which have grown quite large in size and are taking ~17GB of disk space.

Based on the documentation, worker logs are not rotated. In order to free up space, I would like to delete some of these files. Would this cause any issues?

Would the log files be recreated as Ray continues to run? What are best-practices for managing these system files?

ruisearch42 · June 11, 2024, 10:22pm

I think worker logs (prefix=worker) are not rotated, but core-worker logs (prefix=python-core-worker) are:

github.com

ray-project/ray/blob/76f5cd4417bf986c7a6d4a6dec213c2bccdaeaf0/python/ray/tests/test_logging.py#L148


      
          session_path = Path(session_dir)
          log_dir_path = session_path / "logs"
          
          log_rotating_component = [
              (ray_constants.PROCESS_TYPE_DASHBOARD, [".log", ".err"]),
              (ray_constants.PROCESS_TYPE_DASHBOARD_AGENT, [".log"]),
              (ray_constants.PROCESS_TYPE_RUNTIME_ENV_AGENT, [".log", ".out", ".err"]),
              (ray_constants.PROCESS_TYPE_LOG_MONITOR, [".log", ".err"]),
              (ray_constants.PROCESS_TYPE_MONITOR, [".log", ".out", ".err"]),
              (ray_constants.PROCESS_TYPE_PYTHON_CORE_WORKER_DRIVER, [".log"]),
              (ray_constants.PROCESS_TYPE_PYTHON_CORE_WORKER, [".log"]),
              # Below components are not log rotating now.
              (ray_constants.PROCESS_TYPE_RAYLET, [".out", ".err"]),
              (ray_constants.PROCESS_TYPE_GCS_SERVER, [".out", ".err"]),
              (ray_constants.PROCESS_TYPE_WORKER, [".out", ".err"]),
          ]
          
          # Run the basic workload.
          @ray.remote
          def f():
              for i in range(10):

Have you configured log rotation properly? Configuring Logging — Ray 2.24.0

Topic		Replies	Views
Why are worker logs not formatted using the ray logger Ray Core	1	311	August 21, 2023
Ray Writes to Disk Eventually Running Out of Space and Causing Eviction of the Node Ray Core	3	353	December 21, 2021
Out of disk space with fallocate Ray Core	1	478	September 8, 2022
Confused with coreworker and worker Ray Core	3	587	August 7, 2022
Diagnosing unexpected memory management behavior Ray Core	1	248	August 21, 2023

Ray behavior with deleted log files

Related Topics