Why does the Ray job driver process contain autoscaler logs even when autoscaling is disabled?

Ray Clusters Kubernetes
1

I have a KubeRay cluster that does not have autoscaling enabled. I have not set enableInTreeAutoscaling or autoscalerOptions in my RayCluster resource. However, I still see below autoscaler related logs when I do ray job logs.


(autoscaler +27m11s) Removing 2 nodes of type gpu-workers (max number of worker nodes reached).
(autoscaler +27m11s) Resized to 40 CPUs, 8 GPUs.
(autoscaler +27m11s) No available node types can fulfill resource requests {'bundle_group_c584a1a9a19683f36866fd89345c06000000': 0.001}*1, {'bundle_group_c4d493cf10b08e29e1fb554a43d206000000': 0.001}*1, {'bundle_group_b61c9d415ea2ca4ad5ad7220c98106000000': 0.001}*1, {'bundle_group_b20808cb62b252583ddfc0a1748d06000000': 0.001}*1, {'bundle_group_aea09e12d734ba290a924d3f293306000000': 0.001}*1, {'bundle_group_a1e24741314f7e6e3a82eb44d1e806000000': 0.001}*1, {'bundle_group_89b0f4dc412bc2f0c20d1c5ace4206000000': 0.001}*1, {'bundle_group_892ae594b0f5dbc6d475d2b20fc206000000': 0.001}*1, {'bundle_group_84c6675e1ec18355aad154a78bc306000000': 0.001}*1, {'bundle_group_8275120451a23f0cb64880eda98d06000000': 0.001}*1, {'bundle_group_72dcb4dc669c7eaecb4ff5c1c6be06000000': 0.001}*1, {'bundle_group_70c8405112008fedcc75ebaa8d8106000000': 0.001}*1, {'bundle_group_709aba1a5733250d502da131efa906000000': 0.001}*1, {'bundle_group_67036d634c7d8c45a2c8865c207606000000': 0.001}*1, {'bundle_group_6526a39b82c27fd541844628c76406000000': 0.001}*1, {'bundle_group_61c63fb66e259c51ea4d44315dcb06000000': 0.001}*1, {'bundle_group_5e83238fe156d8ac8a496739955706000000': 0.001}*1, {'bundle_group_5bf65ba695ecfb5df7dd1780c8e606000000': 0.001}*1, {'bundle_group_4fecadb7f13c158027ba4849064706000000': 0.001}*1, {'bundle_group_4f70c8dfc60b0bb86be192ece10e06000000': 0.001}*1, {'bundle_group_4e5740196efbcf3729ad91f5f3bc06000000': 0.001}*1, {'bundle_group_3f437b01ce8d7fad4ff2cbcb605906000000': 0.001}*1, {'bundle_group_39ad93dbc776dbb4f24cd9ddc71906000000': 0.001}*1, {'bundle_group_365d6b7aa2817d11ad85f17788e106000000': 0.001}*1, {'bundle_group_2f256386deced7e36a2bd694919306000000': 0.001}*1, {'bundle_group_1cbdb01570dc8a24557682295ac706000000': 0.001}*1, {'bundle_group_13e32dbe688e15f3708561069dc506000000': 0.001}*1, {'bundle_group_123cd655ca00e07259cdccae973006000000': 0.001}*1, {'bundle_group_10a4f32af4e913695a6d5aa1fe4606000000': 0.001}*1. Add suitable node types to this cluster to resolve this issue.

Is there a way to turn them off? It’s kinda polluting the driver logs right now.

2

Ray emits autoscaler-related logs (such as node removals and resource request issues) to the driver by default, even if autoscaling is not explicitly enabled in your RayCluster resource. This is because the autoscaler component is always running in Ray clusters, and its event logging is not gated by the enableInTreeAutoscaling or autoscalerOptions fields. There is currently no documented way to fully disable these autoscaler log messages from appearing in the driver logs, as the logging behavior is hardcoded and not controlled by a user-facing configuration or environment variable in the provided data sources.

If you want to reduce log verbosity, you may consider filtering logs at the application level, but Ray itself does not provide a built-in option to suppress these autoscaler messages in the driver logs as of the latest documentation and codebase. For more details, see the discussion in Ray GitHub PR #49025 and the Ray autoscaler event logger code.

Would you like more detail on how these logs are generated or possible workarounds?

Sources:

Hint: Mention @RunLLM in the post for followups.