No Logging Options for Serve Deployment and/or Ray generated Pool Actors

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

Salutations,

I am looking for an option(s) for controlling the logging level for a serve deployment and pool actors (generated under the Ray Pool API) with Python and have not been able to do so, in either the Ray source code or existing documentation/issues. Specifying the logging level at initialization should pass the level to any actors using the encapsulated resources, though this doesn’t actually appear to be the case. A minimum example code for reproduction of the serve deployment is included below (though similar behavior occurs with pool actors as described/exhibited). Any insights would be appreciated; thank you in advanced.

Versions / Dependencies

Windows 10
Ubuntu 18.04
Python 3.8
Ray 1.10.0

Reproduction script

import logging
import ray
from ray import serve
import time
  
ray.init(configure_logging=True, logging_level=logging.ERROR)
  
@serve.deployment(route_prefix="/ModelServer")
class ModelServer:
    def __init__(self): print('Initializing')
    def __call__(self): print('Called')

serve.start()
ModelServer.deploy()
model = ModelServer.get_handle()
ray.get(model.remote())
serve.shutdown()

What happened

 pid=47948) 2022-02-22 15:30:47,910     INFO checkpoint_path.py:16 -- Using RayInternalKVStore for controller checkpoint and recovery.
 pid=47948) 2022-02-22 15:30:48,017     INFO http_state.py:98 -- Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:mQCdAu:SERVE_PROXY_ACTOR-node:127.0.0.1-0' on node 'node:127.0.0.1-0' listening on '127.0.0.1:8000'
2022-02-22 15:30:48,655 INFO api.py:475 -- Started Serve instance in namespace '0d3370fc-79eb-44c2-9487-469ea0e95e50'.
2022-02-22 15:30:48,670 INFO api.py:249 -- Updating deployment 'ModelServer'. component=serve deployment=ModelServer
 pid=2228) INFO:     Started server process [2228]
 pid=47948) 2022-02-22 15:30:48,769     INFO deployment_state.py:920 -- Adding 1 replicas to deployment 'ModelServer'. component=serve deployment=ModelServer
2022-02-22 15:30:49,437 INFO api.py:261 -- Deployment 'ModelServer' is ready at `http://127.0.0.1:8000/ModelServer`. component=serve deployment=ModelServer
 pid=40024) Initializing
 pid=40024) Called
 pid=47948) 2022-02-22 15:30:49,542     INFO deployment_state.py:940 -- Removing 1 replicas from deployment 'ModelServer'. component=serve deployment=ModelServer

What you expected to happen

 pid=40024) Initializing
 pid=40024) Called
2

Hi @Yatagarasu, thanks for reporting this! I agree, I would expect that setting the logging level as you did would hide the INFO logs. @sangcho can I confirm with you that this is how the configure_logging and logging_level APIs are supposed to work?

If so, this looks like a bug in Ray Serve (and possibly also in Ray Pool), and we can make a Github issue for this to track it.

3

Thank you for your reply; now opened as an issue:

1 Like