Alert `Policy_Client`s when `Policy_Server` completes an epoch

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi all,

I’m using the policy_server + policy_client to do some training on remote clients. My question is if there is any way for a client to know if a policy_server either: started, is in the process of, or completed an epoch?

The reason being, the training epoch takes ~25 seconds. During this time, the clients are stuck waiting for new actions and that gap time causes problems that I could handle more gracefully. (Currently I’m using times between received actions to scrap previous environments that may have completed in that ~25 seconds).

So if there is any way, please let me know - or the closest workaround, thanks!

2

Hi @Denys_Ashikhin ,

you might want to take a look into the Handler class of the PolicyServerInput and override the communication (sending to clients also some information about training).

1 Like
3

If I understood correctly: ray/policy_server_input.py at fbaba01efbe0709e12fd6b51ec1a40553feb232a · ray-project/ray · GitHub

I can just modify it to:

elif command == Commands.GET_ACTION:
                assert inference_thread.is_alive()
                response["action"] = child_rollout_worker.env.get_action(
                    args["episode_id"], args["observation"], {"custom object here": 1234}
                )

And then on policy_client:

action, customObject = client.get_action(episode_id=episode_id, observation=obs)

And then I get my customObject which would have whatever it is I need?