Running PolicyServer code asynchronously in background

import-antigravity · September 26, 2021, 5:02pm

I’m trying to implement something similar to the examples mentioned here with a PolicyClient and PolicyServer but I’d like to start the client programmatically in the background. I don’t think I can just put them in separate scripts and run them completely separately because the server needs information about the state-action space at instantiation time.

I’ve tried doing something like this:

@ray.remote
class RLServer:
    def __init__(self, args):
        # set things up
        self.trainer = SACTrainer(...)

    async def train_loop(self):
        for _ in range(self.stop_iters):
            results = self.trainer.train()
            ...

# Main code
class Experiment:
    def setup(self):
        ...
        self.server = RLServer(...)

    def run(self):
        self.server.train_loop.remote()
        # Start PolicyClient...

Can anyone help me figure out what to do? If I just do .remote() then the server never starts, if I do ray.get() then the server starts but it blocks.

sven1977 · September 27, 2021, 7:21am

Hey @import-antigravity , thanks for the question. I think it would be much easier to use the PolicyClient|Server classes and APIs (+ examples). You can give the server space hints via your (SAC) config, e.g.:

observation_space: Box(-1.0, 1.0, (3, ), float32)
action_space: Discrete(5)

The Server - by default - won’t create an env, but will know via your space-hints, which policy to build (and learn).

import-antigravity · September 27, 2021, 1:12pm

That’s what I’m doing

rusu24edward · September 28, 2021, 5:54pm

@import-antigravity Have you tried using the API that Sven suggested? You can find an example here:

github.com

ray-project/ray/blob/master/rllib/examples/serving/cartpole_server.py

#!/usr/bin/env python
"""
Example of running an RLlib policy server, allowing connections from
external environment running clients. The server listens on
(a simple CartPole env
in this case) against an RLlib policy server listening on one or more
HTTP-speaking ports. See `cartpole_client.py` in this same directory for how
to start any number of clients (after this server has been started).

This script will not create any actual env to illustrate that RLlib can
run w/o needing an internalized environment.

Setup:
1) Start this server:
    $ python cartpole_server.py --num-workers --[other options]
      Use --help for help.
2) Run n policy clients:
    See `cartpole_client.py` on how to do this.

The `num-workers` setting will allow you to distribute the incoming feed over n

This file has been truncated. show original

github.com

ray-project/ray/blob/master/rllib/examples/serving/cartpole_client.py

#!/usr/bin/env python
"""
Example of running an external simulator (a simple CartPole env
in this case) against an RLlib policy server listening on one or more
HTTP-speaking port(s). See `cartpole_server.py` in this same directory for
how to start this server.

This script will only create one single env altogether to illustrate
that RLlib can run w/o needing an internalized environment.

Setup:
1) Start the policy server:
    See `cartpole_server.py` on how to do this.
2) Run this client:
    $ python cartpole_client.py --inference-mode=local|remote --[other options]
      Use --help for help.

In "local" inference-mode, the action computations are performed
inside the PolicyClient used in this script w/o sending an HTTP request
to the server. This reduces network communication overhead, but requires

This file has been truncated. show original

Topic		Replies	Views
RLLIB for interacting with live calls from an external system RLlib	1	249	May 9, 2022
Example of ExternalEnv used to implement a REST policy server RLlib	6	526	July 13, 2022
RLlib's PolicyServer and external simulator as client RLlib	15	1721	April 12, 2021
Trainers with policy client/server(s) on a single machine locks at .train(), is multi-thread explicitly needed? RLlib	3	270	July 26, 2022
Alert `Policy_Client`s when `Policy_Server` completes an epoch RLlib	2	222	March 30, 2023

Running PolicyServer code asynchronously in background

Related topics