Running PolicyServer code asynchronously in background

RLlib
1

I’m trying to implement something similar to the examples mentioned here with a PolicyClient and PolicyServer but I’d like to start the client programmatically in the background. I don’t think I can just put them in separate scripts and run them completely separately because the server needs information about the state-action space at instantiation time.

I’ve tried doing something like this:

@ray.remote
class RLServer:
    def __init__(self, args):
        # set things up
        self.trainer = SACTrainer(...)

    async def train_loop(self):
        for _ in range(self.stop_iters):
            results = self.trainer.train()
            ...

# Main code
class Experiment:
    def setup(self):
        ...
        self.server = RLServer(...)

    def run(self):
        self.server.train_loop.remote()
        # Start PolicyClient...

Can anyone help me figure out what to do? If I just do .remote() then the server never starts, if I do ray.get() then the server starts but it blocks.

2

Hey @import-antigravity , thanks for the question. I think it would be much easier to use the PolicyClient|Server classes and APIs (+ examples). You can give the server space hints via your (SAC) config, e.g.:

observation_space: Box(-1.0, 1.0, (3, ), float32)
action_space: Discrete(5)

The Server - by default - won’t create an env, but will know via your space-hints, which policy to build (and learn).

3

That’s what I’m doing

4

@import-antigravity Have you tried using the API that Sven suggested? You can find an example here: