# Trying to set up external RL environment and having trouble

**URL:** <https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265>\
**Category:** RLlib\
**Created:** [August 16, 2021, 7:07pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265 "2021-08-16T19:07:42Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [August 16, 2021, 7:07pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/1 "2021-08-16T19:07:42Z")

</div>

Hi,

My thesis project involves using an RL policy to manage a hyperparameter tuning setup. This is kind of in reverse to what people usually do, which is use Ray Tune to tune RLLib hyperparameters. At this point I’m quite familiar with Ray Tune, but I’m having trouble figuring out how to integrate RLLib. I’ve spent a long time reading the documentation and some source code but I’m still very confused.

What I’ve tried to do already is implement a Ray Tune `Scheduler` which takes the policy, action space, and observation space (in this case the action space is the set of hyperparameter values) and works similar to the implementations of PBT or PB2. However, it seems like that isn’t how I should be doing this.

Now, I’m trying to implement an `ExternalEnv` which contains the state-action space, where the `run()` method would contain all of my training/tuning code, and then the environment is passed into the `Scheduler`, which will internally call `start_episode()`, `get_action()`, etc. However I’m confused with where the Policy and Trainer fit into this. I’m assuming the environment is passed into the Trainer, but what about the Policy?

I have a dummy Policy implemented which just changes the hyperparameters based on simple logic but I don’t know where to fit it in.

I’m very confused and if someone could help me out that would be great.

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [August 16, 2021, 7:27pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/2 "2021-08-16T19:27:20Z")

</div>

Just to give some extra info, here’s a pseudocode overview of what I’m trying to do:

```python
# Ray Tune code

class RLScheduler(FIFOScheduler):
    def __init__ (env: TuningEnv, ...):
        ...

    def on_trial_result(self, trial, result, ...):
        action = self.env.get_action(result)
        self._assign_new_hparams(trial, action)

    ...

```

```python
# RLLib code
class TuningEnv(ExternalEnv):
    MAX_CONCURRENT = ...

    def __init__ (self, config, experiment, ...):
        action_space = spaces.Dict(...)
        observation_space = spaces.Dict(...)
        self._experiment = experiment
        # policy????

        super(). __init__ (action_space=action_space, observation_space=observation_space,
                         max_concurrent=self.MAX_CONCURRENT)

    def run(self):
        run_args = self._experiment.run_args()
        tune.run(**asdict(run_args))

```

Also, if I want to do multiple concurrent tuning trials, would I want to use a multi-agent env?

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [August 29, 2021, 1:52pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/3 "2021-08-29T13:52:26Z")

</div>

I’m working on a project that is sort of similar to this, although I didn’t end up using the ExternalEnv API so I’m not an expert. The way to implement this if you want to use an ExternalEnv is to use a PolicyClient and PolicyServerInput. This ([RLlib Environments — Ray v1.6.0](https://docs.ray.io/en/latest/rllib-env.html#external-agents-and-applications)) points to a simple example that uses them.

(The server script: [ray/cartpole\_server.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/rllib/examples/serving/cartpole_server.py) and the client script: [ray/cartpole\_client.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/rllib/examples/serving/cartpole_client.py))

The basic idea here is that you need to run the two scripts at the same time, which in your case translates to having to run a script that trains your RL agent, and a script that runs your tune experiments for which you want to optimize the hyperparams. The server script from the example can mostly be left as-is, the client script is where you need to make most of the changes.

You want to make the client script run your hypertuning code, and you’ll want to make a custom scheduler like you said. The PolicyClient needs to be wrapped by that scheduler, so you can report the hypertuning results to the server script, and also so you can easily poll your agent for actions (i.e. new hyperparams to train on)

There are quite a few moving parts in this setup so I might have missed something, but I hope that helps. If not, feel free post a follow up 🙂

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [August 29, 2021, 6:44pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/4 "2021-08-29T18:44:58Z")

</div>

@import-antigravity the docs are a little confusing, it doesn’t look like this approach uses the ExternalEnv. It should work just fine though.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [August 31, 2021, 6:59am UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/5 "2021-08-31T06:59:18Z")

</div>

Hey @import-antigravity , thanks for the question and @RickDW , thanks for your answer 🙂 This is great. Yeah, you don’t have to explicitly use ExternalEnv. Your env will be wrapped as one automatically. Just using the PolicyClient on the client side and setting up your input on your server side will be enough. RLlib automatically generates a dummy-RandomEnv on the server side to fix the observation and action spaces (which you have to define in your config).

Basically, this is the important bit here (server side):

```auto
    # `InputReader` generator (returns None if no input reader is needed on
    # the respective worker).
    def _input(ioctx):
        # We are remote worker or we are local worker with num_workers=0:
        # Create a PolicyServerInput.
        if ioctx.worker_index > 0 or ioctx.worker.num_workers == 0:
            return PolicyServerInput(
                ioctx, SERVER_ADDRESS, SERVER_BASE_PORT + ioctx.worker_index -
                (1 if ioctx.worker_index > 0 else 0))
        # No InputReader (PolicyServerInput) needed.
        else:
            return None

    # Trainer config. Note that this config is sent to the client only in case
    # the client needs to create its own policy copy for local inference.
    config = {
        # Indicate that the Trainer we setup here doesn't need an actual env.
        # Allow spaces to be determined by user (see below).
        "env": None,

        # TODO: (sven) make these settings unnecessary and get the information
        # about the env spaces from the client.
        "observation_space": gym.spaces.Box(
            float("-inf"), float("inf"), (4, )),
        "action_space": gym.spaces.Discrete(2),

        # Use the `PolicyServerInput` to generate experiences.
        "input": _input,
        # Use n worker processes to listen on different ports.
        "num_workers": args.num_workers,
        # Disable OPE, since the rollouts are coming from online clients.
        "input_evaluation": [],
        # Create a "chatty" client/server or not.
        "callbacks": MyCallbacks if args.chatty_callbacks else None,
    }

```

The example scripts are located in `rllib/examples/serving/cartpole_client|server.py`.

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [September 1, 2021, 10:21pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/7 "2021-09-01T22:21:27Z")

</div>

Thanks for posting this! Sorry for not responding sooner, I had sort of given up hope that someone would respond 😅

Is it required to do the server-client setup? Based on this figure from the documentation, it seems like what I would want to do is the second image, because the “environment” is itself a Ray job:

 ![Screen Shot 2021-09-01 at 6.20.47 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/1/10ac90303229927fc98190fd9bbac46341801942.png)

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [September 2, 2021, 12:58pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/8 "2021-09-02T12:58:25Z")

</div>

I’m honestly not sure, maybe @sven1977 can answer your question. My intuition tells me that there isn’t a big difference between using an externalenv explicitly and using a policy client / policy server input approach. While your tune process is probably run on the same ray cluster as your RL training process, it looks like all communication between the two is neatly handled by the policy client. The only thing that might be interesting for you to look into is the requested resources for the RL and hypertuning processes, so they can run at the same time.

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [September 3, 2021, 3:55pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/9 "2021-09-03T15:55:03Z")

</div>

Yeah, it looks like this is probably easier because I can also leave my hyperparameter tuning code as-is and simply write the RL training code for the server side. How would I run both at once? Would I treat it like a regular Ray actor and do `policy_server().remote()` and then run the client code?

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [September 3, 2021, 8:19pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/10 "2021-09-03T20:19:23Z")

</div>

No you don’t need to execute any calls remotely, what you need to do is run the client and server scripts at the same time. You might want to give the example scripts another look, everything is in there since you can reuse most of the code as-is 🙂

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [September 5, 2021, 5:28pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/11 "2021-09-05T17:28:21Z")

</div>

If you run them at the same time, like with `python server.py & python client.py` then how will they connect to the same ray runtime? I looked at the example scripts and couldn’t tell how that was supposed to work

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [September 14, 2021, 9:58pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/12 "2021-09-14T21:58:55Z")

</div>

I still don’t understand what you mean by “run them at the same time,” could you clarify please? Thanks

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [September 20, 2021, 12:24pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/13 "2021-09-20T12:24:31Z")

</div>

Hey, sorry for taking a while to reply. All you need to do is run the two _separate_ python scripts. I’m fairly certain that is the only thing you need to do, but the best thing to do is to just try this and see what happens 😄

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [September 20, 2021, 3:56pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/14 "2021-09-20T15:56:52Z")

</div>

No worries. I guess the confusion is that I’m trying to run both of these on the same SLURM script, so you can’t run two blocking python scripts at the same time. Is there a way to like, run the server script on the head node and run the client scripts on the worker nodes?

---

<div class="post-metadata">

**Author:** ![RickDW](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@RickDW](https://discuss.ray.io/u/RickDW)\
**Post date:** [September 21, 2021, 8:29pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/15 "2021-09-21T20:29:46Z")

</div>

I only know about SLURM from reading some documentation, I don’t have any experience with it at all. That being said, I remember reading somewhere that you can run different scripts on different SLURM nodes in a job allocation. So it should be possible, but you should ask someone else if you need help with it.

---

<div class="post-metadata">

**Author:** ![rusu24edward](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rusu24edward/32/333_2.png) [@rusu24edward](https://discuss.ray.io/u/rusu24edward)\
**Post date:** [September 28, 2021, 6:34pm UTC](https://discuss.ray.io/t/trying-to-set-up-external-rl-environment-and-having-trouble/3265/16 "2021-09-28T18:34:36Z")

</div>

@import-antigravity Take a look at my post [here](https://discuss.ray.io/t/ray-slurm-with-rllib/3073), which includes an example slurm script. You can use the same slurm script to launch the trainer and the worker nodes. Please read the full post because the original slurm script is not correct and I had to make some changes based on the discussion I had with Sven.