Use a remote worker for Evaluation

RLlib
1

Hello,

i am currently coupling ray with a simulation software, that uses a licence for each worker. I noticed, that rllib is not using a remote worker for the evaluation, but generates another worker/environment.

This means I am using an additional licence when doing evaluation, which can lead to failure of the worker and thus the training stops.

Is it possible to use the remote worker for evaluation?

Thanks in advance
Sebastian

2

Hey @SebastianBo1995 , great question! Actually, this should work in 1.4:

config:
    num_workers: n
    create_env_on_driver: True

    evaluation_interval: 0  # no eval worker set!


trainer.train()  # train normally
trainer.evaluate()  # <- this should use the local worker's env to do an evaluation run.
3

Hey @sven1977,

my ray version is:
ray 1.4.0 pypi_0 pypi

and i tried it out today with the exact settings as above.

Rllib still creates a worker to do the evaluation on.

config:
    num_workers: 1
    create_env_on_driver: True

    evaluation_interval: 0  # no eval worker set!

Uses 2 licences.

4

@SebastianBo1995,

You are getting two workers because of your config options. The num_workers :1 will create one worker with an environment in it. The create_env_on_driver: True. Will create a second environment on the driver.You can set the former to 0 or the latter to False.

5

Thanks for the response. Ok, so then there is no way to use for example 16 workers for the training and do the evaluation on them. We will always need an additional “worker”, whether it is the driver or a separate eval worker. Is that right?

1) Set `evaluation_interval` >= 0 to force creating a separate evaluation worker set.
2) Set `create_env_on_driver=True` to force the local (non-eval) worker to have an environment to evaluate on.
6

Hi @SebastianBo1995,

Rllib always creates a seperate worker_sets for the environments that collect experience for training and the environments that perform evaluation between training. You can have as many or as few evaluation workers as you want but they will each require a seperate license in addition to the workers used for training.