Running test evaluation with policy server input

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

hi! just asking for a quick head check here on a workaround i’m using.

i have been building off of the cartpole server example for an integration between RLlib and MATSim, a Java-based traffic simulator. i am finally getting acceptable training results for my problem. i opted out of running any inline tests/evaluations during training, and would now like to go back and manually run tests against saved checkpoints.

the cartpole server example does have a command line argument “–as-test” which i’m assuming was intended to mock up an example of this. but the argument is unused in that file, so there’s no example of what the original author intended. i dove into the git history of the cartpole file and --as-test first appears in this commit but it’s not wired in there either.

what i’ve come up with to approximate running a test (with the same server setup) is to set exploration to False, load the algorithm with policy_server_input, and then spin wait on the process while i send requests from the client:

# config at this point has _input: policy_server_input

if args.as_test:
  config.update({"explore": False})

algo = get_algorithm_class(args.run)(config=config)

if args.as_test:
  # run until user terminates process
  while True:
    time.sleep(0.2)
    pass

does this get me the behavior i’m expecting for a test evaluation of a given algorithm checkpoint?

thank you for reading!

rob