We are using custom multi-agent hierarchical environment to train the PPO policy using rllib, there is dependency over a external C++ environment to get some information to update the observation space, we used gRCP to establish communication between python and c++ environments. The training is working fine we are able to train a PPO policy, now we want to do inference in c++ environment. I have following quires
- Can we export the learned policy to c++ environment which do not have a dependency over ray/rllib? OR
- Do we have c++ support for rllib to load policy from the checkpoint we got from training and perform the inference?
@Siddharth_Jain ,
thanks for stating this question and welcome to the discussion board! My first question is:
Is deploying the RLlib policy via Ray serve und and then calling the server not an option for you?
Hi @Lars_Simon_Zehnder
We are trying to avoid the client server thing because it is taking too much time which we can not afford, we are looking for a faster option so we wanted to move inference directly to c++ environment.
I guess that the experimental C++ API of Ray is not yet at a point where it can handle RLlib. @sven1977 do you have any information about this?
There might be a possibility to use the TensorFlow checkpoint to load the model in C++ and use simply the model with the trained weights and rebuild the rest of the agent in C++.
Otherwise, you might have to fall back to Ray Serve.
@Lars_Simon_Zehnder it would be great if can you point me to some example which shows how to load the trained policy without rllib dependency for inference.
@Siddharth_Jain, I would take a look at the example for DQN on CartpoleV1.
You need actually only the trained model and can replicate everything else (action choice, etc.) in C++ code then. Train in RLlib and apply in your custom application.
This seems like the best answer for now.
Out of curiosity though, is there a reason why you also can’t write python bindings for your environment. We don’t have c++ support for rllib at the moment.