We are using custom multi-agent hierarchical environment to train the PPO policy using rllib, there is dependency over a external C++ environment to get some information to update the observation space, we used gRCP to establish communication between python and c++ environments. The training is working fine we are able to train a PPO policy, now we want to do inference in c++ environment. I have following quires
- Can we export the learned policy to c++ environment which do not have a dependency over ray/rllib? OR
- Do we have c++ support for rllib to load policy from the checkpoint we got from training and perform the inference?
thanks for stating this question and welcome to the discussion board! My first question is:
Is deploying the RLlib policy via Ray serve und and then calling the server not an option for you?
We are trying to avoid the client server thing because it is taking too much time which we can not afford, we are looking for a faster option so we wanted to move inference directly to c++ environment.
I guess that the experimental C++ API of Ray is not yet at a point where it can handle RLlib. @sven1977 do you have any information about this?
There might be a possibility to use the TensorFlow checkpoint to load the model in C++ and use simply the model with the trained weights and rebuild the rest of the agent in C++.
Otherwise, you might have to fall back to Ray Serve.
@Lars_Simon_Zehnder it would be great if can you point me to some example which shows how to load the trained policy without rllib dependency for inference.
@Siddharth_Jain, I would take a look at the example for DQN on
You need actually only the trained model and can replicate everything else (action choice, etc.) in C++ code then. Train in RLlib and apply in your custom application.
This seems like the best answer for now.
Out of curiosity though, is there a reason why you also can’t write python bindings for your environment. We don’t have c++ support for rllib at the moment.