How severe does this issue affect your experience of using Ray?
High: It blocks me to complete my task.
I have a C++ based simulator running on my local machine. I am communicating with said C++ based simulator through a python client program. The python client program requests information from the simulator, once the client has information it uploads it into a neo4j graph database. I can then create embeddings from said database which I could use for observations.
Here is my question:
Based on the sound of this (C++ simulator which acts as a server that I communicate to via a python client, and I represent the environment through a neo4j graph), which path of environment choice would be best for me?
After browsing RLlib, it seems that I should use an external environment. However, as I understand it the server in the external environment relationship would essentially be what my client (interface to my simulator) is right now. Or maybe I am misunderstanding. Perhaps the relationship would be:
ExternalEnv Server <-> Python Client <-> C++ Simulator Server ?
I appreciate any guidance, assistance, etc.
(note, I can also change the c++ simulator to act as a client, though that would change the way I am currently representing the environment. So if you do suggest that please provide discussion on how to represent my environment to the agent, since it won’t be able to “see” inside of the simulator, only receive information as requested. Which is why I have constructed a graph.)
@Lars_Simon_Zehnder is also dealing with an external C++ based simulator and has already added a question to the [office hours docs] (RLlib Office Hours - Google Docs) for the next office hours. Please consider registering your question and joining the session so that we can pool topics a little bit!
Your understanding is accurate.
For your setup, you can imagine your client acting as a controller, coordinate data flowing between simulator server and RLlib server.
Essentially, your client would get obs from C++ simulator / graph_db, send it to RLlib server, get some actions back, feed the actions back to the C++ server, and do this repeatedly.
Would this work for you?
We ended up using the rllib client / server external environment paradigm. However, we are in a situation where we are attempting to use a ray cluster to distribute computation.
Essentially, this C++ simulator and associated programs can only run on a single machine, so we need to have multiple machines going if we want to have multiple simulators going.
Currently we are trying to figure out the best way to put this C++ simulator and associated programs in a 1:1 mapping paradigm with ray pods, that is one simulator per pod on a ray cluster.
Any advice on how to move forward with this? I’ve been taking a look at other posts and have seen the recommendation for custom ray resources, but I am a bit lost. It’s worth noting that we were able to achieve some scaling using docker containers. 1 docker container per simulator, all communicating back with a central training server. We just need this to work on a ray cluster now. Happy to make this a new thread if need be.