Could you ask on the Ray core forum here, how to set up a ray cluster using your local GPU machine as the head node and your CPU servers as the worker nodes? Then you can connect in your RLlib (driver) script to that server via ray.init(...) and RLlib should automatically utilize your CPU servers for doing the sample collection work and your GPU for the training updates.
2 Likes