Does rllib support multi-gpu plus multi-cpu training?

If I have two gpu machines and one CPU machine, how can I use the gpu of two nodes and the cpu of three nodes?
If ppo is used, only one gpu can be used; If use ddppo, the CPU node is not used.

We support Single-Node-Multi-GPU training.
But today, RLlib does not support Multi-GPU-Multi-Node training. We are working on this though.

You can generally make use of heterogeneous clusters with RLlib. For example, you can use one node with a GPU for training and another node with a GPU for sampling.
The different algorithms impose different restrictions here.
For example, as you said, DDPPO will use only GPU nodes and other algorithms will learn on the head node but use all other nodes only for sampling.

In order to get around this, you’ll have to put some effort in and heavily modify the algorithm that you are looking at.