Hi I am using xgboost_ray and was wondering if xgboost actors open extra ports to communicate with each other and if so, is there a way to fix the ports on each actor to a specific value beforehand?
For example, this can be done on lightgbm_ray, using either machines or local_list_port arguments.
Also, is there a document showing on how many additional ports are required to use lightgbm/xgboost on Ray? I can guess the number of additional ports per node should be equal to number of actors started on each node but would like to confirm this
@kai Any help on this would be appreciated.
These are nothing different than Ray Core Actors. I think they do communicate to each other through some port but I am not aware of any way of configuring that. cc Core oncall to add some context.
cc @jjyao
@xwjiang2010 Thanks for a quick reply.
In that case, I’d guess I can specify these ports through “worker_port_list” argument when calling “ray start” CLI.
@jjyao Please let me know if this is correct, and that Actors do not open their own new ports for communication.
Actually, both xgboost_ray and lightgbm_ray bypass Ray communication and open their own ports (xgboost does that internally in its Rabit library, while we do it in lightgbm_ray logic for lightgbm, using those parameters you mentioned). I am not sure if xgboost supports setting the ports manually - I’d look in its documentation/make an issue in its repo.
Each worker (actor) needs one free port.
Also, does lightgbm_ray open any more ports other than the ones provided by “machines” or “local_list_port” arguments?
No, only that (and if you do not provide those parameters, it will pick random free ports). This is in addition to ports used by Ray Core.
I’d refer to xgboost documentation in regards to distributed training communication and algorithms (it uses its own custom backend called Rabit, though I think NCCL can also be used nowadays)
2 Likes