When Running
collective.init_collective_group(
world_size=train.world_size(),
rank=train.local_rank(),
backend="gloo"
)
I get
AttributeError: module 'pygloo.rendezvous' has no attribute 'CustomStore'
How could this be?
Hi, could you provide more info or provide us with a repro script?
cc: @matthewdeng Is this a ray train question?
Hey @Huaiwei_Sun! I don’t have access to that code anymore but I was using Ray 1.13.
Looks like CustomStore
was added to pygloo
a few months ago, but was not officially released.
In the meantime, if you run into this issue again you may workaround this by building from source.