AttributeError: module 'pygloo.rendezvous' has no attribute 'CustomStore'

Ray Libraries (Data, Train, Tune, Serve) Ray Train
1

When Running

collective.init_collective_group(
           world_size=train.world_size(), 
           rank=train.local_rank(), 
           backend="gloo"
		   )

I get

AttributeError: module 'pygloo.rendezvous' has no attribute 'CustomStore'

How could this be?

2

Hi, could you provide more info or provide us with a repro script?

cc: @matthewdeng Is this a ray train question?

3

Hey @Huaiwei_Sun! I don’t have access to that code anymore but I was using Ray 1.13.

4

Looks like CustomStore was added to pygloo a few months ago, but was not officially released.

In the meantime, if you run into this issue again you may workaround this by building from source.