How to tell parameter sharing indeed happened in RaySGD?

Using RaySGD writing a model, it is pretty much working (awesome!)

With a draft roughly working, I’d like to double check and make sure the “parallel” part truly happened during the training. In particular, those parameter sharing across parallel training instances indeed happens. Any suggestions?

Thanks a lot!

May you can actually just print the norm of the weights per layer. If these values are the same (or very very close, off by numerical precision), then you should be confident that it is sharing!

1 Like