I am using two workers in the ScalingConfig
and want to log wandb metrics during training at rank 0 worker as default setting in setup_wandb
which turns on rank_zero_only. But got this error
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Could there be multiple zero workers?
I even do something like this but still failed
worker_rank = ray.train.get_context().get_world_rank()
if worker_rank == 0:
wandb_run = setup_wandb(...)
wandb_run.log(...)