Wandb logger does not work on worker node

Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have a ray cluster with a head node (2 GPUs) and one worker node (1 GPU). Each trial uses 1 GPU. Unfortunately, the wandb logger only works properly if the trial is placed on the head node. If the trial is placed on the worker node, only system metrics and the config are logged to wandb. My custom metrics are not visible in the wandb dashboard. I don’t get any error in the command line output.

The runs are started like this:
tune.run(
trainer,
callbacks=[WandbLoggerCallback()],
resources_per_trial=dict(gpu=1, cpu=1)
)