[Tune] Use WandBCallback Logger with the FIFO scheduler and multiple runs in parallel

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

I’m trying to do my own custom logging callback setup using Weights and Biases and basing it on the code integration provided for the WandbCallback. I use the same MultiProcessing setup and it works for me with some schedulers (Hyperband) but it fails on a simple one like FIFO. I can confirm that all parallel trials are logging and have their own subprocesses getting logged. WandB shows all the experiments, but only one of them is receiving active logs after setup.

I essentially have my own custom version of the WandBCallback provided, reusing a lot of the code, and it didn’t work with FIFO when running more than one experiment at once. I have a suspicion that WandbCallback suffers from the same issues but I haven’t had a chance to test it yet as I can’t directly plug that into my code. If anyone has this exact setup working please let me know.

2

Hmm, when you say “running more than 1 experiment at once” do you mean running multiple trials within 1 experiment, or multiple experiments?

3

I mean multiple trials, 1 Ray experiment

4

Hey Philip, sorry for the slow response. FIFO should work with the WandbCallback.

Also, I’m curious why you can’t plug it into your code?