Ray tune not outputting events file for tensorboard

1

I am running ray tune with tensorflow 2.0. Ray tune is not outputting the events file that can be used for tensorboard. I have install ray by pip install ray[default]. Could you please tell me how can I make ray tune output the events file? Thanks

2

Hey @Arif_Jahangir Tensorboard logging should be enabled by default. What version of Ray do you have? Do you have tensorboardX installed? Also make sure not to have TUNE_DISABLE_AUTO_CALLBACK_LOGGERS environment variable set

3

I have tensorflow version 1.15.5 and ray version ‘1.5.2’. When I use trainner it is outputting events file that can be used with tensorboard but when I use ray tune it does not output the events file for tensorboardx. I am using windows 10

4

Sorry what version of tensorboardX, not tensorflow, do you have?

Just making sure that you have tensorboardX installed. In general you need to use pip install ray[tune] in order to install the additional dependencies needed for Tune.

5

1.15.0 tensorboard version

6

I have seen this article

Could that be the problem?

7

Yeah, possibly. Could you also follow the steps in that issue and share your stdout?

8

My output is as follows. Everything is running but it is not outputting events file for tensorboard

(pid=5116) 2021-08-16 13:36:15,160 INFO trainer.py:706 – Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
(pid=5116) 2021-08-16 13:36:15,160 WARNING ppo.py:135 – train_batch_size (4000) cannot be achieved with your other settings (num_workers=3 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting rollout_fragment_length to 1333.
(pid=5116) 2021-08-16 13:36:15,161 INFO trainer.py:718 – Current log_level is WARN. For more information, set ‘log_level’: ‘INFO’ / ‘DEBUG’ or use the -v and -vv flags.
(pid=5116) 2021-08-16 13:37:06,648 INFO trainable.py:106 – Trainable.setup took 51.918 seconds. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
(pid=5116) 2021-08-16 13:37:06,654 WARNING util.py:55 – Install gputil for GPU system monitoring.
9

Is this all that’s being printed out? Would you be able to send the entirety of your stdout? You can redirect all of this to a file to make it easier by following the steps here: User Guide & Configuring Tune — Ray v2.0.0.dev0