Trials fail when using with hyperoptSearch

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Trial fails with the below error when used with hyperoptSearch algorithm

2021-06-22 19:46:29,892	ERROR ray_trial_executor.py:534 -- Trial PPO_StockTradingEnv_104bd69e: Unexpected error starting runner.
Traceback (most recent call last):
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\site-packages\ray\tune\ray_trial_executor.py", line 524, in start_trial
    return self._start_trial(trial, checkpoint, train=train)
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\site-packages\ray\tune\ray_trial_executor.py", line 426, in _start_trial
    runner = self._setup_remote_runner(trial)
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\site-packages\ray\tune\ray_trial_executor.py", line 255, in _setup_remote_runner
    trial.init_logdir()
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\site-packages\ray\tune\trial.py", line 412, in init_logdir
    self.local_dir)
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\site-packages\ray\tune\trial.py", line 139, in create_logdir
    os.makedirs(logdir, exist_ok=True)
  File "C:\Users\P2834048\AppData\Local\Continuum\anaconda2\envs\dl\lib\os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: "C:\\Users\\P2834048\\ray_results\\PPO_StockTradingEnv_104bd69e_1__fake_gpus=False,batch_mode=truncate_episodes,callbacks=<class 'ray.rllib.agents.callbacks.DefaultC_2021-06-22_19-46-29"

However only a couple of parameters are being used for tuning purpose.

2

happens similarly when used with optunaSearch.

@rliaw: is there way to mitigate the lengthy folder being created with symbols in it

OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: "C:\\Users\\P2834048\\ray_results\\PPO_StockTradingEnv_104bd69e_1__fake_gpus=False,batch_mode=truncate_episodes,callbacks=<class 'ray.rllib.agents.callbacks.DefaultC_2021-06-22_19-46-29"
3

Hmm. Is there a function in python that helps clean out directory/file names for Windows OS? Let me search for it.

4

I also have this error when using optunaSearch

5

I created a workaround. The error seems to stem from ray\tune\trial.py method _generate_dirname(self), line 757,

I split the string generated_durname by the first invalid character. In my case, it’s always =.

    `return re.sub("[/()]", "_", generated_dirname.split("=")[0])`
6

= should be a valid character in directories, my suspicion is that it’s the < from the <class 'ray.rllib.agents.callbacks.....

Can you post your tune hyperparameter config?

In any case, we are filtering this since [tune] Simplify experiment tag formatting, clean directory names by krfricke · Pull Request #23672 · ray-project/ray · GitHub and the fix will be included in the next release (1.13.0).