Trainable.stop did not return in time.Consider making `stop` a faster operation

Ray Core
1

System Info:

  • Ray version:
    1.8.0
  • Python version:
    3.8

Hello, I have been encountered some problems by using Tune, when i set TimeoutStopper. It arise if no error occurs: After a period of time，the set time threshold is reached, then always output the following logs:

2023-04-13 09:02:40,851 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation. 2023-04-13 09:03:40,857 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation. 2023-04-13 09:04:40,867 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation. 2023-04-13 09:06:08,007 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation. 2023-04-13 09:07:08,015 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation. 2023-04-13 09:08:08,022 WARNING ray_trial_executor.py:146 -- Skipping cleanup - trainable.stop did not return in time. Consider making `stop` a faster operation.

tune trainable code :

_analysis = tune.run(
                tune.with_parameters(self._objective,
                                     data=(x_train, x_valid, y_train, y_valid),
                                     eval_metric=self.eval_metric,
                                     haco_config=haco_config
                                     ),
                name=self.name,
                resume=False,
                stop=TimeoutStopper(self.max_run_time),
                metric=metric,
                raise_on_failed_trial=False,
                mode=self.mode,
                search_alg=algo,
                num_samples=self.max_iteration,
                config=None,
                verbose = 2,
                progress_reporter= progress_reporter,
                callbacks=self.callbacks
            )
	
def _objective(self, config, data, eval_metric, haco_config):
    ...
    choice_type = haco_config["target_type"]
    component_name = config["component_name"]
    params = config["params"]

    component = components.get(self.target_type).get(component_name)
    estimator = component(**params)
   
    estimator.fit(data[0], data[2])
    ...
    tune.report(**metrics)

I appreciate any help or hints.