After trainer.train(), there is a bug coming out

RLlib
1

File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py”, line 461, in _resume_span
TypeError: ‘NoneType’ object is not callable
I think there are some None exist in the checkpoint

2

Could you provide a repro script and paste the full stacktrace? Thanks!

4

Thanks for helping me :slight_smile:

ray.init(ignore_reinit_error=True)
trainer = A2CTrainer
register_env(env_name, lambda config: env_creator(self.env_name)
                     (config))
model_config["env"] = env_name
model_config["env_config"] =train_env_config
sharpes = []
checkpoints = []
trainer = trainer(env=env_name,
                                    config=model_config)
for i in range(2):
        trainer.train()
        valid_env_instance = env_creator(env_name)(
                valid_env_config)
        state = valid_env_instance.reset()
        done = False
        while not done:
                action = trainer.compute_single_action(state)
                state, reward, done, information = valid_env_instance.step(
                    action)
        sharpes.append(information["sharpe_ratio"])
        checkpoint = self.trainer.save()
        trainer.restore(checkpoint)
        checkpoints.append(checkpoint)
loc = sharpes.index(max(sharpes))
trainer.restore(checkpoints[loc])
trainer.save(trained_model_dict)
ray.shutdown()

and here is the problem:
2022-07-08 13:41:42,119 INFO services.py:1470 – View the Ray dashboard at http://127.0.0.1:8265
2022-07-08 13:41:44,078 INFO trainer.py:903 – Current log_level is WARN. For more information, set ‘log_level’: ‘INFO’ / ‘DEBUG’ or use the -v and -vv flags.
(RolloutWorker pid=1692368) 2022-07-08 13:41:48,494 WARNING env.py:135 – Your env doesn’t have a .spec.max_episode_steps attribute. This is fine if you have set ‘horizon’ in your config dictionary, or soft_horizon. However, if you haven’t, ‘horizon’ will default to infinity, and your environment will not be reset.
2022-07-08 13:41:48,670 WARNING util.py:65 – Install gputil for GPU system monitoring.
(RolloutWorker pid=1692368) =================================
(RolloutWorker pid=1692368) the profit margin is 64.1782399823528 %
(RolloutWorker pid=1692368) the sharpe ratio is 1.4976463038091303
(RolloutWorker pid=1692368) the Volatility is 0.008453999055043442
(RolloutWorker pid=1692368) the max drawdown is 0.40600774513409027
(RolloutWorker pid=1692368) the Calmar Ratio is 1.3984378526411951
(RolloutWorker pid=1692368) the Sortino Ratio is 2.042888761004623
(RolloutWorker pid=1692368) =================================

the profit margin is 7.941804289270293 %
the sharpe ratio is 0.39396025059480905
the Volatility is 0.022334853455848163
the max drawdown is 0.35554847692589864
the Calmar Ratio is 0.3936379399741129
the Sortino Ratio is 0.4875073433055513

/home/sunshuo/ray_results/A2CTrainer_portfolio_2022-07-08_13-41-44rkwv_e2u/checkpoint_000001/checkpoint-1
<class ‘bool’>
False
<class ‘float’>
0.9
<class ‘float’>
0.999
<class ‘bool’>
False
<class ‘float’>
1e-08
<class ‘NoneType’>
None
Traceback (most recent call last):
File “/home/sunshuo/qml/TradeMaster_reframe/agent/SOTA/SOTA.py”, line 215, in
a.train_with_valid()
File “/home/sunshuo/qml/TradeMaster_reframe/agent/SOTA/SOTA.py”, line 187, in train_with_valid
self.trainer.restore(checkpoint)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/tune/trainable.py”, line 583, in restore
self.load_checkpoint(checkpoint_path)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/agents/trainer.py”, line 2070, in load_checkpoint
self.setstate(extra_data)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/agents/trainer.py”, line 2693, in setstate
self.workers.local_worker().restore(state[“worker”])
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py”, line 1542, in restore
self.policy_map[pid].set_state(state)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/policy/torch_policy.py”, line 766, in set_state
optim_state_dict = convert_to_torch_tensor(s, device=self.device)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/utils/torch_utils.py”, line 163, in convert_to_torch_tensor
return tree.map_structure(mapping, x)
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/tree/init.py”, line 430, in map_structure
[func(*args) for args in zip(*map(flatten, structures))])
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/tree/init.py”, line 430, in
[func(*args) for args in zip(*map(flatten, structures))])
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/rllib/utils/torch_utils.py”, line 157, in mapping
tensor = torch.from_numpy(np.asarray(item))
TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
Exception ignored in: <function RolloutWorker.del at 0x7f83f49cf820>
Traceback (most recent call last):
File “/home/sunshuo/miniconda3/envs/TradeMaster/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py”, line 461, in _resume_span
TypeError: ‘NoneType’ object is not callable

5

Thanks for the help. I look up in the issue of Ray and find a very similiar problem which will happen in python 3.8, I downgrad my python to 3.7 and now my code works perfectly.

6

Thanks!
Just by looking at the stacktrace - seems very similar to this issue.
It may be worthwhile to document there what you have tried and discovered.