How to make checkpoint by and load it?

What is the best way to :

  1. make checkpoint by
  2. load checkpoint by PPOTrainer
  3. Show checkpoint by tensorflow model.summary ?

I try to make checkpoint by :

from ray import tune"PPO",
             "framework": "tf2",

next, load model by:

from ray.rllib.agents.ppo.ppo import PPOTrainer
import gym

config = {
    "env": "CartPole-v0",
    "framework": "tf2",
    "evaluation_interval": 2,
    "evaluation_duration": 20,

agent = PPOTrainer(config=config)

env = gym.make("CartPole-v0")
obs = env.reset()
while True:
    action = agent.compute_action(obs)
    obs, reward, done, _ = env.step(action)
    if done:

but there is error:

 File "/home/ppirog/projects/Mastering-Reinforcement-Learning-with-Python/3D_observations/", line 13, in <module>
TypeError: load_checkpoint() missing 1 required positional argument: 'checkpoint_path'
Exception ignored in: <function RolloutWorker.__del__ at 0x7fd2f9785e50>
Traceback (most recent call last):
  File "/home/ppirog/projects/Mastering-Reinforcement-Learning-with-Python/venv/lib/python3.8/site-packages/ray/util/tracing/", line 461, in _resume_span
TypeError: 'NoneType' object is not callable
Exception ignored in: <function RolloutWorker.__del__ at 0x7fd2f9772a60>
Traceback (most recent call last):
  File "/home/ppirog/projects/Mastering-Reinforcement-Learning-with-Python/venv/lib/python3.8/site-packages/ray/util/tracing/", line 461, in _resume_span
TypeError: 'NoneType' object is not callable
Process finished with exit code 1

At last i try to show model:

import tensorflow as tf
new_model = tf.keras.models.load_model(model_path)


But there is error:

OSError: SavedModel file does not exist at: /home/ppirog/projects/Mastering-Reinforcement-Learning-with-Python/3D_observations/cartpole/PPO/PPO_CartPole-v0_2890c_00000_0_2022-06-30_16-03-19/checkpoint_000002/{saved_model.pbtxt|saved_model.pb}

For loading a checkpoint with RLlib agent, please refer to this example: Training APIs — Ray 1.13.0

Let me know how it goes!

1 Like

@xwjiang2010 Thank You, I will try it :slight_smile:

@xwjiang2010 , I tested the code it works fine :slight_smile:

Here is the code how to visualize the trained model:

import ray.rllib.agents.ppo as ppo
from tensorflow.keras.utils import plot_model

checkpoint_path=" " # <-in this place put the path to saved checkpoint
config={ } # <- in this place put configuration the same like during training process

trainer = ppo.PPOTrainer(config=config)
model=trainer.get_policy().model.base_model # this is typical tf2 model format 
# typical tensorflow visualization
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True,rankdir="TB",expand_nested=False,show_layer_activations=True)

Result is:

Model: "model"
 Layer (type)                   Output Shape         Param #     Connected to                     
 observations (InputLayer)      [(None, 4)]          0           []                               
 fc_1 (Dense)                   (None, 256)          1280        ['observations[0][0]']           
 fc_value_1 (Dense)             (None, 256)          1280        ['observations[0][0]']           
 fc_2 (Dense)                   (None, 256)          65792       ['fc_1[0][0]']                   
 fc_value_2 (Dense)             (None, 256)          65792       ['fc_value_1[0][0]']             
 fc_out (Dense)                 (None, 2)            514         ['fc_2[0][0]']                   
 value_out (Dense)              (None, 1)            257         ['fc_value_2[0][0]']             
Total params: 134,915
Trainable params: 134,915
Non-trainable params: 0

1 Like