# Restoring a policy or a keras model from a checkpoint

**URL:** <https://discuss.ray.io/t/restoring-a-policy-or-a-keras-model-from-a-checkpoint/9521>\
**Category:** RLlib\
**Created:** [February 26, 2023, 10:21am UTC](https://discuss.ray.io/t/restoring-a-policy-or-a-keras-model-from-a-checkpoint/9521 "2023-02-26T10:21:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shmiak](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shmiak/32/3970_2.png) [@shmiak](https://discuss.ray.io/u/shmiak)\
**Post date:** [February 26, 2023, 10:21am UTC](https://discuss.ray.io/t/restoring-a-policy-or-a-keras-model-from-a-checkpoint/9521/1 "2023-02-26T10:21:41Z")

</div>

Hi!

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

**The problem**  
During training I create checkpoints and export Keras models following the guidelines at [Saving and Loading your RL Algorithms and Policies — Ray 2.3.0](https://docs.ray.io/en/latest/rllib/rllib-saving-and-loading-algos-and-policies.html#model-exports)

```auto
path_to_checkpoint = algo.save()
policy = algo.get_policy()
policy.export_model(path_to_checkpoint + "/keras_model")

```

When I am trying to restore a policy I get a Runtime error from tf

```auto
    rllib_policy = Policy.from_checkpoint(checkpoint_dir + "/policies/default_policy")
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/policy.py", line 269, in from_checkpoint
    return Policy.from_state(state)
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/policy.py", line 302, in from_state
    return TFPolicy._tf1_from_state_helper(state)
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/tf_policy.py", line 506, in _tf1_from_state_helper
    new_policy = pol_spec.policy_class(
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/algorithms/ppo/ppo_tf_policy.py", line 83, in __init__
    base. __init__ (
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/dynamic_tf_policy_v2.py", line 89, in __init__
    timestep, explore = self._init_input_dict_and_dummy_batch(existing_inputs)
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/dynamic_tf_policy_v2.py", line 488, in _init_input_dict_and_dummy_batch
    ) = self._create_input_dict_and_dummy_batch(self.view_requirements, {})
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/policy/dynamic_tf_policy_v2.py", line 547, in _create_input_dict_and_dummy_batch
    input_dict[view_col] = get_placeholder(
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/ray/rllib/utils/tf_utils.py", line 214, in get_placeholder
    return tf1.placeholder(
  File "/home/sia/miniconda3/envs/ray2.3/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py", line 3340, in placeholder
    raise RuntimeError("tf.placeholder() is not compatible with "
RuntimeError: tf.placeholder() is not compatible with eager execution.

```

The workaround is to restore an algorithm from a checkpoint, and then get a policy from it. It works fine, but I do not need the entire algorithm but only a policy from it.

I thought that I could also use a TensorFlow model for inference. I can successfully get it using `tf.saved_model.load(checkpoint_dir + "/keras_model/")`. But the issue is that predictions from a policy and a model are different. Ok policy is stochastic so they should not be exactly the same, but predictions were very different. At first, I thought that it is due to some default wrappers, but when I compared model weights from a policy and a model they were also different. Also, for different checkpoints sometimes weights are almost the same, but sometimes they differ a lot.

So, the question is how to properly restore a model or a policy from a checkpoint without the RLLIB algorithm initialization.

**Environment info / some configs**  
Python: 3.9.16  
Tensorflow: 2.11.0  
Ray: 2.3.0  
Ray algorithm is PPO, the framework is `tf` (`tf2` with tracing raises an error during getting `is_training` from `input_dict` in `forward` method of TFModelV2, so it is also an issue, but I do not use it currently and since PPO is an `on policy` it should not be a huge problem I believe), and the model is a custom subclassing Keras model.

---

<div class="post-metadata">

**Author:** ![shmiak](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shmiak/32/3970_2.png) [@shmiak](https://discuss.ray.io/u/shmiak)\
**Post date:** [March 1, 2023, 5:33pm UTC](https://discuss.ray.io/t/restoring-a-policy-or-a-keras-model-from-a-checkpoint/9521/2 "2023-03-01T17:33:16Z")

</div>

I have prepared an example derived from [ray/custom\_env.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/rllib/examples/custom_env.py) to reproduce the errors I mentioned. - [rllib-env-model/rllib\_env\_model.ipynb at main · shmyak-ai/rllib-env-model · GitHub](https://github.com/shmyak-ai/rllib-env-model/blob/main/rllib_env_model.ipynb)

The problem is in a subclass Keras model. So far I have found two requirements to use subclassing:

1. A model instance must be called self.base\_model to be saved,
2. The model should be run once during initialization.