# Evaluating a Trained Model in Hierarchical Reinforcement Learning

**URL:** <https://discuss.ray.io/t/evaluating-a-trained-model-in-hierarchical-reinforcement-learning/21810>\
**Category:** Checkpointing, Restoring\
**Created:** [February 14, 2025, 3:36pm UTC](https://discuss.ray.io/t/evaluating-a-trained-model-in-hierarchical-reinforcement-learning/21810 "2025-02-14T15:36:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hunk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hunk/32/6675_2.png) [@Hunk](https://discuss.ray.io/u/Hunk)\
**Post date:** [February 14, 2025, 3:36pm UTC](https://discuss.ray.io/t/evaluating-a-trained-model-in-hierarchical-reinforcement-learning/21810/1 "2025-02-14T15:36:34Z")

</div>

**Background:**

We are working with `ray-2.12.0` and `RLlib` for hierarchical reinforcement learning (HRL). We are trying to load and evaluate the already trained low-level policy and high-level policy. Below is the current code snippet we’re using:

```
# The following code has issues:
from ray.rllib.policy.policy import Policy
model_save_path = "./checkpoint_000000" 
policy = Policy.from_checkpoint(model_save_path)
env = HierarchicalWindyMazeEnv(env_config)
for episode in range(3):
    obs = env.reset()
    action = policy['high_level_policy'].compute_single_action(obs)
    next_obs, reward, terminated, truncated, info = env.step(action)
    obs = next_obs

    # Run for a maximum of 200 time steps
    for time_step in range(200):
        # Concatenate current observation with the goal
        full_obs = np.concatenate([env.cur_obs, env.current_goal])
        
        # Low-level policy decision
        low_level_action = policy['low_level_policy'].compute_single_action(full_obs)
        next_obs, reward, terminated, truncated, info = env.step(low_level_action)
        
        # Update observation
        obs = next_obs

```

reference link:

1. [demo\_after\_training](https://github.com/DeUmbraTX/practical_rllib_tutorial/blob/main/demo/demo_after_training.py)
2. [hierarchical\_training]  
([ray/rllib/examples/hierarchical/hierarchical\_training.py at ray-2.12.0 · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/ray-2.12.0/rllib/examples/hierarchical/hierarchical_training.py))

**Problems Encountered:**

_1. How to load a trained model using PPOConfig?_  
We are currently using Policy.from\_checkpoint() to load the model, but we would like to know how to load a pre-trained PPO model using PPOConfig.  
_2. How to correctly evaluate low\_level\_policy and high\_level\_policy?_  
We are trying to access policy[‘low\_level\_policy’] and policy[‘high\_level\_policy’] directly for evaluation, but something seems wrong. We are unsure how to correctly evaluate and invoke these two policies in a hierarchical environment.

**Expected Solution:**

1. Clarification on how to load and evaluate a trained PPO model using `PPOConfig`.
2. Guidance on how to correctly evaluate and invoke `low_level_policy` and `high_level_policy` in a hierarchical RL setup.

**Thank you so much for your help, I really appreciate it!**