Printing input_dict["obs_flat"] with custom DefaultCallbacks

mg64ve · December 9, 2021, 6:51pm

Question regarding standard LSTM model in:

ray-project/ray/blob/8a72824c63dbae9d8d189ce9c553c2e8c4299494/rllib/models/torch/recurrent_net.py

import numpy as np
import gym
from gym.spaces import Discrete, MultiDiscrete
from typing import Dict, List, Union

from ray.rllib.models.modelv2 import ModelV2
from ray.rllib.models.torch.misc import SlimFC
from ray.rllib.models.torch.torch_modelv2 import TorchModelV2
from ray.rllib.policy.rnn_sequencing import add_time_dimension
from ray.rllib.policy.sample_batch import SampleBatch
from ray.rllib.policy.view_requirement import ViewRequirement
from ray.rllib.utils.annotations import override, DeveloperAPI
from ray.rllib.utils.framework import try_import_torch
from ray.rllib.utils.torch_ops import one_hot
from ray.rllib.utils.typing import ModelConfigDict, TensorType

torch, nn = try_import_torch()


@DeveloperAPI

This file has been truncated. show original

with a custom DefaultCallbacks may I print input_dict[“obs_flat”] variables ?
What functions should I call?
I have tried:

trainer.get_policy().model.input_dict["obs_prev_action"].items()

from on_train_result but it does not work.

bmanczak · December 9, 2021, 7:38pm

Hi,

You may want to look into the data stored by the Episode object.

You can access the data stored by this object in on_episode_step and on_episode_end in a callback that inherits from DefaultCallbacks. See this for an example.

mg64ve · December 9, 2021, 9:07pm

Thank you @bmanczak for your help.
The problem is that this data is stored inside the model and that is the data I want to print out.
Is there any way to access the model from Episode? I had a quick look to Episode definition in:

github.com

ray-project/ray/blob/83c639ea76b9793b40fbd78ebb981a81f0ed28cf/rllib/evaluation/episode.py

from collections import defaultdict
import numpy as np
import random
from typing import Any, Callable, Dict, List, Optional, TYPE_CHECKING

from ray.rllib.env.base_env import _DUMMY_AGENT_ID
from ray.rllib.policy.policy_map import PolicyMap
from ray.rllib.utils.annotations import Deprecated, DeveloperAPI
from ray.rllib.utils.deprecation import deprecation_warning
from ray.rllib.utils.spaces.space_utils import flatten_to_single_ndarray
from ray.rllib.utils.typing import SampleBatchType, AgentID, PolicyID, \
    EnvActionType, EnvID, EnvInfoDict, EnvObsType
from ray.util import log_once

if TYPE_CHECKING:
    from ray.rllib.evaluation.rollout_worker import RolloutWorker
    from ray.rllib.evaluation.sample_batch_builder import \
        MultiAgentSampleBatchBuilder

This file has been truncated. show original

and I do not think so…

sven1977 · December 14, 2021, 8:22am

Hey @mg64ve , could you post a short repro script here so we can run something and debug?
I’m not sure I understand what you are trying to print out. The key “obs_prev_actions” in the input dicts does not exist in RLlib. “prev_actions” and “prev_rewards” will be available if you set:

config:
    model:
        lstm_use_prev_action: True
        lstm_use_prev_reward: True

mg64ve · December 14, 2021, 5:06pm

Hi @sven1977 you are right I made some confusion with variables. But concept is the same.
I have prepared the following example:

github.com

mg64ve/ray_tests/blob/main/DefaultCallbacks/DefaultCallbacksFrameStacking.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dc33e18e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym.spaces import Box\n",
    "import numpy as np\n",
    "import gym\n",
    "from gym.spaces import Discrete, MultiDiscrete\n",
    "from typing import Dict, List, Union\n",
    "\n",
    "from gym.envs.classic_control import CartPoleEnv"
   ]
  },
  {
   "cell_type": "code",

This file has been truncated. show original

It is taken from the official FrameStacking example. The concept is that I want to print some variable in the model class. In this case it is input_dict[“prev_n_obs”].
This notebook fails with error:

AttributeError: 'Episode' object has no attribute 'input_dict'

How can I do it?

mannyv · December 16, 2021, 9:09pm

Hi @mg64ve,

The input_dict is not a member of of the policy it if a parameter of the forward function. The most straight forward way if you just want to see what they look would be to add a print statement in the code there or add a breakpoint and debug interactively.

Topic		Replies	Views
Question on code: _wrapped_forward RLlib	2	342	November 30, 2021
[rllib] Dict Action Space and Custom Model RLlib	5	2333	March 30, 2021
Custom RNN Model with Examples - why do they fail? RLlib	11	2295	May 5, 2022
Get_initial_state for LSTM custom model without initial FC RLlib	1	390	January 12, 2022
Compute_single_action randomly errors without changing input RLlib	0	239	October 16, 2023

Printing input_dict["obs_flat"] with custom DefaultCallbacks

Related topics