# AttributeError: 'APPO' object has no attribute '\_warmup\_time'

**URL:** <https://discuss.ray.io/t/attributeerror-appo-object-has-no-attribute-warmup-time/9531>\
**Category:** RLlib\
**Created:** [February 27, 2023, 5:40am UTC](https://discuss.ray.io/t/attributeerror-appo-object-has-no-attribute-warmup-time/9531 "2023-02-27T05:40:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![elliottower](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/elliottower/32/3856_2.png) [@elliottower](https://discuss.ray.io/u/elliottower)\
**Post date:** [February 27, 2023, 5:40am UTC](https://discuss.ray.io/t/attributeerror-appo-object-has-no-attribute-warmup-time/9531/1 "2023-02-27T05:40:34Z")

</div>

I made an issue on GitHub but posting here in case someone has any idea how to figure this out.

> <https://github.com/ray-project/ray/issues/32850>
>
> \### What happened + What you expected to happen
> 
> I'm trying to run the action\_…masking.py file for my custom environment and I keep running into this error for both PPO and APPO, but when I run it with the toy custom environment specified in the example it works fine. I have been able to run ray with my own custom environments by modifying the self\_play\_with\_open\_spiel file so I don't think it's an environment issue.
> 
> \`\`\`
> File "/Users/elliottower/Library/Caches/pypoetry/virtualenvs/cathedral-rl-6lsEyOq--py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 2335, in get\_auto\_filled\_metrics
> auto\_filled = super().get\_auto\_filled\_metrics(
> File "/Users/elliottower/Library/Caches/pypoetry/virtualenvs/cathedral-rl-6lsEyOq--py3.9/lib/python3.9/site-packages/ray/tune/trainable/trainable.py", line 267, in get\_auto\_filled\_metrics
> "warmup\_time": self.\_warmup\_time,
> AttributeError: 'PPO' object has no attribute '\_warmup\_time'
> \`\`\`
> 
> The only thing changed from my code and the action\_masking.py file is I added lines right after ray.init() which define my environment, and specifying the action\_space and observation\_space explicitly (when I debug I can see that env.action\_space is a discrete gymnasium space and observation space is a dict of observation and action mask). 
> 
> \`\`\`
> def env\_creator(args):
> return PettingZooEnv(cathedral\_v0.env())
> 
> env = env\_creator({})
> register\_env("cathedral", env\_creator)
> 
> \# main part: configure the ActionMaskEnv and ActionMaskModel
> config = (
> ppo.PPOConfig()
> .environment(
> env="cathedral",
> action\_space=env.action\_space,
> observation\_space=env.observation\_space,
> env\_config={ 
> "action\_space": env.action\_space,
> "observation\_space": env.observation\_space,
> },
> )
> \`\`\`
> 
> When I try running not in local mode I get the error that the rollout worker is unhealthy (another bug which I've been getting a lot and don't know how to fix, any help with that would be greatly appreciated as well). 
> 
> I found a potential cause for this being that my observation space has the keys 'action\_mask' and 'observation' from pettingzoo, but the action mask models both look for the key 'observations'
> 
> My run script depends on a custom repo I'm working on, but I'll put the script here anyways. The script is run from cathedral-rl/examples which needs to be cloned from https://github.com/elliottower/cathedral-rl
> 
> \### Versions / Dependencies
> 
> python = "\>=3.8, \<3.12"
> PettingZoo = "^1.22.3"
> gymnasium = "^0.26.3"
> pygame = "^2.1.3"
> SuperSuit = "^3.7.1"
> poetry = "^1.3.2"
> ray = {extras = \["rllib"\], version = "^2.3.0"}
> numpy = "^1.24.2"
> pandas = "^1.5.3"
> pymunk = "^6.4.0"
> tensorflow-probability = "^0.19.0"
> protobuf = "3.19.6"
> torch = "^1.13.1"
> tensorflow = "^2.11.0"
> 
> \### Reproduction script
> 
> \`\`\`
> """Example showing how to use "action masking" in RLlib.
> 
> "Action masking" allows the agent to select actions based on the current
> observation. This is useful in many practical scenarios, where different
> actions are available in different time steps.
> Blog post explaining action masking: https://boring-guy.sh/posts/masking-rl/
> 
> RLlib supports action masking, i.e., disallowing these actions based on the
> observation, by slightly adjusting the environment and the model as shown in
> this example.
> 
> Here, the ActionMaskEnv wraps an underlying environment (here, RandomEnv),
> defining only a subset of all actions as valid based on the environment's
> observations. If an invalid action is selected, the environment raises an error
> \- this must not happen!
> 
> The environment constructs Dict observations, where obs\["observations"\] holds
> the original observations and obs\["action\_mask"\] holds the valid actions.
> To avoid selection invalid actions, the ActionMaskModel is used. This model
> takes the original observations, computes the logits of the corresponding
> actions and then sets the logits of all invalid actions to zero, thus disabling
> them. This only works with discrete actions.
> 
> \---
> Run this example with defaults (using Tune and action masking):
> 
> $ python action\_masking.py
> 
> Then run again without action masking, which will likely lead to errors due to
> invalid actions being selected (ValueError "Invalid action sent to env!"):
> 
> $ python action\_masking.py --no-masking
> 
> Other options for running this example:
> 
> $ python action\_masking.py --help
> """
> 
> import argparse
> import os
> 
> from gymnasium.spaces import Box, Discrete
> import ray
> from ray import air, tune
> from ray.rllib.algorithms import ppo
> from ray.rllib.env import PettingZooEnv
> from ray.rllib.examples.env.action\_mask\_env import ActionMaskEnv
> from ray.rllib.examples.models.action\_mask\_model import (
> ActionMaskModel,
> TorchActionMaskModel,
> )
> from ray.tune import register\_env
> from ray.tune.logger import pretty\_print
> 
> from cathedral\_rl import cathedral\_v0
> 
> 
> def get\_cli\_args():
> """Create CLI parser and return parsed arguments"""
> parser = argparse.ArgumentParser()
> 
> # example-specific args
> parser.add\_argument(
> "--no-masking",
> action="store\_true",
> help="Do NOT mask invalid actions. This will likely lead to errors.",
> )
> 
> # general args
> parser.add\_argument(
> "--run", type=str, default="APPO", help="The RLlib-registered algorithm to use."
> )
> parser.add\_argument("--num-cpus", type=int, default=0)
> parser.add\_argument(
> "--framework",
> choices=\["tf", "tf2", "torch"\],
> default="tf",
> help="The DL framework specifier.",
> )
> parser.add\_argument("--eager-tracing", action="store\_true")
> parser.add\_argument(
> "--stop-iters", type=int, default=10, help="Number of iterations to train."
> )
> parser.add\_argument(
> "--stop-timesteps",
> type=int,
> default=10000,
> help="Number of timesteps to train.",
> )
> parser.add\_argument(
> "--stop-reward",
> type=float,
> default=80.0,
> help="Reward at which we stop training.",
> )
> parser.add\_argument(
> "--no-tune",
> action="store\_true",
> help="Run without Tune using a manual train loop instead. Here,"
> "there is no TensorBoard support.",
> )
> parser.add\_argument(
> "--local-mode",
> action="store\_true",
> help="Init Ray in local mode for easier debugging.",
> )
> 
> args = parser.parse\_args()
> print(f"Running with following CLI args: {args}")
> return args
> 
> 
> if \_\_name\_\_ == "\_\_main\_\_":
> args = get\_cli\_args()
> 
> ray.init(num\_cpus=args.num\_cpus or None, local\_mode=args.local\_mode)
> 
> def env\_creator(args):
> return PettingZooEnv(cathedral\_v0.env())
> 
> env = env\_creator({})
> register\_env("cathedral", env\_creator)
> 
> # main part: configure the ActionMaskEnv and ActionMaskModel
> config = (
> ppo.PPOConfig()
> .environment(
> env="cathedral",
> action\_space=env.action\_space,
> observation\_space=env.observation\_space,
> env\_config={
> "action\_space": env.action\_space,
> "observation\_space": env.observation\_space,
> },
> )
> .training(
> # the ActionMaskModel retrieves the invalid actions and avoids them
> model={
> "custom\_model": ActionMaskModel
> if args.framework != "torch"
> else TorchActionMaskModel,
> # disable action masking according to CLI
> "custom\_model\_config": {"no\_masking": args.no\_masking},
> },
> )
> .framework(args.framework, eager\_tracing=args.eager\_tracing)
> .resources(
> # Use GPUs iff \`RLLIB\_NUM\_GPUS\` env var set to \> 0.
> num\_gpus=int(os.environ.get("RLLIB\_NUM\_GPUS", "0"))
> )
> )
> 
> stop = {
> "training\_iteration": args.stop\_iters,
> "timesteps\_total": args.stop\_timesteps,
> "episode\_reward\_mean": args.stop\_reward,
> }
> 
> # manual training loop (no Ray tune)
> if args.no\_tune:
> if args.run not in {"APPO", "PPO"}:
> raise ValueError("This example only supports APPO and PPO.")
> 
> algo = config.build()
> 
> # run manual training loop and print results after each iteration
> for \_ in range(args.stop\_iters):
> result = algo.train()
> print(pretty\_print(result))
> # stop training if the target train steps or reward are reached
> if (
> result\["timesteps\_total"\] \>= args.stop\_timesteps
> or result\["episode\_reward\_mean"\] \>= args.stop\_reward
> ):
> break
> 
> # manual test loop
> print("Finished training. Running manual test/inference loop.")
> # prepare environment with max 10 steps
> config\["env\_config"\]\["max\_episode\_len"\] = 10
> env = ActionMaskEnv(config\["env\_config"\])
> obs, info = env.reset()
> done = False
> # run one iteration until done
> print(f"ActionMaskEnv with {config\['env\_config'\]}")
> while not done:
> action = algo.compute\_single\_action(obs)
> next\_obs, reward, done, truncated, \_ = env.step(action)
> # observations contain original observations and the action mask
> # reward is random and irrelevant here and therefore not printed
> print(f"Obs: {obs}, Action: {action}")
> obs = next\_obs
> 
> # run with tune for auto trainer creation, stopping, TensorBoard, etc.
> else:
> tuner = tune.Tuner(
> args.run,
> param\_space=config.to\_dict(),
> run\_config=air.RunConfig(stop=stop, verbose=2),
> )
> tuner.fit()
> 
> print("Finished successfully without selecting invalid actions.")
> ray.shutdown()
> 
> \`\`\`
> 
> \### Issue Severity
> 
> High: It blocks me from completing my task.

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [March 22, 2023, 1:44am UTC](https://discuss.ray.io/t/attributeerror-appo-object-has-no-attribute-warmup-time/9531/2 "2023-03-22T01:44:20Z")

</div>

Hi @elliottower ,

Thanks for raising this issue.  
I’ve replied and closed the issue. Feel free to reopen it if you think that there is more to discuss related to this issue. 🙂