# Tune and custom logger fails

**URL:** <https://discuss.ray.io/t/tune-and-custom-logger-fails/14172>\
**Category:** RLlib\
**Created:** [March 27, 2024, 11:31am UTC](https://discuss.ray.io/t/tune-and-custom-logger-fails/14172 "2024-03-27T11:31:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbk80](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dbk80/32/1822_2.png) [@dbk80](https://discuss.ray.io/u/dbk80)\
**Post date:** [March 27, 2024, 11:31am UTC](https://discuss.ray.io/t/tune-and-custom-logger-fails/14172/1 "2024-03-27T11:31:06Z")

</div>

Hi,

I am using a custom callback that works fine when I train an RL model without using tune:

```auto
config = ppo.PPOConfig(
        ).environment(MyCustomEnv, env_config=env_config, disable_env_checking=True
        ).training(
            model={
                'custom_model': model_class_name,
                'custom_model_config': custom_model_config
                },
        ).framework(eager_tracing=True, framework=framework
        ).resources(num_gpus=1, num_cpus_per_worker=1, num_cpus_per_learner_worker=2,
        ).rollouts(enable_connectors=False, num_rollout_workers=2).callbacks(MyCallbacks
        ).reporting(keep_per_episode_custom_metrics=True)
    
    algo = config.build()
    
    num_iterations = 2
    for _ in range(num_iterations):
        result = algo.train()

```

However, when I try to use tune. Namely:

```auto
    tuner = tune.Tuner(
        "PPO",
        run_config=air.RunConfig(
            storage_path=run_path,
            stop={
                "training_iteration": num_iterations,
            },
            # callbacks=[MyCallbacks],
        ),
        param_space=config,
    )
result = tuner.fit()

```

The custom callback is ignored. As stated above, the custom callback works fine when tune is not used.  
If I try to include the custom callback also in my `air.RunConfig` by commenting out the line above, I receive the following error:

```auto
AttributeError: type object 'MyCallbacks' has no attribute 'setup'

```

I also tried the following variation: `callbacks=[MyCallbacks()]`. This leads to the same error.

My current ray version is 2.5.1 (`pip install ray[default,rllib]==2.5.1`)

---

<div class="post-metadata">

**Author:** ![jacob-thrackle](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jacob-thrackle/32/5426_2.png) [@jacob-thrackle](https://discuss.ray.io/u/jacob-thrackle)\
**Post date:** [May 7, 2024, 5:40pm UTC](https://discuss.ray.io/t/tune-and-custom-logger-fails/14172/2 "2024-05-07T17:40:28Z")

</div>

I am having the same issue on Ray 2.20.0 (updated from 2.8.0 which also had the same problem). Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![jacob-thrackle](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jacob-thrackle/32/5426_2.png) [@jacob-thrackle](https://discuss.ray.io/u/jacob-thrackle)\
**Post date:** [May 7, 2024, 8:03pm UTC](https://discuss.ray.io/t/tune-and-custom-logger-fails/14172/3 "2024-05-07T20:03:37Z")

</div>

Replying to myself:

I was able to get it to run by subclassing `tune.Callback` but have not yet been able to see the custom metrics in the output. Simplified Callback class for posterity:

```auto
import numpy as np
from collections import deque
# from ray.rllib.algorithms.callbacks import DefaultCallbacks
from ray.tune import Callback
from ray.rllib.env import BaseEnv
from ray.rllib.policy import Policy
from ray.rllib.evaluation import RolloutWorker
from ray.rllib.evaluation.episode import Episode
from typing import Dict

class MarketCallbacks(Callback):
    def on_episode_start(
        self,
        *,
        worker: RolloutWorker,
        base_env: BaseEnv,
        policies: Dict[str, Policy],
        episode: Episode,
        env_index: int,
        **kwargs,
    ):
        # Initialize storage for prices in this episode
        episode.user_data["prices"] = []

    def on_episode_step(
        self,
        *,
        worker: RolloutWorker,
        base_env: BaseEnv,
        policies: Dict[str, Policy],
        episode: Episode,
        env_index: int,
        **kwargs,
    ):
        # Retrieve the market object from the environment
        market = base_env.get_unwrapped()[
            env_index
        ].market
        current_price = market.get_current_price()
        episode.user_data["prices"].append(current_price)

    def on_episode_end(
        self,
        *,
        worker: RolloutWorker,
        base_env: BaseEnv,
        policies: Dict[str, Policy],
        episode: Episode,
        env_index: int,
        **kwargs,
    ):
        prices = np.array(episode.user_data["prices"])
        episode.custom_metrics["max_price"] = np.max(prices)
        episode.custom_metrics["mean_price"] = np.mean(prices)
        episode.custom_metrics["std_price"] = np.std(prices)

        episode.hist_data["price"] = episode.user_data["price"]

    def on_train_result(self, *, algorithm, result: dict, **kwargs):
        # you can mutate the result dict to add new fields to return
        result["callback_ok"] = True

        p = result["custom_metrics"]["price"]
        result["custom_metrics"]["prices_max"] = np.max(p)
        result["custom_metrics"]["prices_mean"] = np.mean(p)
        result["custom_metrics"]["prices_std"] = np.std(p)

```

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [May 8, 2024, 2:44pm UTC](https://discuss.ray.io/t/tune-and-custom-logger-fails/14172/4 "2024-05-08T14:44:24Z")

</div>

Hi @jacob-thrackle,

The terminology is overloaded which makes this confusing. The tune callbacks and rllib callbacks and they are different types. You are providing an rllib callback to the tune config. You need to pass it to the rllib `AlgorithmConfig`.

I could not find an example in the project but here is an issue which shows an example using both an rllib callback and a tune callback. You may need to adapt it to use the builder pattern instead if a dictionary.

> <https://github.com/ray-project/ray/issues/27744>
>
> \### What happened + What you expected to happen
> 
> When logging custom metrics to …wandb via \`WandbLoggerCallback\` the metrics are duplicated and visible as:
> \- \`custom\_metrics/\*'
> \- \`sampler\_resutls/custom\_metrics/\*\`
> 
> I would expect to see just \`custom\_metrics/\*\` I have implemented.
> 
> !\[image\](https://user-images.githubusercontent.com/16837172/183909752-3077e346-0553-4732-a011-f1fbad2bdfb7.png)
> 
> After some drilling I found most probably responsible line https://github.com/ray-project/ray/blob/master/rllib/algorithms/algorithm.py#L2541
> with potentialy related \`# TODO: Don't dump sampler results into top-level.\`
> 
> \### Versions / Dependencies
> 
> 1.13.0
> 
> \### Reproduction script
> 
> Modified example from https://github.com/ray-project/ray/blob/master/rllib/examples/custom\_metrics\_and\_callbacks.py enriched with WandbLoggerCallback
> \`\`\`
> """Example of using RLlib's debug callbacks.
> Here we use callbacks to track the average CartPole pole angle magnitude as a
> custom metric.
> """
> 
> from typing import Dict, Tuple
> import argparse
> import numpy as np
> import os
> 
> import ray
> from ray import tune
> from ray.rllib.agents.callbacks import DefaultCallbacks
> from ray.rllib.env import BaseEnv
> from ray.rllib.evaluation import Episode, RolloutWorker
> from ray.rllib.policy import Policy
> from ray.rllib.policy.sample\_batch import SampleBatch
> from ray.tune.integration.wandb import WandbLoggerCallback
> 
> parser = argparse.ArgumentParser()
> parser.add\_argument(
> "--framework",
> choices=\["tf", "tf2", "tfe", "torch"\],
> default="tf",
> help="The DL framework specifier.",
> )
> parser.add\_argument("--stop-iters", type=int, default=2000)
> 
> 
> class MyCallbacks(DefaultCallbacks):
> def on\_episode\_start(
> self,
> \*,
> worker: RolloutWorker,
> base\_env: BaseEnv,
> policies: Dict\[str, Policy\],
> episode: Episode,
> env\_index: int,
> \*\*kwargs
> ):
> # Make sure this episode has just been started (only initial obs
> # logged so far).
> assert episode.length == 0, (
> "ERROR: \`on\_episode\_start()\` callback should be called right "
> "after env reset!"
> )
> print("episode {} (env-idx={}) started.".format(episode.episode\_id, env\_index))
> episode.user\_data\["pole\_angles"\] = \[\]
> episode.hist\_data\["pole\_angles"\] = \[\]
> 
> def on\_episode\_step(
> self,
> \*,
> worker: RolloutWorker,
> base\_env: BaseEnv,
> policies: Dict\[str, Policy\],
> episode: Episode,
> env\_index: int,
> \*\*kwargs
> ):
> # Make sure this episode is ongoing.
> assert episode.length \> 0, (
> "ERROR: \`on\_episode\_step()\` callback should not be called right "
> "after env reset!"
> )
> pole\_angle = abs(episode.last\_observation\_for()\[2\])
> raw\_angle = abs(episode.last\_raw\_obs\_for()\[2\])
> assert pole\_angle == raw\_angle
> episode.user\_data\["pole\_angles"\].append(pole\_angle)
> 
> def on\_episode\_end(
> self,
> \*,
> worker: RolloutWorker,
> base\_env: BaseEnv,
> policies: Dict\[str, Policy\],
> episode: Episode,
> env\_index: int,
> \*\*kwargs
> ):
> # Check if there are multiple episodes in a batch, i.e.
> # "batch\_mode": "truncate\_episodes".
> if worker.policy\_config\["batch\_mode"\] == "truncate\_episodes":
> # Make sure this episode is really done.
> assert episode.batch\_builder.policy\_collectors\["default\_policy"\].batches\[
> -1
> \]\["dones"\]\[-1\], (
> "ERROR: \`on\_episode\_end()\` should only be called "
> "after episode is done!"
> )
> pole\_angle = np.mean(episode.user\_data\["pole\_angles"\])
> print(
> "episode {} (env-idx={}) ended with length {} and pole "
> "angles {}".format(
> episode.episode\_id, env\_index, episode.length, pole\_angle
> )
> )
> episode.custom\_metrics\["pole\_angle"\] = pole\_angle
> episode.hist\_data\["pole\_angles"\] = episode.user\_data\["pole\_angles"\]
> 
> def on\_sample\_end(self, \*, worker: RolloutWorker, samples: SampleBatch, \*\*kwargs):
> print("returned sample batch of size {}".format(samples.count))
> 
> def on\_train\_result(self, \*, trainer, result: dict, \*\*kwargs):
> print(
> "trainer.train() result: {} -\> {} episodes".format(
> trainer, result\["episodes\_this\_iter"\]
> )
> )
> # you can mutate the result dict to add new fields to return
> result\["callback\_ok"\] = True
> 
> def on\_learn\_on\_batch(
> self, \*, policy: Policy, train\_batch: SampleBatch, result: dict, \*\*kwargs
> ) -\> None:
> result\["sum\_actions\_in\_train\_batch"\] = np.sum(train\_batch\["actions"\])
> print(
> "policy.learn\_on\_batch() result: {} -\> sum actions: {}".format(
> policy, result\["sum\_actions\_in\_train\_batch"\]
> )
> )
> 
> def on\_postprocess\_trajectory(
> self,
> \*,
> worker: RolloutWorker,
> episode: Episode,
> agent\_id: str,
> policy\_id: str,
> policies: Dict\[str, Policy\],
> postprocessed\_batch: SampleBatch,
> original\_batches: Dict\[str, Tuple\[Policy, SampleBatch\]\],
> \*\*kwargs
> ):
> print("postprocessed {} steps".format(postprocessed\_batch.count))
> if "num\_batches" not in episode.custom\_metrics:
> episode.custom\_metrics\["num\_batches"\] = 0
> episode.custom\_metrics\["num\_batches"\] += 1
> 
> 
> if \_\_name\_\_ == "\_\_main\_\_":
> args = parser.parse\_args()
> 
> ray.init()
> trials = tune.run(
> "PG",
> stop={
> "training\_iteration": args.stop\_iters,
> },
> config={
> "env": "CartPole-v0",
> "num\_envs\_per\_worker": 2,
> "callbacks": MyCallbacks,
> "framework": args.framework,
> # Use GPUs iff \`RLLIB\_NUM\_GPUS\` env var set to \> 0.
> "num\_gpus": int(os.environ.get("RLLIB\_NUM\_GPUS", "0")),
> },
> callbacks=\[WandbLoggerCallback(
> name="test",
> project="test",
> \*\*{"api\_key\_file": "~/.wandb"},
> )\],
> ).trials
> \`\`\`
> 
> \### Issue Severity
> 
> Low: It annoys or frustrates me.

[https://docs.ray.io/en/latest/rllib/package\_ref/doc/ray.rllib.algorithms.algorithm\_config.AlgorithmConfig.html#ray.rllib.algorithms.algorithm\_config.AlgorithmConfig](https://docs.ray.io/en/latest/rllib/package_ref/doc/ray.rllib.algorithms.algorithm_config.AlgorithmConfig.html#ray.rllib.algorithms.algorithm_config.AlgorithmConfig)