# Running a random policy with RLlib

**URL:** <https://discuss.ray.io/t/running-a-random-policy-with-rllib/5198>\
**Category:** RLlib\
**Created:** [February 25, 2022, 10:02am UTC](https://discuss.ray.io/t/running-a-random-policy-with-rllib/5198 "2022-02-25T10:02:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sn73jq](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@sn73jq](https://discuss.ray.io/u/sn73jq)\
**Post date:** [February 25, 2022, 10:02am UTC](https://discuss.ray.io/t/running-a-random-policy-with-rllib/5198/1 "2022-02-25T10:02:09Z")

</div>

Hi, I want to run a random policy (`ray.rllib.examples.policy.random_policy`) on the `CartPole-v1` environment. So far, I have build a custom trainer and want to run it via the `ray.tune.run` interface. Unfortunately, the execution fails with the error message:

```auto
File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 471, in load_batch_into_buffer
raise NotImplementedError

```

I’ve looked into overriding the `Policy` class (`ray.rllib.policy.policy`) but am unsure how to implement the methods `load_batch_into_buffer`, `get_num_samples_loaded_into_buffer` etc. This seems to be an unnecessary complicated way to do something so simple? Any suggestions or help would greatly be appreciated! (Ray version 1.10.0)

The minimal code to reproduce this issue is below:

```python
import ray
from ray.rllib.examples.policy.random_policy import RandomPolicy
from ray.rllib.agents.trainer_template import build_trainer

RandomTrainer = build_trainer(
name="RandomTrainer",
default_policy=RandomPolicy
)
  
config = {"env": "CartPole-v1", "num_gpus": 0, "num_workers": 1, "framework": "torch"}

ray.tune.run(
  RandomTrainer,
  config=config,
  stop={"timesteps_total": 5},
)

```

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [February 25, 2022, 8:23pm UTC](https://discuss.ray.io/t/running-a-random-policy-with-rllib/5198/2 "2022-02-25T20:23:18Z")

</div>

Hi @sn73jq ,

and welcome to the ray discussion board. I have tested your code and have made some modification such that it runs:

```auto
import ray
from ray.rllib.examples.policy.random_policy import RandomPolicy
from ray.rllib.agents.trainer_template import build_trainer

RandomTrainer = build_trainer(
name="RandomTrainer",
default_policy=RandomPolicy
)
  
config = {"env": "CartPole-v1", "num_gpus": 0, "num_workers": 1, "framework": None}
ray.init(ignore_reinit_error=True, local_mode=True)
ray.tune.run(
  RandomTrainer,
  config=config,
  stop={"timesteps_total": 5},
)
ray.shutdown()

```

First of all `RandomPolicy` is a policy that cannot be trained (we have no model that helps to make better decisions as actions are always randomly chosen). Using `torch` as the `framework` will automatically set `MultiGPUTrainOneStep()` for training and this training step requires a function that bulk-loads the data into the different devices’ memories (that makes sense for `torch` or `tf`), however the `RandomPolicy` has not such function defined and the super-class `Policy` neither (it returns a [`NotImplementedError`](https://github.com/ray-project/ray/blob/7f1bacc7dc9caf6d0ec042e39499bbf1d9a7d065/rllib/policy/policy.py#L522)). As a result you receive a `NotImplementedError`.

When the `Trainer` is initialized it validates its `config` and therein it sets the `simple_optimizer` parameter to `True` (from `DEPRECATED_VALUE`, @sven1977, @avnishn what is the plan here?) when the `framework` is not `tf` or `torch`, which in turn will let the `Trainer` use `TrainOneStep()` instead of `MultiGPUTrainOneStep()`. The former function, however, does not need the `load_batch_into_buffer()` be defined and runs through. Setting `framework` to `None` implies exactly this behavior (setting directly `simple_optimizer: True` of course as well).

I admit that this behavior does not make much sense, but has right now historical reasons as the RLlib team is modifying a lot by integrating `ray.train` for training. I guess in the next releases these things will become more well-arranged (also more powerful 🙂 )