# Feeding issue for timestep placeholder in Ray 1.0.1.post1

**URL:** <https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715>\
**Category:** RLlib\
**Created:** [February 3, 2021, 8:02pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715 "2021-02-03T20:02:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![kepricon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kepricon/32/378_2.png) [@kepricon](https://discuss.ray.io/u/kepricon)\
**Post date:** [February 3, 2021, 8:02pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/1 "2021-02-03T20:02:03Z")

</div>

Hi Ray team,  
I have used Ray 1.0.0 and am considering upgrade the version to 1.0.1.post1 now.  
I used the same training code and the same prediction code.  
but I got the below issue when I try to predict using the model trained by Ray 1.0.1.post1.  
(the model with Ray 1.0.0 is working well though).

```
Exception during discrete event execution:
You must feed a value for placeholder tensor 'default_policy/timestep' with dtype int64
	 [[{{node default_policy/timestep}}]]
org.tensorflow.exceptions.TFInvalidArgumentException: You must feed a value for placeholder tensor 'default_policy/timestep' with dtype int64
	 [[{{node default_policy/timestep}}]]
	at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:87)
	at org.tensorflow.Session.run(Session.java:666)
	at org.tensorflow.Session.access$100(Session.java:72)
	at org.tensorflow.Session$Runner.runHelper(Session.java:381)
	at org.tensorflow.Session$Runner.run(Session.java:329)

```

was there a change w.r.t. `default_policy/timestep` ?

---

<div class="post-metadata">

**Author:** ![kepricon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kepricon/32/378_2.png) [@kepricon](https://discuss.ray.io/u/kepricon)\
**Post date:** [February 8, 2021, 7:19pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/2 "2021-02-08T19:19:16Z")

</div>

@sven1977 Hi Sven,  
I think [[RLlib] Problem with TFModelV2 loading after having saved one with `TFPolicy.export_model()` - #2 by morsias](https://discuss.ray.io/t/rllib-problem-with-tfmodelv2-loading-after-having-saved-one-with-tfpolicy-export-model/745/2) is a similar issue.  
Looks like you need to have a reproducible script, please check it out mine.

here’s my test code.

> <https://gist.github.com/kepricon/516800ad7d8ef1b0b23df429c4d49490>

with Ray 1.0.0,  
executed my test code and try to load the model.  
tf.saved\_model.load(‘model\_1\_0\_0’) works well.

with Ray 1.0.1.post1  
executed my test code and try to load the model.  
tf.saved\_model.load(‘model\_1\_0\_1\_post1’) doesn’t work.

---

<div class="post-metadata">

**Author:** ![kepricon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kepricon/32/378_2.png) [@kepricon](https://discuss.ray.io/u/kepricon)\
**Post date:** [February 8, 2021, 7:20pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/3 "2021-02-08T19:20:17Z")

</div>

here’s error message.

> <https://gist.github.com/kepricon/d0204fcd3b497d2aed2ac4d11eb47f5b>

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [February 18, 2021, 3:41pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/4 "2021-02-18T15:41:02Z")

</div>

Hey @kepricon , sorry I missed this. Thanks for the repro script. Taking a look now …

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [February 18, 2021, 4:00pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/5 "2021-02-18T16:00:31Z")

</div>

I cannot reproduce this on the latest master. Here is what I did:

1. run (your script):

```auto
import random

import ray
from ray.tune import run, sample_from
from ray.tune.schedulers import PopulationBasedTraining

if __name__ == " __main__":
    class Stopper:
        def __init__ (self):
            self.too_many_iter = False

        def stop(self, trial_id, result):
            self.too_many_iter = result['training_iteration'] >= 10

            if self.too_many_iter:
                return True

    # Postprocess the perturbed config to ensure it's still valid
    def explore(config):
        if config["train_batch_size"] < config["sgd_minibatch_size"] * 2:
            config["train_batch_size"] = config["sgd_minibatch_size"] * 2
        if config["num_sgd_iter"] < 1:
            config["num_sgd_iter"] = 1
        return config

    pbt = PopulationBasedTraining(
        time_attr="time_total_s",
        metric="episode_reward_mean",
        mode="max",
        perturbation_interval=120,
        resample_probability=0.25,
        # Specifies the mutations of these hyperparams
        hyperparam_mutations={
            "lambda": lambda: random.uniform(0.9, 1.0),
            "clip_param": lambda: random.uniform(0.01, 0.5),
            "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
            "num_sgd_iter": lambda: random.randint(1, 30),
            "sgd_minibatch_size": lambda: random.randint(128, 16384),
            "train_batch_size": lambda: random.randint(200, 1600),
        },
        custom_explore_fn=explore)

    ray.init()
    run(
        "PPO",
        name="cartpole",
        scheduler=pbt,
        num_samples=1,
        config={
            "env": "CartPole-v0",
            "kl_coeff": 1.0,
            "num_workers": 1,
            "num_gpus": 0,
            "model": {
                "free_log_std": True
            },
            # These params are tuned from a fixed starting value.
            "lambda": 0.95,
            "clip_param": 0.2,
            "lr": 1e-4,
            # These params start off randomly drawn from a set.
            "num_sgd_iter": sample_from(
                lambda spec: random.choice([10, 20])),
            "sgd_minibatch_size": sample_from(
                lambda spec: random.choice([128, 512])),
            "train_batch_size": sample_from(
                lambda spec: random.choice([1000, 2000]))
        },
        stop = Stopper().stop,
        local_dir = '/tmp/PPO',
        export_formats = ['model']
    )

```

1. Check, whether the model has been stored (all ok).
2. run:

```auto
    import tensorflow as tf
    result = tf.saved_model.load("/tmp/PPO/cartpole/[path-to-/model/]")

```

---

<div class="post-metadata">

**Author:** ![kepricon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kepricon/32/378_2.png) [@kepricon](https://discuss.ray.io/u/kepricon)\
**Post date:** [February 18, 2021, 7:20pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/6 "2021-02-18T19:20:29Z")

</div>

Hi @sven1977

thank you for your comments.  
I tried to use the current master that I built from my local.  
It looks like it doesn’t have the issue.

`/model` was generated and I could load the model.  
At the end of training model via my script, It said the below error though.

Do you happen to have this error message from your tests?

```auto
2021-02-18 11:05:05,076	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.
(pid=23161) 2021-02-18 11:05:05,069	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.
(pid=23161) 2021-02-18 11:05:05,070	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.

```

I tried to test on [Ray 1.2.0](https://github.com/ray-project/ray/releases/tag/ray-1.2.0)  
but I had no luck installing Ray 1.2.0 on my Linux machine(Ubuntu 16.04).

```auto
(conda) kepricon@kepricon-G751JL:~/conda$ pip install ray[rllib]==1.2.0
Collecting ray[rllib]==1.2.0
  Could not find a version that satisfies the requirement ray[rllib]==1.2.0 (from versions: 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 1.0.0rc0, 1.0.0rc1, 1.0.0rc2, 1.0.0, 1.0.1, 1.0.1.post1)

```

Does Ray support installation new version on Linux?

Thank you.

---

<div class="post-metadata">

**Author:** ![kepricon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kepricon/32/378_2.png) [@kepricon](https://discuss.ray.io/u/kepricon)\
**Post date:** [February 23, 2021, 7:23pm UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/7 "2021-02-23T19:23:45Z")

</div>

Hi @sven1977  
This issue doesn’t happen on ray[rllib]==1.2.0  
I just tested it and it worked well.

Thank you.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [February 24, 2021, 9:03am UTC](https://discuss.ray.io/t/feeding-issue-for-timestep-placeholder-in-ray-1-0-1-post1/715/8 "2021-02-24T09:03:49Z")

</div>

Awesome! Thanks for the feedback. Glad it’s working on 1.2. Yes, we may have fixed this in one of the recent PRs (based on some other user’s github issue?).