Feeding issue for timestep placeholder in Ray 1.0.1.post1

Hi Ray team,
I have used Ray 1.0.0 and am considering upgrade the version to 1.0.1.post1 now.
I used the same training code and the same prediction code.
but I got the below issue when I try to predict using the model trained by Ray 1.0.1.post1.
(the model with Ray 1.0.0 is working well though).

Exception during discrete event execution:
You must feed a value for placeholder tensor 'default_policy/timestep' with dtype int64
	 [[{{node default_policy/timestep}}]]
org.tensorflow.exceptions.TFInvalidArgumentException: You must feed a value for placeholder tensor 'default_policy/timestep' with dtype int64
	 [[{{node default_policy/timestep}}]]
	at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:87)
	at org.tensorflow.Session.run(Session.java:666)
	at org.tensorflow.Session.access$100(Session.java:72)
	at org.tensorflow.Session$Runner.runHelper(Session.java:381)
	at org.tensorflow.Session$Runner.run(Session.java:329)

was there a change w.r.t. default_policy/timestep ?

@sven1977 Hi Sven,
I think [RLlib] Problem with TFModelV2 loading after having saved one with `TFPolicy.export_model()` - #2 by morsias is a similar issue.
Looks like you need to have a reproducible script, please check it out mine.

here’s my test code.

with Ray 1.0.0,
executed my test code and try to load the model.
tf.saved_model.load(‘model_1_0_0’) works well.

with Ray 1.0.1.post1
executed my test code and try to load the model.
tf.saved_model.load(‘model_1_0_1_post1’) doesn’t work.

here’s error message.

Hey @kepricon , sorry I missed this. Thanks for the repro script. Taking a look now …

1 Like

I cannot reproduce this on the latest master. Here is what I did:

  1. run (your script):
import random

import ray
from ray.tune import run, sample_from
from ray.tune.schedulers import PopulationBasedTraining

if __name__ == "__main__":
    class Stopper:
        def __init__(self):
            self.too_many_iter = False

        def stop(self, trial_id, result):
            self.too_many_iter = result['training_iteration'] >= 10

            if self.too_many_iter:
                return True

    # Postprocess the perturbed config to ensure it's still valid
    def explore(config):
        if config["train_batch_size"] < config["sgd_minibatch_size"] * 2:
            config["train_batch_size"] = config["sgd_minibatch_size"] * 2
        if config["num_sgd_iter"] < 1:
            config["num_sgd_iter"] = 1
        return config

    pbt = PopulationBasedTraining(
        time_attr="time_total_s",
        metric="episode_reward_mean",
        mode="max",
        perturbation_interval=120,
        resample_probability=0.25,
        # Specifies the mutations of these hyperparams
        hyperparam_mutations={
            "lambda": lambda: random.uniform(0.9, 1.0),
            "clip_param": lambda: random.uniform(0.01, 0.5),
            "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
            "num_sgd_iter": lambda: random.randint(1, 30),
            "sgd_minibatch_size": lambda: random.randint(128, 16384),
            "train_batch_size": lambda: random.randint(200, 1600),
        },
        custom_explore_fn=explore)

    ray.init()
    run(
        "PPO",
        name="cartpole",
        scheduler=pbt,
        num_samples=1,
        config={
            "env": "CartPole-v0",
            "kl_coeff": 1.0,
            "num_workers": 1,
            "num_gpus": 0,
            "model": {
                "free_log_std": True
            },
            # These params are tuned from a fixed starting value.
            "lambda": 0.95,
            "clip_param": 0.2,
            "lr": 1e-4,
            # These params start off randomly drawn from a set.
            "num_sgd_iter": sample_from(
                lambda spec: random.choice([10, 20])),
            "sgd_minibatch_size": sample_from(
                lambda spec: random.choice([128, 512])),
            "train_batch_size": sample_from(
                lambda spec: random.choice([1000, 2000]))
        },
        stop = Stopper().stop,
        local_dir = '/tmp/PPO',
        export_formats = ['model']
    )

  1. Check, whether the model has been stored (all ok).
  2. run:
    import tensorflow as tf
    result = tf.saved_model.load("/tmp/PPO/cartpole/[path-to-/model/]")
1 Like

Hi @sven1977

thank you for your comments.
I tried to use the current master that I built from my local.
It looks like it doesn’t have the issue.

/model was generated and I could load the model.
At the end of training model via my script, It said the below error though.

Do you happen to have this error message from your tests?

2021-02-18 11:05:05,076	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.
(pid=23161) 2021-02-18 11:05:05,069	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.
(pid=23161) 2021-02-18 11:05:05,070	ERROR worker.py:74 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): The actor died unexpectedly before finishing this task. Check python-core-worker-*.log files for more information.

I tried to test on Ray 1.2.0
but I had no luck installing Ray 1.2.0 on my Linux machine(Ubuntu 16.04).

(conda) kepricon@kepricon-G751JL:~/conda$ pip install ray[rllib]==1.2.0
Collecting ray[rllib]==1.2.0
  Could not find a version that satisfies the requirement ray[rllib]==1.2.0 (from versions: 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 1.0.0rc0, 1.0.0rc1, 1.0.0rc2, 1.0.0, 1.0.1, 1.0.1.post1)

Does Ray support installation new version on Linux?

Thank you.

1 Like

Hi @sven1977
This issue doesn’t happen on ray[rllib]==1.2.0
I just tested it and it worked well.

Thank you.

1 Like

Awesome! Thanks for the feedback. Glad it’s working on 1.2. Yes, we may have fixed this in one of the recent PRs (based on some other user’s github issue?).