# Definition of "one timestep" when using ExternalEnv

**URL:** <https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546>\
**Category:** RLlib\
**Created:** [June 19, 2022, 9:26am UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546 "2022-06-19T09:26:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![phipag](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/phipag/32/2756_2.png) [@phipag](https://discuss.ray.io/u/phipag)\
**Post date:** [June 19, 2022, 9:26am UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546/1 "2022-06-19T09:26:00Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Low: It annoys or frustrates me for a moment.

I am training a DQN agent using the PolicyServer / PolicyClient with the default ExternalEnv similar to the cartpole example.

From my external simulator (written in Java) I call a wrapper API hosted with Ray Serve which uses the PolicyClient to interact with the PolicyServer. My training loop is as follows:

1. client.start\_episode
2. client.get\_action
3. client.log\_returns (12 times)
4. client.get\_action
5. client.log\_returns (12 times)
6. client.end\_episode

Between each client.get\_action call I log the returns 12 times because the simulator takes a few timesteps until the full consequences of the action are visible. This can make sense because ExternalEnv accumulates the rewards until the next client.get\_action call, correct?

My main question is how a “timestep” is defined in this scenario. Does one episode in this example consist of 2 timesteps (because get\_action is called twice) or 24 timesteps because 24 returns are logged?

I am asking because I am confused by the “learning\_starts” and “timesteps\_per\_iteration” options in the DQN config.

---

<div class="post-metadata">

**Author:** ![rusu24edward](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rusu24edward/32/333_2.png) [@rusu24edward](https://discuss.ray.io/u/rusu24edward)\
**Post date:** [June 21, 2022, 5:55pm UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546/2 "2022-06-21T17:55:33Z")

</div>

Why not just delay your call to `get_action` until the sim gets to a state where it’s ready for the next action?

---

<div class="post-metadata">

**Author:** ![phipag](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/phipag/32/2756_2.png) [@phipag](https://discuss.ray.io/u/phipag)\
**Post date:** [June 22, 2022, 8:18am UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546/3 "2022-06-22T08:18:31Z")

</div>

This is what I am doing right now. I call get\_action and wait until it is ready for the next action. And while it waits it logs the returns it gets from the environment.

But my question is if one timestep equals one call to get\_action or one call to log\_returns?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [June 22, 2022, 10:30am UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546/4 "2022-06-22T10:30:51Z")

</div>

One time step equals a call to get/log actions. Logging returns does not advance the step.

There is one extra detail in multiagent configs where you can count steps by the number of environment steps or the number of agent steps. I do not think this was your question though. You can find more detail if you are interested in the link below. Search for **multiagent.count\_steps\_by**.  
[https://docs.ray.io/en/latest/rllib/rllib-sample-collection.html#the-samplecollector-class-is-used-to-store-and-retrieve-temporary-data](https://docs.ray.io/en/latest/rllib/rllib-sample-collection.html#the-samplecollector-class-is-used-to-store-and-retrieve-temporary-data)

Here is where the commands are handled:

> <https://github.com/ray-project/ray/blob/f18fc31c7562990955556899090f8e8656b48d2d/rllib/env/policy_server_input.py#L211-L243>

Here is the start of log\_returns:

> <https://github.com/ray-project/ray/blob/b5bc2b93c33f0f475af69dd6eca656dcf264612d/rllib/env/external_env.py#L154>

---

<div class="post-metadata">

**Author:** ![phipag](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/phipag/32/2756_2.png) [@phipag](https://discuss.ray.io/u/phipag)\
**Post date:** [June 23, 2022, 1:20pm UTC](https://discuss.ray.io/t/definition-of-one-timestep-when-using-externalenv/6546/5 "2022-06-23T13:20:34Z")

</div>

Thank you, this clarifies my question.