# The “trajectory\_view\_api” does not support the DQN algorithm, and the program will run in error

**URL:** <https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085>\
**Category:** RLlib\
**Created:** [August 5, 2022, 3:11am UTC](https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085 "2022-08-05T03:11:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hawk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hawk/32/2979_2.png) [@hawk](https://discuss.ray.io/u/hawk)\
**Post date:** [August 5, 2022, 3:11am UTC](https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085/1 "2022-08-05T03:11:15Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

When I use the default **“trajectory\_view\_api”** functionality, the file path is

> “C:\ProgramData\Anaconda3\Lib\site-packages\ray\rllib\examples\trajectory\_view\_api.py”

the default algorithm used is PPO, and the program can run normally.  
However, when I change the default algorithm to **DQN** , the program cannot run and the following error occurs(Please refer to the reply on the first floor).  
I have changed the:

> “–run”, type=str, default=“PPO”, help=“The RLlib-registered algorithm to use.”

to:

> “–run”, type=str, default=“DQN”, help=“The RLlib-registered algorithm to use.”

and, the following two parameters of PPO algorithm are deleted:

> #“num\_sgd\_iter”: 5,  
> #“vf\_loss\_coeff”: 0.0001,

I don’t understand the cause of the error, is that the DQN algorithm is off-policy and does not support the trajectory\_view\_api? Or is it something else? If you know the specific reason, I would really appreciate your answer.

---

<div class="post-metadata">

**Author:** ![hawk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hawk/32/2979_2.png) [@hawk](https://discuss.ray.io/u/hawk)\
**Post date:** [August 5, 2022, 3:12am UTC](https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085/2 "2022-08-05T03:12:07Z")

</div>

Here is the error:

> 2022-08-05 10:51:56,358 ERROR ray\_trial\_executor.py:102 – An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\tune\ray\_trial\_executor.py”, line 93, in post\_stop\_cleanup  
> ray.get(future, timeout=0)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\_private\client\_mode\_hook.py”, line 105, in wrapper  
> return func(\*args, \*\*kwargs)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\worker.py”, line 1811, in get  
> raise value  
> File “python\ray\_raylet.pyx”, line 797, in ray.\_raylet.task\_execution\_handler  
> File “python\ray\_raylet.pyx”, line 616, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 760, in ray.\_raylet.execute\_task  
> ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::DQNTrainer. **init** () (pid=10264, ip=127.0.0.1, repr=DQNTrainer)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> return method(self, \*\_args, \*\*\_kwargs)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 1035, in \_init  
> raise NotImplementedError  
> NotImplementedError
> 
> During handling of the above exception, another exception occurred:
> 
> ray::DQNTrainer. **init** () (pid=10264, ip=127.0.0.1, repr=DQNTrainer)  
> File “python\ray\_raylet.pyx”, line 656, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 697, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 663, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 667, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 614, in ray.\_raylet.execute\_task.function\_executor  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\_private\function\_manager.py”, line 701, in actor\_method\_executor  
> return method(\_\_ray\_actor, \*args, \*\*kwargs)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> return method(self, \*\_args, \*\*\_kwargs)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 830, in **init**  
> super(). **init** (  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\tune\trainable.py”, line 149, in **init**  
> self.setup(copy.deepcopy(self.config))  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> return method(self, \*\_args, \*\*\_kwargs)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 911, in setup  
> self.workers = WorkerSet(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\worker\_set.py”, line 162, in **init**  
> self.\_local\_worker = self.\_make\_worker(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\worker\_set.py”, line 567, in \_make\_worker  
> worker = cls(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\rollout\_worker.py”, line 626, in **init**  
> self.\_build\_policy\_map(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\rollout\_worker.py”, line 1722, in _build\_policy\_map  
> self.policy\_map.create\_policy(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\policy\_map.py”, line 140, in create\_policy  
> self[policy\_id] = class_(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\tf\_policy\_template.py”, line 256, in **init**  
> DynamicTFPolicy. **init** (  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 439, in **init**  
> self.\_initialize\_loss\_from\_dummy\_batch(auto\_remove\_unneeded\_view\_reqs=True)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 758, in \_initialize\_loss\_from\_dummy\_batch  
> losses = self.\_do\_loss\_init(train\_batch)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 867, in \_do\_loss\_init  
> losses = self.\_loss\_fn(self, self.model, self.dist\_class, train\_batch)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\dqn\dqn\_tf\_policy.py”, line 251, in build\_q\_losses  
> q\_t, q\_logits\_t, q\_dist\_t, \_ = compute\_q\_values(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\dqn\dqn\_tf\_policy.py”, line 390, in compute\_q\_values  
> model\_out, state = model(input\_batch, state\_batches or , seq\_lens)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\models\modelv2.py”, line 251, in **call**  
> res = self.forward(restored, state or , seq\_lens)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\examples\models\trajectory\_view\_utilizing\_models.py”, line 63, in forward  
> obs = tf.cast(input\_dict[“prev\_n\_obs”], tf.float32)  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\sample\_batch.py”, line 744, in **getitem**  
> value = dict. **getitem** (self, key)  
> KeyError: ‘prev\_n\_obs’
> 
> (DQNTrainer pid=10264) 2022-08-05 10:51:56,333 ERROR worker.py:449 – Exception raised in creation task: The actor died because of an error raised in its creation task, ray::DQNTrainer. **init** () (pid=10264, ip=127.0.0.1, repr=DQNTrainer)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> (DQNTrainer pid=10264) return method(self, \*\_args, \*\*\_kwargs)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 1035, in \_init  
> (DQNTrainer pid=10264) raise NotImplementedError  
> (DQNTrainer pid=10264) NotImplementedError  
> (DQNTrainer pid=10264)  
> (DQNTrainer pid=10264) During handling of the above exception, another exception occurred:  
> (DQNTrainer pid=10264)  
> (DQNTrainer pid=10264) ray::DQNTrainer. **init** () (pid=10264, ip=127.0.0.1, repr=DQNTrainer)  
> (DQNTrainer pid=10264) File “python\ray\_raylet.pyx”, line 656, in ray.\_raylet.execute\_task  
> (DQNTrainer pid=10264) File “python\ray\_raylet.pyx”, line 697, in ray.\_raylet.execute\_task  
> (DQNTrainer pid=10264) File “python\ray\_raylet.pyx”, line 663, in ray.\_raylet.execute\_task  
> (DQNTrainer pid=10264) File “python\ray\_raylet.pyx”, line 667, in ray.\_raylet.execute\_task  
> (DQNTrainer pid=10264) File “python\ray\_raylet.pyx”, line 614, in ray.\_raylet.execute\_task.function\_executor  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\_private\function\_manager.py”, line 701, in actor\_method\_executor  
> (DQNTrainer pid=10264) return method(\_\_ray\_actor, \*args, \*\*kwargs)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> (DQNTrainer pid=10264) return method(self, \*\_args, \*\*\_kwargs)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 830, in **init**  
> (DQNTrainer pid=10264) super(). **init** (  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\tune\trainable.py”, line 149, in **init**  
> (DQNTrainer pid=10264) self.setup(copy.deepcopy(self.config))  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\util\tracing\tracing\_helper.py”, line 462, in \_resume\_span  
> (DQNTrainer pid=10264) return method(self, \*\_args, \*\*\_kwargs)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\trainer.py”, line 911, in setup  
> (DQNTrainer pid=10264) self.workers = WorkerSet(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\worker\_set.py”, line 162, in **init**  
> (DQNTrainer pid=10264) self.\_local\_worker = self.\_make\_worker(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\worker\_set.py”, line 567, in \_make\_worker  
> (DQNTrainer pid=10264) worker = cls(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\rollout\_worker.py”, line 626, in **init**  
> (DQNTrainer pid=10264) self.\_build\_policy\_map(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\evaluation\rollout\_worker.py”, line 1722, in _build\_policy\_map  
> (DQNTrainer pid=10264) self.policy\_map.create\_policy(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\policy\_map.py”, line 140, in create\_policy  
> (DQNTrainer pid=10264) self[policy\_id] = class_(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\tf\_policy\_template.py”, line 256, in **init**  
> (DQNTrainer pid=10264) DynamicTFPolicy. **init** (  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 439, in **init**  
> (DQNTrainer pid=10264) self.\_initialize\_loss\_from\_dummy\_batch(auto\_remove\_unneeded\_view\_reqs=True)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 758, in \_initialize\_loss\_from\_dummy\_batch  
> (DQNTrainer pid=10264) losses = self.\_do\_loss\_init(train\_batch)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\dynamic\_tf\_policy.py”, line 867, in \_do\_loss\_init  
> (DQNTrainer pid=10264) losses = self.\_loss\_fn(self, self.model, self.dist\_class, train\_batch)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\dqn\dqn\_tf\_policy.py”, line 251, in build\_q\_losses  
> (DQNTrainer pid=10264) q\_t, q\_logits\_t, q\_dist\_t, \_ = compute\_q\_values(  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\agents\dqn\dqn\_tf\_policy.py”, line 390, in compute\_q\_values  
> (DQNTrainer pid=10264) model\_out, state = model(input\_batch, state\_batches or , seq\_lens)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\models\modelv2.py”, line 251, in **call**  
> (DQNTrainer pid=10264) res = self.forward(restored, state or , seq\_lens)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\examples\models\trajectory\_view\_utilizing\_models.py”, line 63, in forward  
> (DQNTrainer pid=10264) obs = tf.cast(input\_dict[“prev\_n\_obs”], tf.float32)  
> (DQNTrainer pid=10264) File “C:\ProgramData\Anaconda3\lib\site-packages\ray\rllib\policy\sample\_batch.py”, line 744, in **getitem**  
> (DQNTrainer pid=10264) value = dict. **getitem** (self, key)  
> (DQNTrainer pid=10264) KeyError: ‘prev\_n\_obs’  
> Traceback (most recent call last):  
> File “C:\ProgramData\Anaconda3\Lib\site-packages\ray\rllib\examples\trajectory\_view\_api.py”, line 85, in   
> results = tune.run(  
> File “C:\ProgramData\Anaconda3\lib\site-packages\ray\tune\tune.py”, line 695, in run  
> raise TuneError(“Trials did not complete”, incomplete\_trials)  
> ray.tune.error.TuneError: (‘Trials did not complete’, [DQN\_StatelessCartPole\_8dd05\_00000])

---

<div class="post-metadata">

**Author:** ![hawk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hawk/32/2979_2.png) [@hawk](https://discuss.ray.io/u/hawk)\
**Post date:** [August 5, 2022, 3:14am UTC](https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085/4 "2022-08-05T03:14:34Z")

</div>

Here is the code of the “trajectory\_view\_api.py”:

> import argparse  
> import numpy as np
> 
> import ray  
> from ray.rllib.agents.ppo import PPOTrainer  
> from ray.rllib.examples.env.stateless\_cartpole import StatelessCartPole  
> from ray.rllib.examples.models.trajectory\_view\_utilizing\_models import (  
> FrameStackingCartPoleModel,  
> TorchFrameStackingCartPoleModel,  
> )  
> from ray.rllib.models.catalog import ModelCatalog  
> from ray.rllib.utils.framework import try\_import\_tf  
> from ray.rllib.utils.test\_utils import check\_learning\_achieved  
> from ray import tune
> 
> tf1, tf, tfv = try\_import\_tf()
> 
> parser = argparse.ArgumentParser()  
> parser.add\_argument(  
> “–run”, type=str, default=“DQN”, help=“The RLlib-registered algorithm to use.”  
> )  
> parser.add\_argument(  
> “–framework”,  
> choices=[“tf”, “tf2”, “tfe”, “torch”],  
> default=“tf”,  
> help=“The DL framework specifier.”,  
> )  
> parser.add\_argument(  
> “–as-test”,  
> action=“store\_true”,  
> help="Whether this script should be run as a test: --stop-reward must "  
> “be achieved within --stop-timesteps AND --stop-iters.”,  
> )  
> parser.add\_argument(  
> “–stop-iters”, type=int, default=50, help=“Number of iterations to train.”  
> )  
> parser.add\_argument(  
> “–stop-timesteps”, type=int, default=200000, help=“Number of timesteps to train.”  
> )  
> parser.add\_argument(  
> “–stop-reward”, type=float, default=150.0, help=“Reward at which we stop training.”  
> )
> 
> if **name** == “ **main** ”:  
> args = parser.parse\_args()  
> ray.init(num\_cpus=3)
> 
> ```
> num_frames = 16
> 
> ModelCatalog.register_custom_model(
> "frame_stack_model",
> FrameStackingCartPoleModel
> if args.framework != "torch"
> else TorchFrameStackingCartPoleModel,
> )
> 
> config = {
> "env": StatelessCartPole,
> "model": {
> "vf_share_layers": True,
> "custom_model": "frame_stack_model",
> "custom_model_config": {
> "num_frames": num_frames,
> },
> # To compare against a simple LSTM:
> # "use_lstm": True,
> # "lstm_use_prev_action": True,
> # "lstm_use_prev_reward": True,
> # To compare against a simple attention net:
> # "use_attention": True,
> # "attention_use_n_prev_actions": 1,
> # "attention_use_n_prev_rewards": 1,
> },
> #"num_sgd_iter": 5,
> #"vf_loss_coeff": 0.0001,
> "framework": args.framework,
> }
> 
> stop = {
> "training_iteration": args.stop_iters,
> "timesteps_total": args.stop_timesteps,
> "episode_reward_mean": args.stop_reward,
> }
> results = tune.run(
> args.run, config=config, stop=stop, verbose=2, checkpoint_at_end=True
> )
> 
> if args.as_test:
> check_learning_achieved(results, args.stop_reward)
> 
> checkpoints = results.get_trial_checkpoints_paths(
> trial=results.get_best_trial("episode_reward_mean", mode="max"),
> metric="episode_reward_mean",
> )
> 
> checkpoint_path = checkpoints[0][0]
> trainer = PPOTrainer(config)
> trainer.restore(checkpoint_path)
> 
> # Inference loop.
> env = StatelessCartPole()
> 
> # Run manual inference loop for n episodes.
> for _ in range(10):
> episode_reward = 0.0
> reward = 0.0
> action = 0
> done = False
> obs = env.reset()
> while not done:
> # Create a dummy action using the same observation n times,
> # as well as dummy prev-n-actions and prev-n-rewards.
> action, state, logits = trainer.compute_single_action(
> input_dict={
> "obs": obs,
> "prev_n_obs": np.stack([obs for _ in range(num_frames)]),
> "prev_n_actions": np.stack([0 for _ in range(num_frames)]),
> "prev_n_rewards": np.stack([1.0 for _ in range(num_frames)]),
> },
> full_fetch=True,
> )
> obs, reward, done, info = env.step(action)
> episode_reward += reward
> 
> print(f"Episode reward={episode_reward}")
> 
> ray.shutdown()
> 
> ```

---

<div class="post-metadata">

**Author:** ![hawk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hawk/32/2979_2.png) [@hawk](https://discuss.ray.io/u/hawk)\
**Post date:** [August 7, 2022, 7:37am UTC](https://discuss.ray.io/t/the-trajectory-view-api-does-not-support-the-dqn-algorithm-and-the-program-will-run-in-error/7085/5 "2022-08-07T07:37:14Z")

</div>

@ [sven1977](https://discuss.ray.io/u/sven1977)@ [mannyv](https://discuss.ray.io/u/mannyv)@ [arturn](https://discuss.ray.io/u/arturn)  
I’m sorry to bother you. I’m really anxious to solve this problem. Could you please take some time to look at this problem？