# Example of A3C only use CPU for trainer **URL:** \ **Category:** RLlib\ **Created:** [July 22, 2021, 6:26pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984 "2021-07-22T18:26:06Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![daniel](https://avatars.discourse-cdn.com/v4/letter/d/3ec8ea/32.png) [@daniel](https://discuss.ray.io/u/daniel)\ **Post date:** [July 22, 2021, 6:26pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/1 "2021-07-22T18:26:06Z")
When I follow the tutorial at [RLlib: Scalable Reinforcement Learning — Ray v1.4.1](https://docs.ray.io/en/master/rllib.html#running-rllib) running ```auto from ray import tune from ray.rllib.agents.ppo import PPOTrainer tune.run(PPOTrainer, config={"env": "CartPole-v0"}) ``` and ```auto rllib train --run=PPO --env=CartPole-v0 # -v [-vv] for verbose, # --config='{"framework": "tf2", "eager_tracing": True}' for eager, # --torch to use PyTorch OR --config='{"framework": "torch"}' ``` can only use CPU for training, is there any configuration setting that I am missing for GPU training? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 1:10pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/2 "2021-07-23T13:10:52Z")
Hi @daniel, Try this and see if it works: ```auto rllib train --run=PPO --env=CartPole-v0 --ray-num-gpus=1 --config='{"num_gpus": 1}' ``` ---
**Author:** ![daniel](https://avatars.discourse-cdn.com/v4/letter/d/3ec8ea/32.png) [@daniel](https://discuss.ray.io/u/daniel)\ **Post date:** [July 23, 2021, 3:55pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/3 "2021-07-23T15:55:05Z")
In the case of `rllib/agents/a3c/tests/test_a3c.py` when I change the test example to ```auto def test_a3c_compilation(self): """Test whether an A3CTrainer can be built with both frameworks.""" config = a3c.DEFAULT_CONFIG.copy() config["num_workers"] = 2 config["num_envs_per_worker"] = 2 config["framework"] = "torch" config["num_gpus"] = 1 num_iterations = 100 # Test against all frameworks. for _ in framework_iterator(config): # for env in ["CartPole-v0", "Pendulum-v0", "PongDeterministic-v0"]: for env in ["CartPole-v0", "Pendulum-v0", "PongDeterministic-v0"]: print("env={}".format(env)) # sys.stdout.write("world\n") trainer = a3c.A3CTrainer(config=config, env=env) for i in tqdm(range(num_iterations)): results = trainer.train() # print(results) # check_compute_single_action(trainer) trainer.stop() ``` I return me something like ```auto else: logger.info("TorchPolicy (worker={}) running on {} GPU(s).".format( worker_idx if worker_idx > 0 else "local", config["num_gpus"])) gpu_ids = ray.get_gpu_ids() self.devices = [ torch.device("cuda:{}".format(i)) for i, id_ in enumerate(gpu_ids) if i < config["num_gpus"] ] > self.device = self.devices[0] E IndexError: list index out of range ``` It seems that the ray could not locate the available GPU in this case. How should we use GPU for training A3C? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 3:58pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/4 "2021-07-23T15:58:12Z")
@daniel, Try changing this part to: ```auto def setUp(self): ray.init(num_cpus=4, num_gpus=1) ``` There have been some bugs reported about detecting gpus in torch. You could search the issues in github if this does not work. ---
**Author:** ![daniel](https://avatars.discourse-cdn.com/v4/letter/d/3ec8ea/32.png) [@daniel](https://discuss.ray.io/u/daniel)\ **Post date:** [July 23, 2021, 4:02pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/5 "2021-07-23T16:02:31Z")
> [@mannyv](#): > > ```auto > ray.init(num_cpus=4, num_gpus=1) > > ``` After changing this line, I still get the same error as above, is there any other solution I can try? Thanks! ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 4:26pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/6 "2021-07-23T16:26:05Z")
Hi @daniel, I think the first thing to do, and maybe you have already done this is to test that pytorch is able to see your gpu in the python interpreter from the (venv, conda environment, container, …) you run ray in. Something like: ```python import torch print(torch.cuda.is_available(), torch.cuda.current_device(), torch.cuda.device(0), torch.cuda.device_count(), torch.cuda.get_device_name(0)) ``` ---
**Author:** ![daniel](https://avatars.discourse-cdn.com/v4/letter/d/3ec8ea/32.png) [@daniel](https://discuss.ray.io/u/daniel)\ **Post date:** [July 23, 2021, 4:27pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/7 "2021-07-23T16:27:38Z")
> [@mannyv](#): > > ```auto > print(torch.cuda.is_available(), torch.cuda.current_device(), torch.cuda.device(0), torch.cuda.device_count(), torch.cuda.get_device_name(0)) > > ``` Here is the returned information ```auto True 0 2 NVIDIA RTX 3090 ``` ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 5:37pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/8 "2021-07-23T17:37:46Z")
Ok then lets wrap it in a ray remote function call. Try this: ```python import ray ray.init(num_gpus=1) @ray.remote(num_gpus=1) def test_torchgpu(): import torch print(torch.cuda.is_available(), torch.cuda.current_device(), torch.cuda.device(0), torch.cuda.device_count(), torch.cuda.get_device_name(0)) ray.get(test_torchgpu.remote()) print("ray.get_gpu_ids(): ", ray.get_gpu_ids()) ray.shutdown() ``` ---
**Author:** ![daniel](https://avatars.discourse-cdn.com/v4/letter/d/3ec8ea/32.png) [@daniel](https://discuss.ray.io/u/daniel)\ **Post date:** [July 23, 2021, 5:57pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/9 "2021-07-23T17:57:35Z")
It returns the similar output ```auto (pid=391663) True 0 1 NVIDIA RTX 3090 ``` ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 6:02pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/10 "2021-07-23T18:02:54Z")
HI @daniel looking at the issues in github it looks like it has been broken for some weeks now. Several people seem to be waiting on a fix. ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [July 23, 2021, 6:09pm UTC](https://discuss.ray.io/t/example-of-a3c-only-use-cpu-for-trainer/2984/11 "2021-07-23T18:09:52Z")
@daniel, This would be a good issue to read and track: > > > \### What is the problem? > > When running a simple RLlib training script, almost …identical to the example \[here\](https://docs.ray.io/en/master/rllib-training.html#basic-python-api), I get the following error: > > \`\`\` > Traceback (most recent call last): > File "test.py", line 12, in \ > trainer = ppo.PPOTrainer(config=config, env="CartPole-v0") > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py", line 123, in \_\_init\_\_ > Trainer.\_\_init\_\_(self, config, env, logger\_creator) > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 562, in \_\_init\_\_ > super().\_\_init\_\_(config, logger\_creator) > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/tune/trainable.py", line 100, in \_\_init\_\_ > self.setup(copy.deepcopy(self.config)) > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 722, in setup > self.\_init(self.config, self.env\_creator) > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py", line 147, in \_init > self.workers = self.\_make\_workers( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 804, in \_make\_workers > return WorkerSet( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/evaluation/worker\_set.py", line 92, in \_\_init\_\_ > self.\_local\_worker = self.\_make\_worker( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/evaluation/worker\_set.py", line 368, in \_make\_worker > worker = cls( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py", line 545, in \_\_init\_\_ > self.policy\_map, self.preprocessors = self.\_build\_policy\_map( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py", line 1323, in \_build\_policy\_map > policy\_map\[name\] = cls(obs\_space, act\_space, merged\_conf) > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/policy/policy\_template.py", line 256, in \_\_init\_\_ > self.parent\_cls.\_\_init\_\_( > File "/home/cassidy/miniconda3/envs/rllib\_test/lib/python3.8/site-packages/ray/rllib/policy/torch\_policy.py", line 156, in \_\_init\_\_ > self.device = self.devices\[0\] > IndexError: list index out of range > \`\`\` > > The script that reproduces the error is below in the reproduction section. It looks like this error is caused by \`ray.get\_gpu\_ids()\` returning an empty list (\`\[\]\`) despite there being GPUs attached to the system: > > \`\`\` > \>\>\> import ray > \>\>\> ray.init(num\_gpus=4) > 2021-06-28 12:57:14,182 INFO services.py:1330 -- View the Ray dashboard at http://127.0.0.1:8265 > {'node\_ip\_address': '128.32.175.10', 'raylet\_ip\_address': '128.32.175.10', 'redis\_address': '128.32.175.10:6379', 'object\_store\_address': '/tmp/ray/session\_2021-06-28\_12-57-13\_130563\_33105/sockets/plasma\_store', 'raylet\_socket\_name': '/tmp/ray/session\_2021-06-28\_12-57-13\_130563\_33105/sockets/raylet', 'webui\_url': '127.0.0.1:8265', 'session\_dir': '/tmp/ray/session\_2021-06-28\_12-57-13\_130563\_33105', 'metrics\_export\_port': 59050, 'node\_id': 'b6854536c3fea8b39eac4a0723a2af43d56571adf35a750f99bc9982'} > \>\>\> ray.get\_gpu\_ids() > \[\] > \>\>\> import torch > \>\>\> torch.cuda.is\_available() > True > \`\`\` > > I'm not sure why this is happening—it didn't happen with RLLib 1.3. Interestingly, I can train on GPUs using \`rllib train\` (the RLlib CLI) with no issue. > > \*Ray version and other system information (Python version, TensorFlow version, OS):\* > \* Ubuntu 18.04.5 > \* Python 3.8.10 (anaconda) > \* CUDA 11.3 > > \ > \Installed Python packages\ > > \`\`\` > Package Version > \------------------------ ------------------- > aiohttp 3.7.4.post0 > aiohttp-cors 0.7.0 > aioredis 1.3.1 > async-timeout 3.0.1 > attrs 21.2.0 > blessings 1.7 > cachetools 4.2.2 > certifi 2021.5.30 > chardet 4.0.0 > click 8.0.1 > cloudpickle 1.6.0 > colorama 0.4.4 > dm-tree 0.1.6 > filelock 3.0.12 > google-api-core 1.30.0 > google-auth 1.32.0 > googleapis-common-protos 1.53.0 > gpustat 0.6.0 > grpcio 1.38.1 > gym 0.18.3 > hiredis 2.0.0 > idna 2.10 > jsonschema 3.2.0 > msgpack 1.0.2 > multidict 5.1.0 > numpy 1.21.0 > nvidia-ml-py3 7.352.0 > opencensus 0.7.13 > opencensus-context 0.1.2 > opencv-python 4.5.2.54 > packaging 20.9 > pandas 1.2.5 > Pillow 8.2.0 > pip 21.1.2 > prometheus-client 0.11.0 > protobuf 3.17.3 > psutil 5.8.0 > py-spy 0.3.7 > pyasn1 0.4.8 > pyasn1-modules 0.2.8 > pydantic 1.8.2 > pyglet 1.5.15 > pyparsing 2.4.7 > pyrsistent 0.18.0 > python-dateutil 2.8.1 > pytz 2021.1 > PyYAML 5.4.1 > ray 2.0.0.dev0 > redis 3.5.3 > requests 2.25.1 > rsa 4.7.2 > scipy 1.7.0 > setuptools 52.0.0.post20210125 > six 1.16.0 > tabulate 0.8.9 > torch 1.9.0 > typing-extensions 3.10.0.0 > urllib3 1.26.6 > wheel 0.36.2 > yarl 1.6.3 > \`\`\` > \ > > \### Reproduction (REQUIRED) > > \ > \test.py\ > > \`\`\` > import ray > import ray.rllib.agents.ppo as ppo > from ray.tune.logger import pretty\_print > > ray.init() > config = ppo.DEFAULT\_CONFIG.copy() > config.update({ > "num\_gpus": 1, > "num\_workers": 1, > "framework": "torch", > }) > trainer = ppo.PPOTrainer(config=config, env="CartPole-v0") > > \# Can optionally call trainer.restore(path) to load a checkpoint. > > for i in range(1000): > # Perform one iteration of training the policy with PPO > result = trainer.train() > print(pretty\_print(result)) > > if i % 100 == 0: > checkpoint = trainer.save() > print("checkpoint saved at", checkpoint) > \`\`\` > \ > > \- \[x\] I have verified my script runs in a clean environment and reproduces the issue. > \- \[x\] I have verified the issue also occurs with the \[latest wheels\](https://docs.ray.io/en/master/installation.html).