# MAML Discrete actions update
**URL:** https://discuss.ray.io/t/maml-discrete-actions-update/546
**Category:** RLlib
**Created:** [January 22, 2021, 7:51pm UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546 "2021-01-22T19:51:04Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Chace\_Ashcraft](https://discuss.ray.io/u/Chace_Ashcraft)
#### Post date: [January 22, 2021, 7:51pm UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/1 "2021-01-22T19:51:04Z")
I noticed that there is a commit (47b499d899f8eccc00905992a6603ede97d1d44e) with the message “Cartpole MAML + Discrete,” does this mean that MAML handles discrete actions now?
Before noticing that the MAML documentation states that it doesn’t handle discrete action spaces, I tried setting it up to run on an environment with a discrete action space, and it runs… but I’m guessing I should’t expect learning to occur?
Thanks!
---
### Author:  [@sven1977](https://discuss.ray.io/u/sven1977)
#### Post date: [January 25, 2021, 9:43am UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/2 "2021-01-25T09:43:11Z")
It should work for PyTorch, yes!
I have added a test case for this as well: rllib.agents.maml.tests.test\_maml.py
Here is the PR (with the CartPole task env and a simple test added): [[RLlib] MAML: Add cartpole mass test for PyTorch. by sven1977 · Pull Request #13679 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/13679)
---
### Author:  [@Chace\_Ashcraft](https://discuss.ray.io/u/Chace_Ashcraft)
#### Post date: [January 25, 2021, 3:37pm UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/3 "2021-01-25T15:37:42Z")
Thanks!
It does seem to be working for me, so that is awesome! I can’t run with more than 2 workers though. I’ll post the traceback at the end, but it looks to me like in `maml_torch_policy.py` there is a default value set for the `split` variable on line 274 that doesn’t work with the loop in the MAMLLoss **init** at line 167 when `'num_workers'` \> 2. I’m going to see if I can make a change to fix it, but if you know a solution that would be great! 🙂 Especially considering my solution may just be a quick hack.
Traceback (most recent call last):
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/tune/trial\_runner.py”, line 519, in \_process\_trial
result = self.trial\_executor.fetch\_result(trial)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/tune/ray\_trial\_executor.py”, line 497, in fetch\_result
result = ray.get(trial\_future[0], timeout=DEFAULT\_GET\_TIMEOUT)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/worker.py”, line 1379, in get
raise value.as\_instanceof\_cause()
ray.exceptions.RayTaskError(IndexError): e[36mray::MAML.train()e[39m (pid=73302, ip=192.168.0.4)
File “python/ray/\_raylet.pyx”, line 422, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 456, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 459, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 463, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 415, in ray.\_raylet.execute\_task.function\_executor
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py”, line 106, in **init**
Trainer. **init** (self, config, env, logger\_creator)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/trainer.py”, line 465, in **init**
super(). **init** (config, logger\_creator)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/tune/trainable.py”, line 96, in **init**
self.setup(copy.deepcopy(self.config))
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/trainer.py”, line 629, in setup
self.\_init(self.config, self.env\_creator)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py”, line 133, in \_init
self.workers = self.\_make\_workers(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/trainer.py”, line 700, in \_make\_workers
return WorkerSet(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/evaluation/worker\_set.py”, line 87, in **init**
self.\_local\_worker = self.\_make\_worker(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/evaluation/worker\_set.py”, line 315, in \_make\_worker
worker = cls(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py”, line 462, in **init**
self.policy\_map, self.preprocessors = self.\_build\_policy\_map(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py”, line 1076, in \_build\_policy\_map
policy\_map[name] = cls(obs\_space, act\_space, merged\_conf)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/policy/torch\_policy\_template.py”, line 249, in **init**
self.\_initialize\_loss\_from\_dummy\_batch(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/policy/policy.py”, line 654, in \_initialize\_loss\_from\_dummy\_batch
self.\_loss(self, self.model, self.dist\_class, train\_batch)
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/maml/maml\_torch\_policy.py”, line 277, in maml\_loss
policy.loss\_obj = MAMLLoss(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/maml/maml\_torch\_policy.py”, line 173, in **init**
ppo\_loss, \_, inner\_kl\_loss, \_, \_ = self.compute\_losses(
File “/Users/me/anaconda3/envs/my\_env/lib/python3.8/site-packages/ray/rllib/agents/maml/maml\_torch\_policy.py”, line 216, in compute\_losses
obs = self.obs[inner\_adapt\_iter][task\_iter]
IndexError: tuple index out of range
---
### Author:  [@Chace\_Ashcraft](https://discuss.ray.io/u/Chace_Ashcraft)
#### Post date: [January 25, 2021, 6:24pm UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/4 "2021-01-25T18:24:57Z")
I have parallel environments working with modified RLlib code here:
>
The code is running, and I don’t think my “fix” broke anything else, but please let me know if it did!
---
### Author:  [@sven1977](https://discuss.ray.io/u/sven1977)
#### Post date: [February 1, 2021, 9:28am UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/5 "2021-02-01T09:28:54Z")
Awesome @Chace_Ashcraft . Did you create a PR, so we can merge your fix into master?
---
### Author:  [@Chace\_Ashcraft](https://discuss.ray.io/u/Chace_Ashcraft)
#### Post date: [February 1, 2021, 3:41pm UTC](https://discuss.ray.io/t/maml-discrete-actions-update/546/6 "2021-02-01T15:41:47Z")
@sven1977 I had not, but created one just now here: [Pytorch MAML fix for more than two workers with discrete actions by ChaceAshcraft · Pull Request #13835 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/13835). Hope it helps! Thanks for your awesome work on ray and rllib by the way!