Tune Layer/Neurons combinations

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hi,

I want to tune to find the optimal configuration of Dense layers for my network. For example, how many layers and for each layer how many neurons. I’m not sure how to configure this w/ Ray.

The best I could do is fix the number of layers and have the neuron count be tuned, eg if I fix 3 layers:

        'mlp_layers': [tune.randint(1,256), tune.randint(1,256), tune.randint(1,256)],

But I don’t want to fix 3 layers I want it to be sampled, eg tune.randint(1,4)

2

Hi @eggie5,

You could use tune.sample_from. sample_from takes a function that produces a sample.

Something like this:

'mlp_layers': tune.sample_from(lambda _: [tune.randint(1,256) for _ in tune.randint(1,4)])
3 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-150-7cfb472460bb> in <module>
----> 1 tune.sample_from(lambda _: [tune.randint(1,256) for _ in tune.randint(1,4)]).sample()

~/.local/lib/python3.7/site-packages/ray/tune/sample.py in sample(self, spec, size)
     45     def sample(self, spec=None, size=1):
     46         sampler = self.get_sampler()
---> 47         return sampler.sample(self, spec=spec, size=size)
     48 
     49     def is_grid(self):

~/.local/lib/python3.7/site-packages/ray/tune/sample.py in sample(self, domain, spec, size)
    336                 items = [
    337                     domain.func(spec[i] if isinstance(spec, list) else spec)
--> 338                     for i in range(size)
    339                 ]
    340             else:

~/.local/lib/python3.7/site-packages/ray/tune/sample.py in <listcomp>(.0)
    336                 items = [
    337                     domain.func(spec[i] if isinstance(spec, list) else spec)
--> 338                     for i in range(size)
    339                 ]
    340             else:

<ipython-input-150-7cfb472460bb> in <lambda>(_)
----> 1 tune.sample_from(lambda _: [tune.randint(1,256) for _ in tune.randint(1,4)]).sample()

TypeError: 'Integer' object is not iterable
4

Sorry try: range(tune.randint(1,4))

5 
        'mlp_layers': tune.sample_from(lambda _: [tune.randint(1,256) for _ in range(tune.randint(1,4))]),

TypeError: 'Integer' object cannot be interpreted as an integer

6

I guess the tune samplers won’t work here. Replace tune.randint with the standard python random.randint. You will need to import random.

Manny