# Intended behaviour or smth wrong with my code? "no checkpoint for trial. Skip exploit for Trial"

**URL:** <https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376>\
**Category:** Ray Core\
**Created:** [March 24, 2021, 2:25pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376 "2021-03-24T14:25:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![eaakimov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eaakimov/32/664_2.png) [@eaakimov](https://discuss.ray.io/u/eaakimov)\
**Post date:** [March 24, 2021, 2:25pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/1 "2021-03-24T14:25:19Z")

</div>

Dear all,  
is getting this message: “… no checkpoint for trial. Skip exploit for Trial XXX…” considered to be a part of a normal PBT workflow?  
I am pretty sure I have all checkpoints saved, I use:

> TuneReportCheckpointCallback(  
> metrics={“val\_loss”: “val\_loss”},  
> filename=“model”,  
> on =“epoch\_end” )

Thanks!

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [March 24, 2021, 3:05pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/2 "2021-03-24T15:05:41Z")

</div>

@kai @rliaw could you please have a look?

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [March 24, 2021, 4:48pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/3 "2021-03-24T16:48:07Z")

</div>

Hi @eaakimov, can you share your trainable function with us?

---

<div class="post-metadata">

**Author:** ![eaakimov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eaakimov/32/664_2.png) [@eaakimov](https://discuss.ray.io/u/eaakimov)\
**Post date:** [March 24, 2021, 8:33pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/5 "2021-03-24T20:33:26Z")

</div>

Hi Kai,  
This issue happens 25-50% of the time, other weights and params transfers are successful. Also, I am sure that all checkpoints are saved.  
I have a pretty long class, hope it is enough:

#### hyperparams

```
		self.hypers = {	

		
			    "inc_layers" : [0,1,2],
			    "ker1" : { '0':[1,2,3], '1':[1,2,3]}, 
			   	"ker2" : { '0':[1,2,3], '1':[1,2,3]},  
			  
			   	'FEAT_ker_mult1' : lambda: np.random.randint(1,4), # excluding
			   	'FEAT_ker_mult2' : lambda: np.random.randint(1,4),
			    'FEAT_n_fil' : [16,32,64,98],
			  
				'GRU_encoder' : lambda: np.random.randint(0,4),
				'GRU_nodes' : lambda: np.random.randint(5,100), ## upper exclusive
				'attention' : ['both','first','last'],
				'BahN' : lambda: np.random.randint(5,50),
				'comm_D_Drop' : lambda: np.random.uniform(0.85,0.95),
				'common_dense_num' : lambda: np.random.randint(1,4),
				'comm_D_N' : lambda: np.random.randint(50,300),
				'maxP' : lambda: np.random.randint(1,5),
				'bs' : [8,16,24],

				'lr' : [0.01 , 0.005 , 0.001 , 0.0005, 0.0001] 
				
   			     
   			 }

```

#### fixed parameters

```
	self.conf= {

				#model related
				'loss' : 'mse',
				'opt' : 'adam',
				'metrics' : 'mape',

				'val_split' : 0.25,
				'verb' : 1,
				'bs' : 64,

				'activation' : LeakyReLU (alpha= 0.05 )

   	}

```

#### Tuner

```
def tune(self):

	scheduler = PopulationBasedTraining(
		 time_attr="training_iteration",
		 perturbation_interval=2,
		 hyperparam_mutations = self.hypers

		 )

	stopper = MaximumIterationStopper(max_iter=10)

	analysis = tune.run(
		
		self.fit_tuner, 

		scheduler = scheduler,
		raise_on_failed_trial	= False,
		metric = 'mse',
		mode = 'min',
	    verbose = 1, 
	    local_dir = self.tuner_workdir,
	    num_samples = 5,
		config = self.conf
		)

	return(analysis)

```

#### Trainable

```
def fit_tuner(self,hp, checkpoint_dir=None):

	
	callbacks = [ TuneReportCheckpointCallback(
         		  metrics={"val_loss": "val_loss"},
         		  filename="model",
         		  on ="epoch_end") ]

	nn=self.model(hp) ### self.model returns compiled model

	nn.fit(
	self.train_X, self.train_Y, 
	validation_split=self.conf['val_split'],
	verbose=self.conf['verb'], 
	batch_size=hp['bs'], 
	epochs=10, 
	callbacks=callbacks)

```

Thanks!

---

<div class="post-metadata">

**Author:** ![eaakimov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eaakimov/32/664_2.png) [@eaakimov](https://discuss.ray.io/u/eaakimov)\
**Post date:** [March 24, 2021, 9:53pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/6 "2021-03-24T21:53:05Z")

</div>

Also, I use ray 2.0.  
UPDATE:  
Same issue on ray 1.2

---

<div class="post-metadata">

**Author:** ![eaakimov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eaakimov/32/664_2.png) [@eaakimov](https://discuss.ray.io/u/eaakimov)\
**Post date:** [March 25, 2021, 8:03pm UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/7 "2021-03-25T20:03:08Z")

</div>

Still having the issue, I guess it could be a bug since trials do save normally. Should I initiate an issue on github?  
What could I do to try to figure out what happens or maybe there is an alternative approach which I could take to workaround?

Thanks!

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [March 27, 2021, 12:04am UTC](https://discuss.ray.io/t/intended-behaviour-or-smth-wrong-with-my-code-no-checkpoint-for-trial-skip-exploit-for-trial/1376/8 "2021-03-27T00:04:28Z")

</div>

Hmm, maybe you could try running one of the basic PBT examples to see if the same occurs?

I do suspect it’s an obscure issue…