# Using RayTune with tensorflow\_recommenders library and PBT - InternalError

**URL:** <https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299>\
**Category:** Ray Tune\
**Created:** [December 20, 2020, 3:39pm UTC](https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299 "2020-12-20T15:39:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jakubvedral](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jakubvedral/32/186_2.png) [@jakubvedral](https://discuss.ray.io/u/jakubvedral)\
**Post date:** [December 20, 2020, 3:39pm UTC](https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299/1 "2020-12-20T15:39:13Z")

</div>

Hi,

I was trying to use ray tune with tensorflow\_recommenders library but it gives me constatnly error:

```
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
InternalError: Tensorflow type 21 not convertible to numpy dtype.

```

I figured out that the error is present even in the basic tutorial, so reproducible example can be found here:  
[https://colab.research.google.com/drive/1ZypDwbj6aLTK15t3QNQ04MBLsR2auNtS?usp=sharing](https://colab.research.google.com/drive/1ZypDwbj6aLTK15t3QNQ04MBLsR2auNtS?usp=sharing)

Any suggestions why this might happend and how to fix it?

Thank you

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [December 20, 2020, 8:02pm UTC](https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299/2 "2020-12-20T20:02:39Z")

</div>

> [@jakubvedral](#):
>
> ```auto
> InternalError: Tensorflow type 21 not convertible to numpy dtype.
> 
> ```

Hey @jakubvedral! Thanks for dropping by.

We have a new utility on the latest Ray wheels that helps me uncover some issues:

```python
from ray.util import check_serialize

# I first tried this
check_serialize.inspect_serializability(TuneMovielensModel)
# I then ran this again on the returned value
check_serialize.inspect_serializability(CandidateModel)
# I then ran it yet again on the returned value
check_serialize.inspect_serializability(MovieModel. __init__ )

```

This showed me that the Dataset objects (like the `unique_user_ids`) were not pickleable.

```auto
============================================================================
Checking Serializability of <function MovieModel. __init__ at 0x7fa87c6c89d8>
============================================================================
!!! FAIL serialization: Tensorflow type 21 not convertible to numpy dtype.
Detected 3 global variables. Checking serializability...
    Serializing 'tf' <module 'tensorflow' from '/usr/local/lib/python3.6/dist-packages/tensorflow/ __init__.py'>...
    Serializing 'unique_movie_titles' [b"'Til There Was You (1997)" b'1-900 (1994)' b'101 Dalmatians (1996)' ...
 b'Zeus and Roxanne (1997)' b'unknown'
 b'\xc3\x81 k\xc3\xb6ldum klaka (Cold Fever) (1994)']...
    Serializing 'movies' <MapDataset shapes: (), types: tf.string>...
    !!! FAIL serialization: Tensorflow type 21 not convertible to numpy dtype.
...
            Serializing 'unique_movie_titles' [b"'Til There Was You (1997)" b'1-900 (1994)' b'101 Dalmatians (1996)' ...
 b'Zeus and Roxanne (1997)' b'unknown'
 b'\xc3\x81 k\xc3\xb6ldum klaka (Cold Fever) (1994)']...
            Serializing 'movies' <MapDataset shapes: (), types: tf.string>...
            !!! FAIL serialization: Tensorflow type 21 not convertible to numpy dtype.
        Detected 1 nonlocal variables. Checking serializability...
            Serializing ' __class__' <class ' __main__.MovieModel'>...
            !!! FAIL serialization: Tensorflow type 21 not convertible to numpy dtype.
        Serializing '_TF_MODULE_IGNORED_PROPERTIES' frozenset({'_self_unconditional_dependency_names', '_self_unconditional_checkpoint_dependencies', '_train_counter', '_predict_counter', '_obj_reference_counts_dict', '_test_counter', '_steps_per_execution'})...
============================================================================
Variable: 

	FailTuple( __class__ [obj=<class ' __main__.MovieModel'>, parent=<function MovieModel. __init__ at 0x7fa87c6c89d8>])
FailTuple(movies [obj=<MapDataset shapes: (), types: tf.string>, parent=<function MovieModel. __init__ at 0x7fa87c6c89d8>])
FailTuple(_add_variable_with_custom_getter [obj=<bound method Trackable._add_variable_with_custom_getter of <MapDataset shapes: (), types: tf.string>>, parent=<MapDataset shapes: (), types: tf.string>])

was found to be non-serializable. There may be multiple other undetected variables that were non-serializable. 
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class. 
If you have any suggestions on how to improve this error message, please reach out to the Ray developers on github.com/ray-project/ray/issues/
============================================================================
(False,

```

**Workaround** : Consider moving the dataset construction/creation into the `TuneMovielensModel`. Then, you can pass in the dataset objects into each of these Models via their respective ` __init__ ` methods. Let me know if that helps!

---

<div class="post-metadata">

**Author:** ![Yannis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yannis/32/1548_2.png) [@Yannis](https://discuss.ray.io/u/Yannis)\
**Post date:** [September 25, 2021, 2:07pm UTC](https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299/3 "2021-09-25T14:07:17Z")

</div>

> [@rliaw](#):
>
> TuneMovielensModel

Dear @rliaw, can you elaborate a little more on “moving the dataset construction/creation into the TuneMovielensModel”?

I tried the following change but still, get the same error.

```python
class TuneMovielensModel(tune.Trainable):    
    def setup(self, config, cached_train, cached_test, test_ds):                                
        self.model = MovielensModel(layer_sizes=self.config.get("layer_sizes",[32]))                
        self.model.compile(optimizer=tf.keras.optimizers.Adagrad(self.config.get("lr", 0.01)))
        self.cached_train = cached_train
        self.cached_test = cached_test
        self.test_ds = test_ds

    def step(self):        
        # train
        model.fit(
            self.cached_train,
            validation_data=self.cached_test,  
            validation_freq=5,
            epochs=50,                               
            verbose=1         
        )        
        accuracy = self.model.evaluate(test_ds, return_dict=True,verbose=0)['factorized_top_k/top_5_categorical_accuracy']
        return {"top_5_categorical_accuracy": accuracy}

```

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [September 29, 2021, 12:45am UTC](https://discuss.ray.io/t/using-raytune-with-tensorflow-recommenders-library-and-pbt-internalerror/299/4 "2021-09-29T00:45:20Z")

</div>

> [@Yannis](#):
>
> ```auto
> def step(self):        
> # train
> model.fit(
> self.cached_train,
> validation_data=self.cached_test,  
> validation_freq=5,
> epochs=50,                               
> verbose=1    
> 
> ```

this should be `self.model` right?