### System information
- **Have I written custom code (as opposed to using …a stock example script
provided in TensorFlow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 20.04
- **TensorFlow installed from (source or binary)**: binary
- **TensorFlow version (use command below)**: v2.3.0-rc2-23-gb36436b087 2.3.0
- **Python version**: 3.8.2
- **CUDA/cuDNN version**: Cuda 10.1/ cuDNN 7.6.5
- **GPU model and memory**: Nvidia GTX 750Ti
- **Exact command to reproduce**:
```
# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras
fashion_mnist = keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
# At this step I was getting the error which I've posted below in the terminal.
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(10)
])
```
### Describe the problem
I've recently installed ubuntu 20.04 LTS and it comes with python-3.8, so I'll installed **nvidia-cuda-toolkit** and **nvidia drivers** and I can confirm they are working fine.
```
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
```
```
$ nvidia-smi
Mon Aug 3 02:56:11 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 750 Ti Off | 00000000:01:00.0 On | N/A |
| 27% 38C P0 1W / 38W | 245MiB / 1997MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 979 G /usr/lib/xorg/Xorg 20MiB |
+-----------------------------------------------------------------------------+
```
Now, I tried to build a small sequential model I am getting an error which says `InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid`
I don't know what causing the issue. My linux ubuntu is a new installation. I have installed everything correctly.
### Source code / logs
```
2020-08-03 02:48:40.720575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-03 02:48:40.750630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 750 Ti computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 80.47GiB/s
2020-08-03 02:48:40.750735: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-03 02:48:40.791690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-03 02:48:40.815993: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-03 02:48:40.821924: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-03 02:48:40.863910: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-03 02:48:40.870559: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-03 02:48:40.945916: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-03 02:48:40.947130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-03 02:48:40.979471: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3311130000 Hz
2020-08-03 02:48:40.980123: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4aa6700 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-03 02:48:40.980190: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-03 02:48:41.121266: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x49375f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-03 02:48:41.121357: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 750 Ti, Compute Capability 5.0
2020-08-03 02:48:41.122574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 750 Ti computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 80.47GiB/s
2020-08-03 02:48:41.122676: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-03 02:48:41.122762: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-03 02:48:41.122830: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-03 02:48:41.122898: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-03 02:48:41.122963: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-03 02:48:41.123029: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-03 02:48:41.123145: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-03 02:48:41.124618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-03 02:48:41.124716: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
---------------------------------------------------------------------------
InternalError Traceback (most recent call last)
<ipython-input-4-ac4dc71cdd20> in <module>
----> 1 model = keras.Sequential([
2 keras.layers.Flatten(input_shape=(28, 28)),
3 keras.layers.Dense(128, activation='relu'),
4 keras.layers.Dense(10)
5 ])
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
455 self._self_setattr_tracking = False # pylint: disable=protected-access
456 try:
--> 457 result = method(self, *args, **kwargs)
458 finally:
459 self._self_setattr_tracking = previous_value # pylint: disable=protected-access
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py in __init__(self, layers, name)
114 """
115 # Skip the init in FunctionalModel since model doesn't have input/output yet
--> 116 super(functional.Functional, self).__init__( # pylint: disable=bad-super-call
117 name=name, autocast=False)
118 self.supports_masking = True
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
455 self._self_setattr_tracking = False # pylint: disable=protected-access
456 try:
--> 457 result = method(self, *args, **kwargs)
458 finally:
459 self._self_setattr_tracking = previous_value # pylint: disable=protected-access
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in __init__(self, *args, **kwargs)
306 self._steps_per_execution = None
307
--> 308 self._init_batch_counters()
309 self._base_model_initialized = True
310 _keras_api_gauge.get_cell('model').set(True)
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
455 self._self_setattr_tracking = False # pylint: disable=protected-access
456 try:
--> 457 result = method(self, *args, **kwargs)
458 finally:
459 self._self_setattr_tracking = previous_value # pylint: disable=protected-access
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in _init_batch_counters(self)
315 # `evaluate`, and `predict`.
316 agg = variables.VariableAggregationV2.ONLY_FIRST_REPLICA
--> 317 self._train_counter = variables.Variable(0, dtype='int64', aggregation=agg)
318 self._test_counter = variables.Variable(0, dtype='int64', aggregation=agg)
319 self._predict_counter = variables.Variable(
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
260 return cls._variable_v1_call(*args, **kwargs)
261 elif cls is Variable:
--> 262 return cls._variable_v2_call(*args, **kwargs)
263 else:
264 return super(VariableMetaclass, cls).__call__(*args, **kwargs)
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/variables.py in _variable_v2_call(cls, initial_value, trainable, validate_shape, caching_device, name, variable_def, dtype, import_scope, constraint, synchronization, aggregation, shape)
242 if aggregation is None:
243 aggregation = VariableAggregation.NONE
--> 244 return previous_getter(
245 initial_value=initial_value,
246 trainable=trainable,
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/variables.py in <lambda>(**kws)
235 shape=None):
236 """Call on Variable class. Useful to force the signature."""
--> 237 previous_getter = lambda **kws: default_variable_creator_v2(None, **kws)
238 for _, getter in ops.get_default_graph()._variable_creator_stack: # pylint: disable=protected-access
239 previous_getter = _make_getter(getter, previous_getter)
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py in default_variable_creator_v2(next_creator, **kwargs)
2631 shape = kwargs.get("shape", None)
2632
-> 2633 return resource_variable_ops.ResourceVariable(
2634 initial_value=initial_value,
2635 trainable=trainable,
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
262 return cls._variable_v2_call(*args, **kwargs)
263 else:
--> 264 return super(VariableMetaclass, cls).__call__(*args, **kwargs)
265
266
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py in __init__(self, initial_value, trainable, collections, validate_shape, caching_device, name, dtype, variable_def, import_scope, constraint, distribute_strategy, synchronization, aggregation, shape)
1505 self._init_from_proto(variable_def, import_scope=import_scope)
1506 else:
-> 1507 self._init_from_args(
1508 initial_value=initial_value,
1509 trainable=trainable,
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py in _init_from_args(self, initial_value, trainable, collections, caching_device, name, dtype, constraint, synchronization, aggregation, distribute_strategy, shape)
1648 with ops.get_default_graph()._attr_scope({"_class": attr}):
1649 with ops.name_scope("Initializer"), device_context_manager(None):
-> 1650 initial_value = ops.convert_to_tensor(
1651 initial_value() if init_from_fn else initial_value,
1652 name="initial_value", dtype=dtype)
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
1497
1498 if ret is None:
-> 1499 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
1500
1501 if ret is NotImplemented:
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/tensor_conversion_registry.py in _default_conversion_function(***failed resolving arguments***)
50 def _default_conversion_function(value, dtype, name, as_ref):
51 del as_ref # Unused.
---> 52 return constant_op.constant(value, dtype, name=name)
53
54
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name)
261 ValueError: if called on a symbolic tensor.
262 """
--> 263 return _constant_impl(value, dtype, shape, name, verify_shape=False,
264 allow_broadcast=True)
265
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast)
273 with trace.Trace("tf.constant"):
274 return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
--> 275 return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
276
277 g = ops.get_default_graph()
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
298 def _constant_eager_impl(ctx, value, dtype, shape, verify_shape):
299 """Implementation of eager constant."""
--> 300 t = convert_to_eager_tensor(value, ctx, dtype)
301 if shape is None:
302 return t
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
95 except AttributeError:
96 dtype = dtypes.as_dtype(dtype).as_datatype_enum
---> 97 ctx.ensure_initialized()
98 return ops.EagerTensor(value, ctx.device_name, dtype)
99
/mnt/Work/work_env/lib/python3.8/site-packages/tensorflow/python/eager/context.py in ensure_initialized(self)
537 if self._use_tfrt is not None:
538 pywrap_tfe.TFE_ContextOptionsSetTfrt(opts, self._use_tfrt)
--> 539 context_handle = pywrap_tfe.TFE_NewContext(opts)
540 finally:
541 pywrap_tfe.TFE_DeleteContextOptions(opts)
InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
```