Autoscaling issue on Google cloud

I am trying to execute the from this github link.

I have to train a custom model on much smaller dataset so I changed tpu_size = 8 in [configs/6B_roto_256.json]
and running the following command in command prompt
python3.8 --tpu gptj6b-tpu1 --tpu_region us-central1-c --config configs/6B_roto_256.json --preemptible --new

When I execute this and gets the following error:
{‘name’: ‘projects/hellogenius-poc/locations/us-central1-c/nodes/gptj6b-tpu1’, ‘acceleratorType’: ‘v2-8’, ‘state’:
‘READY’, ‘runtimeVersion’: ‘v2-alpha’, ‘cidrBlock’: ‘’, ‘createTime’: ‘2021-07-04T04:54:27.658431621Z’
, ‘schedulingConfig’: {‘preemptible’: True}, ‘networkEndpoints’: [{‘ipAddress’: ‘’, ‘port’: 8470, ‘acces
sConfig’: {‘externalIp’: ‘’}}], ‘health’: ‘HEALTHY’, ‘id’: ‘4392319268867105180’, ‘networkConfig’: {’
network’: ‘projects/hellogenius-poc/global/networks/default’, ‘subnetwork’: ‘projects/hellogenius-poc/regions/us-ce
ntral1/subnetworks/default’, ‘enableExternalIps’: True}, ‘serviceAccount’: {‘email’: ‘889652764764-compute@develope’, ‘scope’: []}, ‘apiVersion’: ‘V2_ALPHA1’}
2021-07-04 07:28:44,371 WARNING – The actor or task with ID ffffffffffffffff13878e716d2e442eb9333d0
501000000 cannot be scheduled right now. It requires {CPU: 1.000000}, {tpu: 1.000000} for placement, however the cl
uster currently cannot provide the requested resources. The required resources may be added as autoscaling takes pl
ace or placement groups are scheduled. Otherwise, consider reducing the resource requirements of the task.
(pid=14384, ip= 2021-07-04 07:28:44.753523: F external/org_tensorflow/tensorflow/core/tpu/tpu_executor_] TpuTransferManager_ReadDynamicShapes not available in this library.
Traceback (most recent call last):
File “”, line 75, in
t = build_model(params, tpu_name, region, preemptible, version=args.version)
File “/home/param_jeet/content-intelligence/Mesh-Transformer/mesh_transformer/”, line 58, in build_
t = TPUCluster((tpu_size // cores_per_replica, cores_per_replica), len(conns), model_fn)
File “/home/param_jeet/.local/lib/python3.8/site-packages/func_timeout/”, line 185, in
return wraps(func)(lambda *args, **kwargs : func_timeout(defaultTimeout, func, args=args, kwargs=kwargs))
File “/home/param_jeet/.local/lib/python3.8/site-packages/func_timeout/”, line 108, in func_timeout
File “/home/param_jeet/.local/lib/python3.8/site-packages/func_timeout/”, line 7, in raise_exception
2021-07-04 07:28:44,930 WARNING – A worker died or was killed while executing task ffffffffffffffff
raise exception[0] from None
File “/home/param_jeet/content-intelligence/Mesh-Transformer/mesh_transformer/”, line 39, in __init
self.param_count = ray.get(params)[0]
File “/home/param_jeet/.local/lib/python3.8/site-packages/ray/_private/”, line 47, in wrapper
return func(args, **kwargs)
File “/home/param_jeet/.local/lib/python3.8/site-packages/ray/”, line 1458, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task. Check python-core-worker-
g files for more information.