Ray start throws `ValueError: acceleratorType should match v(generation)-(cores/chips). Got .`

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Running the cluster launcher leaves the nodes uninitialized so I was manually starting ray on each node. After a fresh conda venv install, ray can’t manually be started. Each node complains with:

$ RAY_memory_monitor_refresh_ms=0 ray start --address='192.168.0.101:6379'
Local node IP: 192.168.0.108
Traceback (most recent call last):
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/scripts/scripts.py", line 2490, in main
    return cli()
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper
    return f(*args, **kwargs)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/scripts/scripts.py", line 920, in start
    node = ray._private.node.Node(
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/node.py", line 310, in __init__
    self.start_ray_processes()
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/node.py", line 1452, in start_ray_processes
    resource_spec = self.get_resource_spec()
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/node.py", line 540, in get_resource_spec
    self._resource_spec = ResourceSpec(
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/resource_spec.py", line 204, in resolve
    accelerator.update_resources_with_accelerator_type(resources)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/accelerator.py", line 39, in update_resources_with_accelerator_type
    accelerator_type=_autodetect_tpu_version(),
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/accelerator.py", line 214, in _autodetect_tpu_version
    return accelerator_type_to_version(accelerator_type_request.text)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/accelerator.py", line 197, in accelerator_type_to_version
    assert_tpu_accelerator_type(accelerator_type)
  File "/home/jernej_m/mambaforge-pypy3/envs/test_ray/lib/python3.9/site-packages/ray/_private/accelerator.py", line 239, in assert_tpu_accelerator_type
    raise ValueError(
ValueError: `acceleratorType` should match v(generation)-(cores/chips). Got .

Which leaves me without any options to run ray on a local cluster. Any help welcome.

2

I believe this is fixed by [Core] TPU detection code can cause ray start to fail · Issue #39913 · ray-project/ray · GitHub! Can you try the latest master? The fix will be included in 2.7.1, which will be released on 10.9

3

Yes, starting ray manually works with nightly version. Hope a test will also be included as a part of the fix to check if ray start… starts.