Ray API to get details about actor or task running on ray nodes

Ray Core
1

Hello,

I would like to know if a ray worker is running an actor or task, is there a developer API to get such details?
I tried ray.nodes() API it does not tell me if a node is running actor or task:

[{'NodeID': '3a2163869db41b4959980031dbf2316f8c51b6981539a408fc5d03c0', 'Alive': True, 'NodeManagerAddress': '172.17.0.4', 'NodeManagerHostname': 'example-cluster-ray-head-type-6zrkv', 'NodeManagerPort': 34921, 'ObjectManagerPort': 37621, 'ObjectStoreSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/raylet', 'MetricsExportPort': 62442, 'alive': True, 'Resources': {'object_store_memory': 581958451.0, 'node:172.17.0.4': 1.0, 'CPU': 1.0, 'memory': 1503238553.0}}, {'NodeID': '185ea8409086d4def9e0c8962b8229c10bc9970f88407cde1fe66d82', 'Alive': True, 'NodeManagerAddress': '172.17.0.5', 'NodeManagerHostname': 'example-cluster-ray-worker-type-8qbc8', 'NodeManagerPort': 43647, 'ObjectManagerPort': 42897, 'ObjectStoreSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/raylet', 'MetricsExportPort': 65271, 'alive': True, 'Resources': {'memory': 375809638.0, 'CPU': 1.0, 'node:172.17.0.5': 1.0, 'object_store_memory': 143407104.0}}, {'NodeID': 'bb05ec85f8c337ae58fe42d21a6743fbfd11381ca722fc7ec600c0d9', 'Alive': True, 'NodeManagerAddress': '172.17.0.7', 'NodeManagerHostname': 'example-cluster-ray-worker-type-648z8', 'NodeManagerPort': 38055, 'ObjectManagerPort': 40627, 'ObjectStoreSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/raylet', 'MetricsExportPort': 64740, 'alive': True, 'Resources': {'CPU': 1.0, 'memory': 375809638.0, 'object_store_memory': 143291596.0, 'node:172.17.0.7': 1.0}}, {'NodeID': 'e28787a67afb6ca2581786bea787cbc591f89a52a40f4fabd0641ddd', 'Alive': True, 'NodeManagerAddress': '172.17.0.6', 'NodeManagerHostname': 'example-cluster-ray-worker-type-xvz2j', 'NodeManagerPort': 44155, 'ObjectManagerPort': 45769, 'ObjectStoreSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-08-09_11-54-14_626105_102/sockets/raylet', 'MetricsExportPort': 49914, 'alive': True, 'Resources': {'object_store_memory': 144262348.0, 'CPU': 1.0, 'memory': 375809638.0, 'node:172.17.0.6': 1.0}}]

Thanks

2

In my ray cluster I have an actor running when I try to run the below code:

LOCAL_PORT = 10001

ray.util.connect(f"127.0.0.1:{LOCAL_PORT}")

from ray.state import actors

print(actors())

I get exception:

 python /home/asmalvan/code/ray140/ray/doc/kubernetes/example_scripts/list_nodes.py
Traceback (most recent call last):
  File "/home/asmalvan/code/ray140/ray/doc/kubernetes/example_scripts/list_nodes.py", line 31, in <module>
    print(actors())
  File "/home/asmalvan/anaconda3/envs/ray1.4/lib/python3.7/site-packages/ray/state.py", line 829, in actors
    return state.actor_table(actor_id=actor_id)
  File "/home/asmalvan/anaconda3/envs/ray1.4/lib/python3.7/site-packages/ray/state.py", line 143, in actor_table
    self._check_connected()
  File "/home/asmalvan/anaconda3/envs/ray1.4/lib/python3.7/site-packages/ray/state.py", line 51, in _check_connected
    "Ray has not been started yet. You can start Ray with "
ray.exceptions.RaySystemError: System error: Ray has not been started yet. You can start Ray with 'ray.init()'.
3

If you run

from ray.state import actors

print(actors())

without ray client, does that work?

4

Also, for actor states, you can use ray.state.actors() (though it is debugging API now) to see the status, but for tasks, there’s currently no good way to help you seeing this. We have some feature requests on it, but it might take some time to implement that.

5

Yes this code snippet works when I start ray cluster on my laptop.

6

@sangcho do you mean there is no mechanism to run ray.state.actors() externally on ray cluster running on kubernetes?

7

Hmm I think it is just not supported yet. @ijrsvt Do we have an open issue about this?

8

Ran another test by exec on the ray head node deployed on minikube, I get below error:

>>> print(ray.state.actors())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/state.py", line 829, in actors
    return state.actor_table(actor_id=actor_id)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/state.py", line 143, in actor_table
    self._check_connected()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/state.py", line 51, in _check_connected
    "Ray has not been started yet. You can start Ray with "
ray.exceptions.RaySystemError: System error: Ray has not been started yet. You can start Ray with 'ray.init()'.
9

Hello, do we have an open issue for the above and can we support ray.state.actors() API for developers?

10

Sorry for the delay, but ray.state.actors() is a Private API and is not currently supported directly via Ray Client. The easiest workaround is wrapping this in a remote function:

@ray.remote
def actors_on_host():
   return ray.state.actors()

ray.get(actors_on_host.remote())
11

Thanks, I did run the above API inside an actor to perform actor discovery inside a ray cluster.