Ray.state.actor output and actor scheduling

Ray Core
1

Hello,

I was able to get run ray.state.actors() inside ray cluster deployed on minikube by deploying another actor, strangely one of the actors does not have an IP address, here is the output with a warning:

WARNING: 3 PYTHON workers have been started on a node of the id: fdd43e9454dad6194cfee89d5d7838a4501b7f8286f1feb6fec8c124 and address: 172.17.0.5. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds. {'52f9757c7e3593056e42024a14000000': {'ActorID': '52f9757c7e3593056e42024a14000000', 'Name': '', 'JobID': '14000000', 'Address': {'IPAddress': '172.17.0.5', 'Port': 10003, 'NodeID': 'fdd43e9454dad6194cfee89d5d7838a4501b7f8286f1feb6fec8c124'}, 'OwnerAddress': {'IPAddress': '172.17.0.4', 'Port': 10022, 'NodeID': '6962fd9e65407b74ae3493d7a1679eba46af9d9b072cd5fe2c8cdfb3'}, 'State': 2, 'NumRestarts': 0, 'Timestamp': 1628773717963.0, 'StartTime': 1628773717963, 'EndTime': 0}, '305b193426974e06ad878a6b14000000': {'ActorID': '305b193426974e06ad878a6b14000000', 'Name': '', 'JobID': '14000000', 'Address': {'IPAddress': '', 'Port': 0, 'NodeID': 'fdd43e9454dad6194cfee89d5d7838a4501b7f8286f1feb6fec8c124'}, 'OwnerAddress': {'IPAddress': '172.17.0.4', 'Port': 10022, 'NodeID': '6962fd9e65407b74ae3493d7a1679eba46af9d9b072cd5fe2c8cdfb3'}, 'State': 1, 'NumRestarts': 0, 'Timestamp': 0.0, 'StartTime': 0, 'EndTime': 0}}

2

Another question, can you please describe what does state: “int value” indicate in the above log snippet?

3

Hello, can you please help with the above two questions? thanks

4

ray.state.actors() is not a supported public API call for Ray (we will be privatizing these internal APIs properly soon).

Are there other APIs you can use for your use case?

5

Hello, @ericl thanks for your reply. ray.state.actors() API is helpful to us to discover actors running on different ray nodes. we are using this API for our use case currently. Can you advise if there is another public API that can help us get actor details from GCS?

Thanks

6

One public API (beta) you can use is ray.util.list_named_actors(), which gives you a list of named actors that you can query via get_actor().

We are considering extending this to include more metadata. Can you say more about what metadata you need and why you need it?

7

I am yet to use list_named_actors() but I think it is not as rich as ray.state.actors():

  1. For list_named_actotrs() to work user actors should be named
  2. It does not provide IP address of workers running actors
  3. It does not provide state information
  4. It does not provide metadata about actors which are detached (same is the case with ray.state.actors())

We need metadata to evict ray nodes in a running fixed-size Kubernetes cluster. We want to evict ray nodes that run tasks only if possible.

We have been discussing it here: Autoscaler node termination behavior when scaled down with helm - #5 by Dmitri