I guess this is a Ray bug when calling Java actors from Python, with Ray.init have tracing_startup hook argument set. Java actor not able to initialized with exception “Cross language remote functions does not support kwargs”
It is easily reproducible with similar code like below
ray.init(job_config=ray.job_config.JobConfig(
code_search_path=["/Users/Shared/Projects/libs"]),
_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing"
)
java_class = ray.cross_language.java_actor_class(
"com.mycompany.ml.operators.FeatureFetcherOperator").remote("constructor argument 1")
counter = counter_class.execute.remote("method argument 1")
And it will produce stack trace. Which the kwargs is set to be the tracing context.
../../build/venv/lib/python3.10/site-packages/ray/actor.py:536: in remote
return self._remote(args=args, kwargs=kwargs, **self._default_options)
../../build/venv/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py:399: in _invocation_actor_class_remote_span
result = method(self, args, kwargs, *_args, **_kwargs)
../../build/venv/lib/python3.10/site-packages/ray/actor.py:886: in _remote
creation_args = cross_language._format_args(worker, args, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
worker = <ray._private.worker.Worker object at 0x11302ffa0>, args = ('/',)
kwargs = {'_ray_trace_ctx': {'traceparent': '00-54311cd6786df9a9d82c8042ff985fbe-df997918ccbcd845-01'}}
def _format_args(worker, args, kwargs):
"""Format args for various languages.
Args:
worker: The global worker instance.
args: The arguments for cross language.
kwargs: The keyword arguments for cross language.
Returns:
List of args and kwargs (if supported).
"""
if not worker.load_code_from_local:
raise ValueError(
"Cross language feature needs --load-code-from-local to be set."
)
if kwargs:
> raise TypeError("Cross language remote functions does not support kwargs.")
E TypeError: Cross language remote functions does not support kwargs.
../../build/venv/lib/python3.10/site-packages/ray/cross_language.py:100: TypeError