When a tensorflow operation is called from inside a remote function or any of it callees, processes are created on all cores importing even before the remote function starts executing. Is there a way to do the imports only whenever a remote process is called?
Attaching the code and screenshots of the output
TF Version: 1.14
Ray Version: 1.2.0
@ray.remote
def f():
time.sleep(10)
tf.get_default_graph()
print("Printing at " + str(time.time()))
if __name__ == "__main__":
ray.init()
next_idx = 1
current_objs = [f.remote()]
while len(current_objs):
done, current_objs = ray.wait(current_objs)
if next_idx < 100:
current_objs.append(f.remote())
next_idx += 1