Data Transmission Errors in Multi-Environment PPO Training with comtypes on Windows 10

RLlib
1
  • High: It blocks me to complete my task.

I am training an agent using the PPO algorithm to interact with other software on the local end. In a custom environment, I use the comtypes module to operate the target software through its API. The code related to the API and PPOConfig is as follows…
During training, when using multiple environments (each environment with a separate software process PID), there are some errors related to data transmission, which subsequently cause the target software to crash and exit. The validity of the environment has already been verified. This issue does not occur with a single environment.
However, using other frameworks, such as parallel environments in tianshou, does not cause the above problem. The version of ray is 2.34.0, Python version is 3.9.18, and the operating system is Windows 10. If anyone can help me, I would be very grateful.

        helper = comtypes.client.CreateObject("ETABSv1.Helper")
        helper = helper.QueryInterface(comtypes.gen.ETABSv1.cHelper)
        try:
            myETABSObject = helper.GetObjectProcess("CSI.ETABS.API.ETABSObject",pid) 
        except (OSError, comtypes.COMError):
            sys.exit(-1)

   register_env("StrucSystemDesiEnv", env_creator)
    pids = [45616,27320]   # The target software process PIDs   

#     PPOConfig
    config = {
        "env": "StrucSystemDesiEnv",
        "env_config": {
            "pid": tune.grid_search(pids),  
            "Check":False
        },
        "num_workers": len(pids), 
        "num_envs_per_worker": 2, 
        "sample_async": False, 
        "seed":42,
        "gamma":0.90,
        "lr": 0.0001,
        "kl_coeff":0.2,
        "entropy_coeff":0.01,
        "train_batch_size": 8, 
        "sgd_minibatch_size":2, 
        "num_sgd_iter":10,      
        "framework": "torch",
        "evaluation_interval": 20,
        "evaluation_duration": 10, 
        "evaluation_duration_unit": "episodes", 
        "evaluation_config": {
            "explore": False
        },
        "model":{
            "fcnet_hiddens":[64,64,64],
            "fcnet_activation": "relu",
            "use_attention": False,
            "custom_model": "my_model",
        },
        "vf_lr" : 0.0005,
        "exploration_config":{
            "type": "StochasticSampling"
        },
         "disable_env_checking": True,  
         "callbacks": MyCallbacks,
    }

Error
Error482×572 91 KB