Escaped string in checkpoints from Experiment_state.json

RLlib
1

I recently updated Ray version 1.2.0 from 1.0.0.

I noticed the content of experiment_state-xxx.json is changed a bit.
precisely, the value of checkpoints has an escape sequence.

from Ray 1.0.0, we got

{
  "checkpoints": [
    {
      "trainable_name": "PPO",
      "trial_id": "5eb76_00000",
...

from Ray 1.2.0, we got

{
  "checkpoints": [
    "{\n  \"trainable_name\": \"PPO\",\n  \"trial_id\": \"8cee3_00000\", ...

it has \n and \".

image
image1714×1688 278 KB

It only happens at the value of checkpoints but runner_data and stats seem to have valid JSON format.
Do you happen to know how to fix this?
or Can you point me out which codes or documents I should see?

Thank you.

1 Like
2

This was a recent change introduced to make checkpoints faster – what are you trying to do here?

3

Hi @rliaw
thanks for the comment.

we are relying on experiment_state.json to periodically get status of trials during the training.
As I mentioned we changed the ray version recently, It doesn’t work with our current code.
So i tried to make sure this is on purpose or not and thank you for the confirmation.

Could you share the relevant code or PR about this change?
If I could know the change, I’d be helpful to change our code.

Thank you.

1 Like
4

I found it.
looks like [tune] cache checkpoint serialization by krfricke · Pull Request #12064 · ray-project/ray · GitHub

5

Yeah, I think it is preferable that you do not rely on experiment_state.json. You should use something like Analysis or ExperimentAnalysis.

1 Like