[FEATURE] Columnar JSON serialization format

[mgovers]

Currently, the PGM JSON (and MessagePack) format (https://power-grid-model.readthedocs.io/en/stable/user_manual/serialization.html#json-serialization-format-specification) are all row-major. That is, one has to specify all attributes for the same component at once.

It would be nice to also be able to specify columnar component data. This addition needs to be made to the specification and the C++ (de-)serialization.

NOTE: the existing format can still be loaded in as either row-based or columnar data format. This issue is specifically about the serialized data.

NOTE: we probably need to bump the JSON schema version from 1.0 to 2.0

Proposal for format

Row-based homogeneous component data (existing)

Introduced in #320

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {
    "sym_load": ["id", "p_specified"],
    "asym_load": ["id", "p_specified"]
  },
  "data": {
    "sym_load": [
      [1, 1.0e4],
      [2, 1.1e5]
    ],
    "asym_load": [
      [4, [1.0e4, 1.1e4, 1.2e4]],
      [3, [1.1e5, 1.2e5, 1.3e5]]
    ]
  }
}

Row-based inhomogeneous component data (current)

Introduced in #320

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "sym_load": [
      {"id": 1, "p_specified": 1.0e4},
      {"id": 2, "p_specified": 1.1e5}
    ],
    "asym_load": [
      {"id": 4, "p_specified": [1.0e4, 1.1e4, 1.2e4]},
      {"id": 3, "p_specified": [1.1e5, 1.2e5, 1.3e5]}
    ]
  }
}

Columnar component data (this issue)

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "sym_load": {
      "id": [1, 2],
      "p_specified": [1.0e4, 1.1e5]
    },
    "asym_load": {
      "id": [3, 4],
      "p_specified": [
        [1.0e4, 1.1e4, 1.2e4], 
        [1.1e5, 1.2e5, 1.3e5]
      ]
    }
  }
}

Row-based homogeneous batch component data (existing)

Introduced in #320

{
  "version": "1.0",
  "type": "update",
  "is_batch": true,
  "attributes": {
    "sym_load": ["id", "p_specified"],
    "asym_load": ["id", "p_specified"]
  },
  "data": [                         // scenarios
    {                               // scenario 0
      "sym_load": [
        [1, 1.0e4],
        [2, 1.1e5]
      ],
      "asym_load": [
        [3, [1.0e4, 1.1e4, 1.2e4]],
        [4, [1.1e5, 1.2e5, 1.3e5]]
      ]
    },
    {                               // scenario 1
      "sym_load": [
        [1, 2.0e4],
        [2, 2.1e5]
      ],
      "asym_load": [
        [3, [2.0e4, 2.1e4, 2.2e4]],
        [4, [2.1e5, 2.2e5, 2.3e5]]
      ]
    }
  ]
}

Row-based inhomogeneous batch component data (current)

Introduced in #320

{
  "version": "1.0",
  "type": "update",
  "is_batch": true,
  "attributes": {},
  "data": [                                              // scenarios
    {                                                    // scenario 0
      "sym_load": [
        {"id": 1, "p_specified": 1.0e4},
        {"id": 2, "p_specified": 1.1e5}
      ],
      "asym_load": [
        {"id": 3, "p_specified": [1.0e4, 1.1e4, 1.2e4]},
        {"id": 4, "p_specified": [1.1e5, 1.2e5, 1.3e5]}
      ]
    },
    {                                                    // scenario 1
      "sym_load": [
        {"id": 1, "p_specified": 2.0e4},
        {"id": 2, "p_specified": 2.1e5}
      ],
      "asym_load": [
        {"id": 3, "p_specified": [2.0e4, 2.1e4, 2.2e4]},
        {"id": 4, "p_specified": [2.1e5, 2.2e5, 2.3e5]}
      ]
    }
  ]
}

Scenario-major columnar batch serialization format (this issue)

The hybrid between the current batch serialization format and columnar component data

{
  "version": "1.0",
  "type": "update",
  "is_batch": true,
  "attributes": {},
  "data": [                           // scenarios
    {                                 // scenario 0
      "sym_load": {
        "id": [1, 2],
        "p_specified": [1.0e4, 1.1e5]
      },
      "asym_load": {
        "id": [3, 4],
        "p_specified": [
          [1.0e4, 1.1e4, 1.2e4], 
          [1.1e5, 1.2e5, 1.3e5]
        ]
      }
    },
    {                                 // scenario 1
      "sym_load": {
        "id": [1, 2],
        "p_specified": [2.0e4, 2.1e5]
      },
      "asym_load": {
        "id": [3, 4],
        "p_specified": [
          [2.0e4, 2.1e4, 2.2e4], 
          [2.1e5, 2.2e5, 2.3e5]
        ]
      }
    }
  ]
}

Component-major columnar batch serialization format

Going columnar component data all the way

{
  "version": "1.0",
  "type": "update",
  "is_batch": true,
  "attributes": {},
  "data": {
    "sym_load": {
      "id": [   // scenarios
        [1, 2], // scenario 0
        [1, 2]  // scenario 1
      ],
      "p_specified": [  // scenarios
        [1.0e4, 1.1e5], // scenario 0
        [2.0e4, 2.1e5]  // scenario 1
      ]
    },
    "asym_load": {
      "id": [   // scenarios
        [3, 4], // scenario 0
        [3, 4]  // scenario 1
      ]
      "p_specified": [           // scenarios
        [                        // scenario 0
          [1.0e4, 1.1e4, 1.2e4], 
          [1.1e5, 1.2e5, 1.3e5]
        ],
        [                        // scenario 1
          [2.0e4, 2.1e4, 2.2e4], 
          [2.1e5, 2.2e5, 2.3e5]
        ]
      ]
    }
  }
}

Additional considerations

• Columnar component data cannot have columns of varying length, because that would result in inconsistencies in the mapping of IDs to the other attributes.
• It is still possible to not specify some attributes.
• Pre-specifying the attributes is still allowed, but doesn't have any benefits, similarly to how it works for inhomogeneous data.
• There is no data format conflict, because the current form has "node": Array, while the added feature looks like "node": Object. Everything is backwards compatible.
• We cannot go with an equivalent format in which we do not specify the attributes at component level, like for row-based homogeneous data, because that would cause a data format conflict and name clashes

[figueroa1395]

Taking inspiration from the internal data structure on how we define columnar data, would it be possible to instead leave the "data" entry empty and add the (batch) columns at the "attributes" level? I haven't yet dealt with (de)serialization implementation details, but perhaps that makes it more natural. Something along the lines of:

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {
     "node": {
      "id": [1, 2],
      "u_rated": [10.5e3, 400.0]
     }
  },
  "data": {}
}

If this isn't crazy, perhaps this can be used (after some refinement) to even allow inhomogeneous columnar data.

In spite of my proposal, would we want to support optional IDs in from this perspective as well? That assuming we can come up with a way to do inhomogeneous columnar data.

[mgovers]

Taking inspiration from the internal data structure on how we define columnar data, would it be possible to instead leave the "data" entry empty and add the (batch) columns at the "attributes" level? I haven't yet dealt with (de)serialization implementation details, but perhaps that makes it more natural. Something along the lines of:

{
"version": "1.0",
"type": "input",
"is_batch": false,
"attributes": {
"node": {
"id": [1, 2],
"u_rated": [10.5e3, 400.0]
}
},
"data": {}
}

If this isn't crazy, perhaps this can be used (after some refinement) to even allow inhomogeneous columnar data.

In spite of my proposal, would we want to support optional IDs in from this perspective as well? That assuming we can come up with a way to do inhomogeneous columnar data.

no, the attributes section is, just like version, type and is_batch part of the header of the JSON structure and should not contain data. The data section is specifically reserved for data.

[mgovers]

we also probably need to bump the JSON schema version from "version": 1.0 to "version": 2.0 as this is a significant change that is not compatible with older PGM versions. NOTE: v1 is a subset of v2, so it is not breaking.

However, the current PGM core does not check the version string, so we need to be very careful with how to deal with v2 JSON format on PGM versions that only support v1

[TonyXiang8787]

@mgovers Thanks for raising this issue. I am hesitating, however, on how much storage footprint can be saved with this new columnar serialization data model.

We introduce columnar memory model for C-API, because the row-based model contains the full attributes, regardless if you are using them of not. But this is not the case for compact list mode of row-based serialization. The compact list is already very dense. I doubt weather a columnar serialization data model can actually provide significant footprint reduction of storage.

Maybe you want to manually create msgpack large datasets for current compact list and the proposed columnar data model, to see if there is significant reduction.

[mgovers]

The main selling point is not the storage footprint. Instead, it is the interface how people interact in real-world settings in which there is a column-major data table, especially in cross-process applications (e.g. across web services).

For instance, creating a PGM JSON dataset from a table can happen as follows using e.g. Azure Functions

def main(star_shaped_database, semi_structured_database):
    result = ""
    write_header(result)

    for component, component_data in star_shaped_database:
        write_component_header(result, component, component_data)

        for attribute, attribute_data in component_data.columns:
            write_attribute_header(result, attribute, attribute_data)
            write_attributes(result, attribute_data)
            write_attribute_footer(result, attribute, attribute_data)

        write_component_footer(result, component, component_data)

    write_footer(result)
    write_result_to_other_database(result, semi_structured_database)

the way to iterate over the database is clearly columnar and therefore this use case is very natural for users to do

[SteffanWullems]

This would help with with using PGM when having pipeline processing. The current format is very hard to use due to most external systems can not deal with the structure.

[TonyXiang8787]

@SteffanWullems Thanks for your valuable insight. Can you confirm the proposed format by @mgovers fits your expectations?

@mgovers, can you also make an example for batch dataset in columnar serialization? Do we have a dict on the data key with all components? Or do we still have a list of scenarios, then inside scenario a dict of attributes for that scenario?

[mgovers]

@mgovers, can you also make an example for batch dataset in columnar serialization? Do we have a dict on the data key with all components? Or do we still have a list of scenarios, then inside scenario a dict of attributes for that scenario?

we can support both. i think the latter is easier to implement (it's exactly the same as for input data) but the former is more useful (c.c. @SteffanWullems please provide your insights). I've updated the issue with both options