Skip to content

Allow integer type for L0A parquet files #114

@ghiggi

Description

@ghiggi

Description

In this commit I had to remove all integer types from the L0A_encodings.yml files because if there is a nan value in an integer column, an error will be raised during column casting.

This change increases the size of the L0A Parquet files.

To reintroduce the integer type, it would be required:

  • to define a dictionary with the nan flag value for each variable which is expected to be cast to an integer format.
  • during L0A processing replace the nan values using the nan_flag dictionary
  • during L0B processing, replace the nan_flag with np.nan (by casting the variable to float) or ensure that the nan_flag value corresponds to the _FillValue of the L0B encoding.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions