Skip to content

dynamic_data_compiler silently drops DAILYENERGYCONSTRAINT from BIDDAYOFFER_D #71

Description

@nick-gorman

Observation

Requesting DAILYENERGYCONSTRAINT from BIDDAYOFFER_D via
dynamic_data_compiler emits a warning and returns a frame without
that column:

```
WARNING: {'DAILYENERGYCONSTRAINT'} not in /PUBLIC_DVD_BIDDAYOFFER_D_201804010000.feather.
Loading ['SETTLEMENTDATE', 'DUID', 'BIDTYPE']
```

The column is present in AEMO's raw CSV (verified against
PUBLIC_DVD_BIDDAYOFFER_D_201804010000.CSVDAILYENERGYCONSTRAINT
appears in the I-row header). Something between the CSV read and the
feather write is dropping it.

Reproduced on eras 2018-05, 2021-02, and 2024-09 — spans both the
PUBLIC_DVD_ and PUBLIC_ARCHIVE# archive formats.

Reproduction

```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 00:00:00",
end_time="2018/05/01 01:00:00",
table_name="BIDDAYOFFER_D",
raw_data_location="/tmp/nemosis_cache",
select_columns=["SETTLEMENTDATE", "DUID", "BIDTYPE", "DAILYENERGYCONSTRAINT"],
)
assert "DAILYENERGYCONSTRAINT" in data.columns # fails
```

Scope

Surfaced during the test-suite refactor. Haven't investigated root cause.
Probably defaults.table_columns["BIDDAYOFFER_D"] or a downstream filter/
cache step.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions