Observation
Requesting DAILYENERGYCONSTRAINT from BIDDAYOFFER_D via
dynamic_data_compiler emits a warning and returns a frame without
that column:
```
WARNING: {'DAILYENERGYCONSTRAINT'} not in /PUBLIC_DVD_BIDDAYOFFER_D_201804010000.feather.
Loading ['SETTLEMENTDATE', 'DUID', 'BIDTYPE']
```
The column is present in AEMO's raw CSV (verified against
PUBLIC_DVD_BIDDAYOFFER_D_201804010000.CSV — DAILYENERGYCONSTRAINT
appears in the I-row header). Something between the CSV read and the
feather write is dropping it.
Reproduced on eras 2018-05, 2021-02, and 2024-09 — spans both the
PUBLIC_DVD_ and PUBLIC_ARCHIVE# archive formats.
Reproduction
```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 00:00:00",
end_time="2018/05/01 01:00:00",
table_name="BIDDAYOFFER_D",
raw_data_location="/tmp/nemosis_cache",
select_columns=["SETTLEMENTDATE", "DUID", "BIDTYPE", "DAILYENERGYCONSTRAINT"],
)
assert "DAILYENERGYCONSTRAINT" in data.columns # fails
```
Scope
Surfaced during the test-suite refactor. Haven't investigated root cause.
Probably defaults.table_columns["BIDDAYOFFER_D"] or a downstream filter/
cache step.
Observation
Requesting
DAILYENERGYCONSTRAINTfromBIDDAYOFFER_Dviadynamic_data_compileremits a warning and returns a frame withoutthat column:
```
WARNING: {'DAILYENERGYCONSTRAINT'} not in /PUBLIC_DVD_BIDDAYOFFER_D_201804010000.feather.
Loading ['SETTLEMENTDATE', 'DUID', 'BIDTYPE']
```
The column is present in AEMO's raw CSV (verified against
PUBLIC_DVD_BIDDAYOFFER_D_201804010000.CSV—DAILYENERGYCONSTRAINTappears in the I-row header). Something between the CSV read and the
feather write is dropping it.
Reproduced on eras 2018-05, 2021-02, and 2024-09 — spans both the
PUBLIC_DVD_ and PUBLIC_ARCHIVE# archive formats.
Reproduction
```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 00:00:00",
end_time="2018/05/01 01:00:00",
table_name="BIDDAYOFFER_D",
raw_data_location="/tmp/nemosis_cache",
select_columns=["SETTLEMENTDATE", "DUID", "BIDTYPE", "DAILYENERGYCONSTRAINT"],
)
assert "DAILYENERGYCONSTRAINT" in data.columns # fails
```
Scope
Surfaced during the test-suite refactor. Haven't investigated root cause.
Probably
defaults.table_columns["BIDDAYOFFER_D"]or a downstream filter/cache step.