Imbalance in electricity sankey demand and supply queries for several II3050v2 scenarios

[kndehaan]

Context

During improving efficiency of Mechanical Turk specs in quintel/mechanical_turk#213, it was found that the electricity_sankey_spec does not pass for several II3050v2 scenarios (NAT, EUR, INT). The underlying queries are turk_electricity_sankey_supply and turk_electricity_sankey_demand.

This comment describes several findings in what the causes might be of the imbalance. It seems to have something to do with the household batteries and DSR in industry ICT, perhaps as well with the forecasting algorithm settings. The actual cause of the problem and the solution still need to be found.

Steps to Reproduce

See the comment reference above for steps to reproduce.

Proposed solutions

Here are some suggestions for possible next steps to solve this issue:

• Find out where exactly in the queries it goes wrong. I think it's in losses_in_storage in the sankey demand query, where the losses of the household batteries are queried. This issue might be related.
• Further dive into the household batteries calculation method
• Further dive into the DSR method

This will require a combination of modeller and developer effort.

[mabijkerk]

It could also be related to quintel/etengine#1218

[mabijkerk]

As described by @kndehaan in quintel/mechanical_turk#213 (comment), this issue was spotted due to the mechanical turk electricity_sankey_spec. For now, the failing scenario set ii3050v2 has been deactivated for this spec. Once we fix this issue we should reactivate to check that it's truly solved.

[mabijkerk]

@kndehaan it also applies to a blank nl2023 scenario (see results below), this means that it's not (only) due to load shifting and storage, since these are not present in a blank scenario.

EACH(
  Q(turk_electricity_sankey_supply),
  Q(turk_electricity_total_supply),
  Q(turk_electricity_sankey_demand)
)

[
  461.22001108102614,
  461.2200110810261,
  461.1900356118175,
]

When I get the hourly electricity download and sum all demand and all supply, I get a much smaller delta of 1 GJ instead of 10 TJ:

For a blank nl2019 scenario there only seems to be a small rounding issue:

[
  544.8512187409992,
  544.8512187409991,
  544.851218740419,
]

[kndehaan]

Interesting insight... Curious what we'll find when we dive deeper into this.

[mabijkerk]

When playing around with the sliders I found that removing the wind turbines would fix the imbalance. When I open a blank scenario again and compare the graph with the merit results, I get the following difference for the future year for the wind turbines. I tested the solar plants and there the issue does not seem to occur:

EACH(
  V(energy_power_wind_turbine_offshore,output_of_electricity),
  SUM(V(energy_power_wind_turbine_offshore,electricity_output_curve))*MJ_PER_MWH,
  V(energy_power_wind_turbine_inland,output_of_electricity),
  SUM(V(energy_power_wind_turbine_inland,electricity_output_curve))*MJ_PER_MWH,
)

[
  42,161,887,831.40724,
  42,152,115,180.79414,
  64,128,404,186.19273,
  64,105,991,317.34586,
]

[louispt1]

I believe the reason for the imbalance in nl2023 (while nl2019 has no issues) is because of an imperfect normalisation of the nl2023 wind curves. The sum of these curves * 3600 is:

2023:
  wind_inland_baseline: 0.999999992091242
  wind_offshore_baseline: 0.999999995105771
2019:
  wind_inland_baseline: 0.999999999999996
  wind_offshore_baseline: 0.999999999999998

2019 is much close to 1 than 2023

There is also higher wind production in 2023 which amplifies the situation.

Further investigation shows that there are several wind curves affected this way, also NO_norway, GB_great_britain, RS_serbia and CH_switzerland. To my view, these is all the recently exported datasets, so this issue probably originates somewhere in the data pipeline.

Proposed fix

I haven't looked into this deeply, but there should be strong validation on the curves coming into ETSource from our data pipeline, and that validation should (imo) take place in Atlas. At the moment the checks are very light, and the sum to 1/3600 is not checked.

This issue is amplified (or perhaps even caused by) our floating point difficulties, and a robust fix for those would also go some way to making validation at each step easier.

[kndehaan]

I thought in another fix recently, we improved the load_profiles_spec since it failed for nl2023 and GB_Great_Britain where the curves did not add to 1/3600 (see PR). So I'm curious why it doesn't fail for nl2023 wind curves. @louispt1 could you have a look at this?

Anyhow, I'm going to update the curves so that they do add to 1/3600.