Blocked validation for build costs (workbook v7.8)

[dylanjmcconnell]

The Build cost table (in Build cost sheet - sheet 55) - has an some hidden additional data that, in an extra column to the right of the table (seems to be the previous column + 1, for every ninth value - 🤷, see screen shot with hidden values highlight, at the bottom of issue). The actual table of data itself looks unchanged (i.e. reading columns B:AI returns the right data / same as version 7.5)

Validation issue

This extra data means that right-adjacent-column check (_check_for_missed_column_on_right_hand_side_of_table) fails - as it should:

TableConfigError: There is data in the column adjacent to the last column

If you tried to increase the the column range - i.e. from B:AI to B:AJ - it would also fail (specifically the last-column check instead, since AJ has no header). Probably not a good idea to do this an anycase, because it's dodgy data / would have to be cleaned up downstream .

Potential (easy?) options

1. Editing workbook: The first "obvious" / easy fix is to just delete the dodgy column from the workbook, and commit a patched version of the workbook in the repo. Downsides of this include:
   • Con: folks that download the AEMO workbook will have to parse config_checks=False when using this table (i.e. the unmodified table will still fail the checks)
   • Con: Having a modified workbook in the repo (which we have historically / probably should?) avoid.
   • Pro: no modification of code or anything
2. Marking expected failed test - using "xfail" for build costs in v7.8 (i.e. so still passes CI)
   • Con: As above, will have to parse config_checks=False when using this table
   • Pro: Don't have to store modified workbook (but do have to add a new test)

More substantial options

1. Omit the table completely - users would have to make their own TableConfig for this table (.. given it's a rather important table not ideal).
2. Mode substantive code/functionality changes - eg an option to skip specific tests for specific tables (?).. Haven't really thought this through / seems overkill - but something like this in the config:

   build_costs:
     ...
    column_range: "B:AI"
    skip_checks: ["missed_column_on_right_hand_side"]

   With additional code to manage..
   • Pro: No changes to usage pattern for end users
   • Con: Potential for this to be forgotten / skipped / not checked as new versions added
   • Con: code change, not just config change

Wishful thinking:

1. Email AEMO to update the table :)

Screenshot

ail

[nick-gorman]

I'm probably in favour of adding the skip_checks option. In fact I half thought we might have already added it, but it appears not. The way I think of checks is they are a crutch to catch typos and errors in the configs, which are always likely to appear given the number of tables. If we, the maintainers, are happy to say we have double checked this tables config, then its fine to opt out of the checks. The code could be simpler if we just had skip_checks: True | False, but I agree the list is nicer, i.e. just skipping the check which is a problem. Seems like an addition claude would code up for you in about 10 mins ;)

[dylanjmcconnell]

Okay cool - thanks.. Will take a look.. It's actually probably not too much now you mention it. Maybe can add a warning message or something so don't forget that check is skipped in the future too.

.. And as optional field in the yaml it wouldn't need to touch too many configs either (..just this one in the immediate term).