Aemo filename format 2024 08#72
Conversation
Bring types libraries up to date with their supporting libraries Update poetry.lock to remove 11 security vulnerabilities
Fix broken URLs Update poetry instructions for current version of poetry Document incompatibility of docs and test
…ython now 3.11 Also upgraded some libraries which were quite old
Exclude 2012 to 2014 as data only in zip files, not in folders, on nemweb
Monitor for different response status codes Respect retry-after header if provided by server
for more information, see https://pre-commit.ci
…iles present Change test fixtures so not incredibly slow and also testing newer data. Eventually we want tests that cover many dates not just one so fixtures may not be the best solution for get_test_year_and_month or valid_download_datetimes
|
Some tests are still failing so I am still updating this pull request |
…aying compatible with old format
netcdf can't save CategoricalDtype columns
if using pandas>=3.0.0 or xarray>=2025.4.0 CategoricalDtype columns were throwing an exception
Remove Googlebot user agent because nemweb blocks with 403
Provide enhanced ENUMERATED_TABLES_BY_MONTH to handle differences in table output by month when change happened
Use a separate field for table counter to cope with different AEMO filename formats
Handle AEMO double escaping of # in new filename format ("%2523" = "#")
Construct sqlloader filename in sections so can be used more universally in library
Deprecate function _enumerate_tables which was changing mutable argument in place
|
All tests are now passing including some new ones I have added. Most of the changes are to do with the filename format change and also file existence changes on nemweb. There was a major problem with saving xarray to netcdf. netcdf format can't save pandas Categorical dtype data and now with recent versions of pandas OR xarray it has started throwing exceptions. I have added some lines to replace categorical data with str before converting to netcdf. nemweb has started blocking googlebot which is one of the useragents being used so I have removed it. |
|
Thanks @timcu appreciate you work on this as I no longer have time to actively maintain nemseer. I will merge this in and increment version number/release |
|
@nick-gorman I am no longer in the CEEM org so I can't approve the pending workflow. Are you able to add me as maintainer for NEMSEER or approve the workflow yourself? A new version/release is also required. Also, would someone in CEEM be able to take over managing PRs etc on NEMSEER? |
Use later versions of poetry and python in action workflow also
|
Hi @prakaa ! -
Have sort of been discussing this and/or related issue last few weeks - (Repository organisation and descriptions) - and nemseer is certainly one of repo's flagged as one we need to do something with. Will sort something out (...eventually / soon) - but will looking to perhaps allocate different people (...i.e. not just @nick-gorman ) to different repo's |
Remove some code which is no longer used
As requested by GitHub actions warning
…rlier than this Fix some spelling mistakes
…ld version of numpy which throws warnings in tests about binary incompatibility
Include tests and docs for new functions Fix some docstring typos Enable test for mtpasa duid availability
Replace grequests with requests-futures to remove incompatibility between docs and test Address warnings regarding data fragmentation and unexpected data types Reverted documentation so now OK to use docs and test together Ignore incorrect warnings created by numpy caused by cython
|
Please run CI/CD workflow on my latest commits. I have altered the tests to ensure it doesn't randomly use 2014 or earlier data which is incompatible with NEMSEER. I am not a big fan of randomness in testing. That was why the test was passing on my computer and then failed here. I have left the randomness in there but ensured it can't go outside the valid range. I have also addressed some of the warnings which were being generated during the tests. I have also replaced grequests with requests-futures. grequests installed gevent which monkey patched threads and was incompatible with anyio (used in docs) potentially causing python to hang during testing. Even the author of grequests has recommended requests-futures because of these incompatibility issues. |
|
Please approve pending workflow @nick-gorman or @dylanjmcconnell |
|
Thanks for running the tests. It appears that nemweb have changed the URL format between me writing the code and you running the tests because these tests all passed on my computer. They were double escaping # in the URLs and now they are just single escaping #, which is what they should have done all along. The tests are now failing on my computer also. I will adjust the code for the change. |
They are now escaped to %23 as they should have been all along.
|
I have updated code to cope with single escaping of # in URLs. I have also updated the CI/CD workflow to use the latest actions to get rid of warnings about Node 20 not available. I have tested the CI/CD workflow in my github area and it passes all tests. I believe the last failure on your github account was due to codecov rate limit for your account. @nick-gorman Please rerun the workflow on the latest commit. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #72 +/- ##
==========================================
- Coverage 96.34% 94.89% -1.45%
==========================================
Files 9 10 +1
Lines 739 823 +84
Branches 153 127 -26
==========================================
+ Hits 712 781 +69
- Misses 15 26 +11
- Partials 12 16 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3 participants
The main purpose of this pull request was to address issue #71 . On the way I made several other changes that were necessary
poetry updatefixed those security issues, except that the requiredrequestslibrary wasn't compatible back to python 3.8 so I increased the lowest supported python to 3.10. That shouldn't be controversial because python.org only support as far back as 3.10.nemseeron python 3.14 so I increased the maximum python version. Then poetry complained about some dependencies not allowing this. I noticed that pandas-stubs was old and mismatched to pandas so I upgraded all the stubs and types libraries to match the installed libraries.numpyrequired minimum python 3.11. I upgraded that inpyproject.toml. This was a more controversial decision, but will only be controversial until Oct 2026 when python.org stop supporting python 3.10.CONTRIBUTING.mdthat docs (requiresanyio) and test (requiresgrequests) were incompatible and couldn't be run together on modern python versions.README.mdDataFrameswhich is not compatible withpandas 3test_allmonths_availableit was hanging at about 2018 with the test continually making the same request of nemweb and continuously getting a 403 response. I rewrote the handling of_rerequest_to_obtain_soupto change the useragent as was so diligently done elsewhere but not here, plus some other handling improvements. This solved the hanging issue.requestsand performance improvements #63 .I hope you find this pull request useful and can pull it into the main code.