PGLearn is hosted on HF which has 50GB file limits. For the medium/large cases, some files exceed this limit, so splits are pushed instead, i.e. SOCOPF/dual.h5 is replaced by a folder SOCOPF/dual containing files xaa xab which when cat'd together give the original SOCOPF/dual.h5. ML4OPF should merge these automatically upon first read (like how the HF dataset loading script does here)
PGLearn is hosted on HF which has 50GB file limits. For the medium/large cases, some files exceed this limit, so splits are pushed instead, i.e.
SOCOPF/dual.h5is replaced by a folderSOCOPF/dualcontaining filesxaaxabwhich whencat'd together give the originalSOCOPF/dual.h5. ML4OPF should merge these automatically upon first read (like how the HF dataset loading script does here)