whole training dataset

[shijiale0609]

The training data path in the code configuration is the author's local path, not a public URL.

starling/configs/dataloader/dataloader.yaml:4 points to

/work/bnovak/projects/sequence2ensemble/lammps_data/combined_data/data

How to get the whole training dataset?

[alexholehouse]

Hi Jiale!

Very good question - so training/test/val is about 2 TBs total (after compression), so too big for the usual sources where we'd deposit open data. We are trying to figure out options to make this work, but it's not trivial!

We have all the data assembled, but want to double-check that the configs used are up to date. We are just about to travel to the Biophysical Society meeting, so I ask if you can wait a couple of weeks as we figure out (a) what the best options here are and (b) actually move the data to a public destination. Thanks for your patience! We'll follow up here asap.

[wengong-jin]

Thank you for your great work! We are also very interested in this dataset and wonder if it is possible to share at least the pdb / xtc / mdtraj files of your data soon. I think huggingface can host datasets of that size.

[mingze-yuan]

Hi Alex,

Do you have any updates on sharing your training data?

[alexholehouse]

Quick update on this: we are very close to having all the training data formatted and organized for sharing.

That said - to temper expectations – the data we will be able to share is the mapping of sequence to distance maps (so one sequence to many distance maps) - i.e., the data STARLING was trained on, but not the xtc/pdb file (which are the files the distance maps were generated from).

We hope to be able to share those XTC/PDB files at some point in the (more distant) future, but the logistics of organizing THOSE data is a massive undertaking because those simulations were run over a series of years across heterogeneous infrastructure. I'm exploring ways to share this with the broader community, as I completely recognize it would be a useful resource. However, getting that specific data out quickly isn't feasible. BUT all the training data (in a format that would be amenable to retraining) should be out soon, and we are working on a strategy for the XTC/PDB files.

Anyway, I will update here as soon as we have the training data up and shareable - thanks for your patience here.