feat: Dataset#category

[tristan-f-r]

• Depends on test: dataset #326

This is a short PR that just adds the Dataset#category parameter and parses it/stores it without usage, as I want to make sure that we agree on category over categories: the latter, while more general, seems biologically useless.

[read-the-docs-community]

Documentation build overview

📚 spras | 🛠️ Build #31218800 | 📁 Comparing dc40e7f against latest (18f2cf8)

🔍 Preview build

No files changed.

[ntalluri]

Could you update the config files so we can what the use of category label would look like? Also is a user now required to add a dataset category to each dataset they put in a config?

[tristan-f-r]

The Optional tag in the schema means that category would be optional.

[tristan-f-r]

Could you update the config files so we can what the use of category label would look like?

I'll avoid this in the main config.yaml until we have sample data for a new dataset. I'll add accompanying documentation once we have some way to have runs only run on specific dataset categories.

[agitter]

I'm missing what we want to support with this category attribute. We want to list many datasets in a single config file and then somewhere else say only run on datasets where category=X? That was part of #309, but I'm not seeing the advantages of doing that all within a single config file.

[tristan-f-r]

This would be useful for cross-dataset-category analysis (i.e. unified statistics that check how well algorithms do on certain dataset categories than other categories). This is currently no longer useful for parameter tuning.

[ntalluri]

Would this PR’s goal also be to compute summary statistics and ML outputs across all datasets and across algorithms? I’m not fully sure what the intended use case is for cross–dataset-class comparisons other than it being maybe being useful for the benchmarking study.

Also, I think there may be a simpler way to implement this: instead of specifying dataset categories per algorithm, could we add an option in the ML step to automatically run over all datasets listed in the config and all the enabled algorithms by default? That would avoid needing extra per-algorithm category settings.

[tristan-f-r]

This PR's goal is motivated by both the benchmarking study and the CI for spras-benchmarking. We talked about this PR some time ago, and decided that it was not needed since we can accomplish the same goal with two configs, though this ends up being quite annoying in practice (spras-benchmarking CI currently does this).

I'm confused by your ML suggestion: we would still need a way to filter datasets to not run on certain algorithms.