Investigate DCAT-US Conversion Errors

[akuny]

NIST and DOT's DCAT-US data has more validation errors in v3.0 than in v1.1. Investigate to determine why the error count rises following the conversion process.

[akuny]

@jbrown-xentity Findings re NIST and DOT's high v3.0 validation error counts:

1. https://data.transportation.gov/data.json

Datasets in this catalog use a repeating interval string (for example, "R/2015-01-26/PT1S") for "temporal". After adding logic to handle repeating interval strings, 33 datasets fail v3.0 validation (down from 415).

2. https://www.nist.gov/sites/default/files/data.json

This catalog contains 17 datasets that use the unreserved "replaces" field in v1.1. "replaces" is a reserved field with specific requirements in v3.0. Since the only valid non-null value for this field is a "list of Datasets replaced by this Dataset", these values cannot be automatically converted:

6998B81EF78777B2E05324570681D4DC1911
ark:/88434/mds2-2313
ark:/88434/mds2-2348
ark:/88434/mds2-2388
ark:/88434/mds2-2448
ark:/88434/mds2-2352
ark:/88434/mds2-2126
ark:/88434/mds2-2428
ark:/88434/mds2-2598
ark:/88434/mds2-2698
ark:/88434/mds2-2734
ark:/88434/mds2-2751
ark:/88434/mds2-2997
ark:/88434/mds2-3054
ark:/88434/mds2-3118
ark:/88434/mds2-3189
ark:/88434/mds2-3348

[jbrown-xentity]

So in summary, we were able to improve transportations counts to be less than the current 1.1 errors, resolving that one.

NIST is still a problem, because they use a reserved field, and we can't make it valid without changing that field name (which may cause problems for NIST, as we don't know what that field is for).

So in summary: every single valid object in 1.1 can be successfully translated to valid 3.0. 5% of invalid 1.1 objects are successfully translated to 3.0.

The only action item is to reach out to NIST to prepare them for what is coming @SueValente , and that the custom field name of replaces will need to change if they still want to track that information.