Conversation
To be used by Nextstrain CLI.
| const searchResponse = await fetch("https://api.github.com/search/repositories?q=org:nextstrain+topic:nextstrain+topic:pathogen", {headers}); | ||
| assertStatusOk(searchResponse); | ||
|
|
||
| const searchResults = await searchResponse.json(); | ||
|
|
||
| const repos = await Promise.all( | ||
| searchResults.items.map(repo => repo.name).map(async (repo) => { | ||
| // Fetch pathogen registration file | ||
| const registration = await fetch(uri`https://api.github.com/repos/nextstrain/${repo}/contents/nextstrain-pathogen.yaml`, {headers}); |
There was a problem hiding this comment.
[Very general comments] making fetches like this, even if they can be cached and only require re-validation, seems like sizeable overhead. We've run into this general problem before, and (e.g.) update the datasets in groups every 6 hours, which avoids expensive fetch calls but results in staleness; the entire resource-indexer is another example. I don't know what's best here, but it stuck out to me when scanning the code.
There was a problem hiding this comment.
For the first fetch, a GitHub query does not have to be the source of truth since the list doesn't change very often. It could be (yet another) static file similar to manifest_core.json.
For the second fetch, it'll be harder to avoid GitHub as the source of truth, and unfortunately this one is more costly with 1 fetch per matching repo.
There was a problem hiding this comment.
Something basic like a cache that re-validates no more than every n minutes might be fine for starters. Adding webhooks to the repos we control was what I was originally thinking of proposing, but it may not be worth it to start with?
| app.routeAsync("/pathogen-repos") | ||
| .getAsync(endpoints.pathogenRepos.listRepos); |
There was a problem hiding this comment.
I think we'd want to use content-negotiation here so only Accept: application/json requests send this response, which allows us to one day have a HTML page at the same URI.
3 participants
Description of proposed changes
To be used by Nextstrain CLI.
Related issue(s)
Checklist