-
Notifications
You must be signed in to change notification settings - Fork 289
Add first draft of nf-core/config specifications #4087
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
b4a831e
e5af88b
8b8094f
3daea74
10fcc07
ffb5e80
f32c992
5567571
de300ca
6ee1dd2
c571683
eb55e41
82df60b
fc0b36f
7508ade
02ffc20
8360529
f08249b
8b86bb4
49511fe
ddb9ace
4d7904d
3f3196c
cb1632f
4f0b628
45ab7ae
d42befb
74e4a2a
ea9a706
1dedf9e
0f30af0
db9a0f3
545416c
c6db0e7
dd1709c
f1421c8
0d333bf
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| title: Reviewing nf-core configs | ||
| subtitle: Review configs pull requests | ||
| shortTitle: Reviewing configs | ||
| markdownPlugin: checklist | ||
| --- | ||
|
|
||
| nf-core/configs reviews ensure that [configs](https://nf-co.re/configs/) adhere to the nf-core/config specifications and ensure consistency across all configs. | ||
| When you review a nf-core/configs pull request, you examine a new config submission or proposed changes to an existing component and provide constructive feedback before maintainers merge them into the nf-core repository. | ||
|
|
||
| :::note{title="Component specifications"} | ||
| For information about best practices for nf-core configs, see [configs specifications](../../../specifications/configs/overview#configs-specifications). | ||
| ::: | ||
|
|
||
| ## General | ||
|
|
||
| - [ ] Permission by infrastructure administrators confirmed | ||
|
|
||
| For institutional configs: | ||
|
|
||
| - [ ] Each config has a `*.config` Nextflow configuration file under `conf/` | ||
| - [ ] Each file has a `*md` documentation file under `docs/` | ||
| - [ ] Does not include `withName` or `withLabel` specifications | ||
| - [ ] Includes `resourceLimits` scope | ||
|
|
||
| For pipeline-specific institutional configs | ||
|
|
||
| - [ ] Each config targets a single pipeline | ||
| - [ ] Each config has a `*.config` Nextflow configuration file under `conf/pipeline/<pipeline_name>/` | ||
| - [ ] Each file has a `*md` documentation file under `docs/pipeline/<pipeline_name>/` | ||
|
|
||
| ## Naming | ||
|
|
||
| - [ ] The name is all lowercase | ||
| - [ ] The name includes only letters, numbers and/or underscores | ||
| - [ ] The name uses no other characters or punctuation | ||
| - [ ] If multiple clusters, uses an institutional suffix (e.g. `ucl_` for all University College London [UCL] configs) | ||
|
|
||
| ## Config contents | ||
|
|
||
| - [ ] Each config adheres to the Nextflow strict syntax | ||
| - [ ] Each config includes parameters: | ||
| - [ ] `config_profile_description` | ||
| - [ ] `config_profile_contact` | ||
| - [ ] `config_profile_url` | ||
| - [ ] Custom parameters: | ||
| - [ ] Are documented | ||
| - [ ] Are added to the `ignoreParams` option of the `validation` scope | ||
|
|
||
|
jfy133 marked this conversation as resolved.
|
||
| For sensitive data or offline clusters (if applicable): | ||
|
|
||
| - [ ] Local container registry configured | ||
| - [ ] External downloads disabled or documented | ||
| - [ ] Data governance/policy requirements documented | ||
|
|
||
| ## Documentation | ||
|
|
||
| - [ ] Contains information on how to use the config in the context of the instruction | ||
| - [ ] (Optional) documentation includes additional recommendations (e.g. scratch, temporary, or environmental variables) | ||
| - [ ] Required parameter or environmental variables described | ||
| - [ ] All commands are copy-paste-able examples | ||
|
|
||
| ## Config infrastructure | ||
|
|
||
| - [ ] Entry added to `nfcore_custom.config` | ||
| - [ ] Entry added to `.github/workflows/main.yaml` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| --- | ||
| title: Documentation | ||
| subtitle: Follow config documentation guidelines | ||
| markdownPlugin: addNumbersToHeadings | ||
| shortTitle: Documentation | ||
| weight: 1 | ||
| --- | ||
|
|
||
|
jfy133 marked this conversation as resolved.
|
||
| The keywords "MUST", "MUST NOT", "SHOULD", etc. are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). | ||
|
|
||
| ## General | ||
|
jfy133 marked this conversation as resolved.
|
||
|
|
||
| A config MUST have a Markdown based documentation file describing the usage of the config. | ||
|
|
||
| - **`docs/<config name>.md`** for an institutional config | ||
| - **`docs/pipeline/<pipeline>/<config name>`** for an institutional pipeline specific config | ||
| - **`docs/pipeline/<pipeline>/<pipeline>.md`** for a global pipeline specific config | ||
|
|
||
| A pipeline specific config MUST be linked to within the institutional level config documentation. | ||
|
|
||
| ## Scope of documentation | ||
|
|
||
| A config documentation Markdown file MUST contain information on how to use the config in the context of the infrastructure. | ||
| For example, the documentation SHOULD describe how to load Nextflow and other requirements for execution, such as `module load <...>` or `conda activate <...>`. | ||
|
|
||
|
jfy133 marked this conversation as resolved.
|
||
| The documentation MAY also provide additional recommendations such as how to specify scratch or temporary directories using Nextflow-related [environment variables](https://www.nextflow.io/docs/latest/reference/env-vars.html#nextflow-settings). | ||
|
|
||
| The config documentation Markdown file MUST NOT contain generic information, such as installing Nextflow. | ||
|
|
||
|
jfy133 marked this conversation as resolved.
|
||
| For offline or sensitive data infrastructures, the documentation MUST describe | ||
| any special steps required (e.g., pre-pulling containers, local reference paths). | ||
|
|
||
| ## Description of environmental variables | ||
|
|
||
| Any mandatory parameter or environment variable required for pipeline execution MUST be described in the documentation. | ||
| Non-mandatory but common parameters or environment variables SHOULD be described. | ||
|
|
||
| ## Examples in documentation | ||
|
|
||
| It is RECOMMENDED to include copy-paste-able command examples of launching a nf-core workflow with the `test` profile (e.g. using `$USER` rather than placeholders). | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,94 @@ | ||||||
| --- | ||||||
| title: Formatting | ||||||
| subtitle: Configuration file formatting requirements | ||||||
| markdownPlugin: addNumbersToHeadings | ||||||
| weight: 9 | ||||||
| --- | ||||||
|
|
||||||
| The keywords "MUST", "MUST NOT", "SHOULD", etc. are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). | ||||||
|
|
||||||
| ## Strict syntax | ||||||
|
|
||||||
| nf-core/configs MUST be [Nextflow strict syntax](https://www.nextflow.io/docs/latest/strict-syntax.html) compliant to ensure compatibility with all nf-core pipelines. | ||||||
|
|
||||||
| Compliance of a config to the strict syntax can be checked using the methods described in the strict syntax migration [guide](../../developing/migration-guides/config-strict-syntax). | ||||||
|
|
||||||
| ### Variables | ||||||
|
|
||||||
| Variables MUST be defined as a parameter (`param`) for reusing information to different parts of a config. | ||||||
|
|
||||||
| ```groovy | ||||||
| - def variable_name = <code> | ||||||
| + params.variable_name = <code> | ||||||
| ``` | ||||||
|
|
||||||
| Variables that are defined using a condition MUST be wrapped in a closure and executed with the `.call()` function. | ||||||
|
|
||||||
| ```groovy | ||||||
| params.variable_name = { | ||||||
| if (large_data == true) { | ||||||
| return 'small' | ||||||
| } else { | ||||||
| return 'big' | ||||||
| } | ||||||
| }.call() | ||||||
| ``` | ||||||
|
|
||||||
| ### Functions | ||||||
|
|
||||||
| Functions MUST NOT be used within a config. | ||||||
|
|
||||||
| ### Conditional statements | ||||||
|
|
||||||
| Simple if-else statements SHOULD be written using a ternary operator. | ||||||
|
|
||||||
| ```groovy | ||||||
| process.executor = params.slurm ? 'slurm' : 'local' | ||||||
| ``` | ||||||
|
|
||||||
| Larger conditions SHOULD be written using if-else statements. | ||||||
| Switch conditions MUST NOT be used. | ||||||
|
|
||||||
| ```groovy | ||||||
| queue = { | ||||||
| if (task.memory >= 216.GB) { | ||||||
| if (task.time >= 7.d) { | ||||||
| return 'longmem' | ||||||
| } else { | ||||||
| return 'mem' | ||||||
| } | ||||||
| } else { | ||||||
| if (task.time >= 21.d) { | ||||||
| return 'long60' | ||||||
| } else if (task.time >= 7.d) { | ||||||
| return 'long' | ||||||
| } else if (task.time >= 48.h) { | ||||||
| return 'medium' | ||||||
| } else { | ||||||
| return 'short' | ||||||
| } | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
|
jfy133 marked this conversation as resolved.
|
||||||
| If related HPCs are being referred to in a single config, conditional sub-config files MAY be included by passing a closure to `includeConfig()`: | ||||||
|
|
||||||
| ```groovy | ||||||
| includeConfig ({ | ||||||
| if (condition) { | ||||||
| return "conf1.config" | ||||||
| } else if (condition2) { | ||||||
| return "conf2.config" | ||||||
| } else { | ||||||
| return "/dev/null" | ||||||
| } | ||||||
| }.call()) | ||||||
| ``` | ||||||
|
|
||||||
| ### Environmental Variables | ||||||
|
prototaxites marked this conversation as resolved.
|
||||||
|
|
||||||
| Environmental variables MUST be referenced with a `System.getenv` call. | ||||||
|
|
||||||
| ```groovy | ||||||
| scratch = "/scratch/${System.getenv('USER')}" | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Isn't this
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't actually know! I've only been seeing the |
||||||
| ``` | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're missing pipeline specific configs (ie viralrecon)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do'nt know anything about that, you need to write that section ;)