release(v0.1.0): base IPG — db_construct + post_ms validated on real data#6
Merged
release(v0.1.0): base IPG — db_construct + post_ms validated on real data#6
Conversation
Signed-off-by: sanjaysgk <44039457+sanjaysgk@users.noreply.github.com>
Add the ms_search pipeline step with three new local modules: - PREPARE_FASTA: generates target-decoy FASTA if decoys absent - MSFRAGGER: runs MSFragger database search (user provides JAR) - MOKAPOT: semi-supervised FDR control on search engine PIN output New subworkflow MS_SEARCH chains: PREPARE_FASTA → MSFRAGGER → MOKAPOT Also adds: - assets/ms_search_params/: MSFragger parameter templates for all instrument/mod combinations (orbitrap/timsTOF × mod/nomod/TMT/mhcii) - assets/schema_ms_input.json: MS data samplesheet validation schema - conf/modules/ms_search.config: publishDir routing - bin/prepare_fasta.py: standalone CLI extracted from Willems core.py New params: --ms_input, --search_fasta, --engines, --msfragger_jar, --msfragger_mem, --instrument, --mod_type, --peptide_length, --peaks_psm_csv Sprint 2 will add Comet, Sage, and CONVERT_MZML modules. Signed-off-by: sanjaysgk <44039457+sanjaysgk@users.noreply.github.com>
Schema now validates the comma-list against msfragger|comet|sage via a pattern regex, so invalid engines fail at validation before any process launches. Updates all four call sites: params default, schema, workflow entry, and the utils_nfcore guard.
Wraps comet-ms bioconda binary. Consumes calibrated mzMLs from MSFragger (or raw mzMLs if MSFragger not in --ms_engines) and emits per-run PIN files for mokapot. Params path passed via -P so the FASTA inside the param file is ignored in favour of -D.
Wraps sage-proteomics. Emits a single combined PIN for all mzMLs in one shot (Sage differs from Comet here). When MSFragger's search_log.txt is staged alongside, Sage inherits the calibrated fragment tolerance and topN peaks — mirrors core.py run_Sage L516-537.
Per-mzML pymzml pass that emits MGF + scans.pkl + index2scan.pkl. Extracted from immunopeptidomics core.py read_mzML L679 and write_MGF L641. scans.pkl is keyed run_scan and carries precursor mz, charge, RT, and ion mobility for downstream MS2Rescore; index2scan.pkl is the spectrum-index→scan mapping PEAKS integration will need in Sprint 4.
mokapot writes both *.mokapot.psms.txt (target) and *.mokapot.decoy.psms.txt (decoy) with overlapping globs, so the Sprint 1 psms output was non-deterministic when --keep_decoys was set. Rename to *.target.psms.txt and *.decoy.psms.txt after mokapot finishes so MS2Rescore prep can pick them up separately. No downstream consumer of the old names exists yet.
Wraps ms2rescore CLI per engine. The prep script extracts per-engine SpecId parsing (MSFragger / Comet / Sage / PEAKS all format it differently) from core.py rescore_* L951-1117, merges mokapot target + decoy PSMs with the combined PIN's percolator features, and pulls per-scan precursor info from scans.pkl. Output TSV is the exact shape ms2rescore expects with prefixed rescoring: feature columns.
Merges rescored PSMs across engines at 1% PSM- and peptide-level FDR, separates chimeric spectra (scans assigned to >1 peptide) into their own audit file, and emits an integrated peptide table with per-run PSM counts. Mirrors core.py read_results L1451 and read_psms L1313 but is self-contained: takes engine_name=path pairs on the CLI so the Nextflow module does not need to know engine count at compile time. Protein info (gene, species, description) is parsed from the search FASTA headers assuming UniProt-style GN= / OS= fields.
Copied verbatim from immunopeptidomics/external_tools/{Comet,Sage,ms2rescore}/params/
so every {instrument}_{mod_type} combination the ms_search subworkflow
references already exists on disk. 11 files per engine covering
orbitrap/timsTOF × mod/nomod/TMT10/TMT16/mhcii/lowres.
Expands the subworkflow from Sprint 1's single-engine flow to the full
open-source search pipeline:
PREPARE_FASTA
→ MSFRAGGER → calibrated mzML + PIN
→ COMET + SAGE run in parallel on the calibrated mzMLs
→ MOKAPOT aliased per-engine (three independent FDR instances)
→ CONVERT_MZML fans out over mzMLs, grouped per sample for MGF + scans
→ MS2RESCORE aliased per-engine, rescores each mokapot output
→ INTEGRATE_ENGINES merges to a single unified peptides/PSMs table
Engine selection is gated on --ms_engines everywhere; when MSFragger is
not selected, Comet/Sage fall back to the raw mzML inputs and Sage skips
the calibrated-settings inheritance path. INTEGRATE receives
engine-name/TSV pairs via groupTuple so it does not need to know which
engines ran at compile time.
Same reference bundle + GATK resources as params_D122_liver.yaml; only the input samplesheet and outdir differ. Used for the ongoing dev/cryptic-port validation run on xy86.
Schema pattern now allows peaks alongside msfragger/comet/sage. Validator rejects a run that lists peaks without also supplying --peaks_psm_csv. Adds --peaks_min_match_fraction knob (default 0.98) that gates how many PEAKS rows must resolve to real scan numbers before conversion proceeds.
Converts a PEAKS Studio db.psms.csv export into a PIN that the existing MOKAPOT module consumes unchanged. PEAKS reports spectrum *indices*, so the script resolves them against one or more index2scan.pkl pickles from CONVERT_MZML and aborts if fewer than peaks_min_match_fraction of rows line up — that catches the common 'PEAKS searched a different mzML' mistake early. Extracted from core.py run_PEAKS L555.
Fourth engine path: CONVERT_PEAKS (gated on params.peaks_psm_csv) → MOKAPOT_PEAKS → MS2RESCORE_PEAKS → INTEGRATE_ENGINES. CONVERT_PEAKS consumes the index2scan pickles already emitted by CONVERT_MZML per sample, so no new scan parsing is needed. MS2RESCORE is aliased a fourth time; INTEGRATE picks PEAKS up automatically through the same groupTuple channel as the other engines.
Introduces --run_netmhcpan, --run_netmhciipan, --run_gibbscluster, --run_flashlfq, --run_blastp_host boolean gates alongside the user-supplied tool paths (--netmhcpan_path, --netmhciipan_path, --gibbscluster_path), --hla allele list, --blast_db prefix, and --host_species. Each downstream tool is individually gated so users only run what they're licensed or configured for. Schema adds a new immunoinformatics_options group referenced from allOf.
Wrap the academic-licensed netMHCpan-4.1 / netMHCIIpan-4.3 binaries supplied by the user. Each module filters the integrated peptides table to the relevant length range (8-12 for class I, 13-18 for class II), calls the binary, and pipes stdout through parse_netmhcpan.py which extracts the PEPLIST / Sequence result rows and keeps the best ranked allele per peptide. Extracted from core.py netMHCpan() L1802 and get_best_binder() L1780.
Runs the user-supplied GibbsCluster-2.0e_SA.pl on immunopeptides and picks the winning cluster count by the largest KLD sum across rows of gibbs.KLDvsClusters.tab. parse_gibbs.py then reads the matching res/gibbs.<N>g.out file and emits a peptide→cluster mapping. Class-I length sets (max < 13) get the -C / -D 4 / -I 1 flags as in core.py run_Gibbs() L2041.
Quantifies peptides across runs using bioconda's flashlfq=2.1.4 so no dotnet/user binary is needed. prepare_flashlfq_input.py melts the PSMs_run_* columns from integrated_peptides.tsv into FlashLFQ's long idt format and de-duplicates on (Full Sequence, charge, File Name). Match-between-runs is toggled on automatically when >1 MS file is present. Derived from core.py run_FlashLFQ() L1119 but fed from the unified integrated peptide table rather than per-engine mokapot outputs.
Filters the integrated peptides to non-host rows (species column does not contain --host_species and peptide sequence doesn't say 'contaminant'), writes a FASTA with I→L substitution, and runs blastp-short against the user-supplied --blast_db. Top hit per peptide is merged back into the peptide table as BLASTP_ident%, BLASTP_match, BLASTP_matchedSeq columns; 100% identity rows have the host species appended for consistent downstream filtering. Mirrors core.py run_BLASTP() L2181.
New IMMUNOINFORMATICS subworkflow gates the five downstream modules on individual --run_* flags so a user can run, say, only netMHCpan + Gibbs without touching BLAST or FlashLFQ. Errors early if a tool is requested without the licensed binary or the HLA list. The main workflow invokes it right after MS_SEARCH when the user has asked for at least one tool — otherwise it's skipped entirely and the ms_search step ends at the integrated peptide table like before.
Nextflow processes with conditional/optional inputs need a concrete file path to stage; zero-byte NO_FILE is the nf-core convention for signalling 'no file supplied' from a subworkflow to a module. Consumed first by IMMUNOINFORMATICS_REPORT in Sprint 6 — SAGE already referenced it but was tolerant of the missing file; this makes the sentinel explicit.
Self-contained per-sample HTML report with embedded PNG plots. Accepts six optional inputs (integrated peptides + netMHCpan/IIpan best-binder tables + Gibbs clusters + FlashLFQ quant + blastp-annotated peptides), each padded with assets/NO_FILE when the matching --run_* gate was off upstream. Sections degrade gracefully: missing tables simply omit the section rather than failing the run. Plot code is the essential subset of core.py histogram_plotter L1615, id_per_run L1718, netMHCpan_overview L1881, and gibbs_plot L2084; sequence logos use logomaker when available.
After every gated tool has had its chance to populate its output channel, fan them all into IMMUNOINFORMATICS_REPORT keyed on meta, padding any absent channel with assets/NO_FILE. The module itself tests each input for the NO_FILE name and skips the corresponding --flag to the python script, so users who only enable netMHCpan still get an HTML page.
Adds a dedicated 'MS search' section to docs/usage.md covering the samplesheet shape, engine selection, licensed-tool placement, and a worked invocation; lists the five optional immunoinformatics gates in a single table. README.md gets a matching paragraph pointing at the new step. conf/test_ms_search.config is a new stub-only profile that exercises the ms_search subworkflow wiring without requiring a real mzML or licensed MSFragger JAR — register it in nextflow.config alongside the existing test / test_full profiles.
sanjaysgk/ipg deliberately does not ship licensed binaries (MSFragger, netMHCpan, netMHCIIpan, GibbsCluster) or user-specific blast DBs. docs/external_tools.md spells out what bioconda already installs, what the user must supply, where to put it on M3, and a full params YAML example. bin/check_external_tools.sh is a pre-flight validator that reads the same params YAML and verifies each configured path is readable/executable (plus .pin/.phr/.psq siblings for blast DBs), so misconfigured runs fail instantly instead of two hours in.
feat(ms_search): add MS search + immunoinformatics steps (Sprints 2-7)
Two distinct fixes: 1. Vendored Comet/Sage/MS2Rescore param templates under assets/ms_search_params/ are upstream files copied verbatim from immunopeptidomics/external_tools/. Reformatting them would diverge from upstream and make future syncs painful — same logic as the existing /containers/ipg-tools/src/** carve-out. Add a matching editorconfig stanza to unset all rules under that path. 2. Three of my own .nf files had Nextflow continuation indents that weren't multiples of 4 (immunoinformatics_report tuple input, immunoinformatics subworkflow comment block, and python heredocs in blastp_host). Tightened to satisfy the linter without changing semantics.
ci: fix pre-commit editorconfig violations
Three pieces: 1. immunoinformatics_report/meta.yml had a missing space after 'netmhciipan:' that broke YAML compact-mapping syntax. 2. Add assets/ms_search_params/ to .prettierignore so prettier leaves the vendored Comet/Sage/MS2Rescore param templates alone — same rationale as the .editorconfig carve-out. 3. Apply prettier auto-format to all module meta.yml / environment.yml and the two new docs files (whitespace-only changes; no semantic diff).
ci: round 2 lint — prettier auto-format + ignore vendored params + yaml fix
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @sanjaysgk, your pull request is larger than the review limit of 150000 diff characters
|
…ists
DB_COMPARE_PHASE1 was being invoked with bare `[]` for the two
optional list inputs. Nextflow still stages a file in that case but
gives it a name that doesn't match the module's 'NO_FILE' guard, so
the phase2_args conditional fires and adds '-j -u ' to the R
command — db_compare_v2.R then fails optparse with:
error: flag "j" requires an argument
Swap to the existing assets/NO_FILE sentinel so the module correctly
skips those flags in phase 1. Surfaced by a real D122_Liver post_ms
run on SLURM; same bug would affect every phase-1 invocation.
fix(post_ms): NO_FILE sentinel for phase-1 empty list inputs
Phase 1 passes NO_FILE for both discard_list and unconventional_list.
Nextflow refuses to stage two input files with the same filename in
the same work dir — errors with:
input file name collision -- There are multiple input files for
each of the following file names: NO_FILE
Scope the two path inputs into separate staging subdirs (discard/ and
unconv/) so the collision is impossible regardless of what the caller
supplies. The downstream script contract is unchanged (both paths are
still only used when discard_list.name != 'NO_FILE'). Second fix in a
row for the D122_Liver post_ms run — first was the NO_FILE sentinel
itself (PR #7).
Two pieces: 1. .gitignore — carve tests/data/ out of the existing 'data/' rule with a negation so test fixtures can be committed while real run outputs stay ignored. 2. tests/data/post_ms/ — minimum-viable synthetic fixture exercising the --step post_ms subworkflow (Phase 1 → ORIGINS simple → Phase 2 → ORIGINS full). Hand-built peptide CSVs with 5 cryptic-only rows, 3 shared across DBs, and 2 below-threshold noise rows — guarantees every downstream module sees non-empty input. Runs in ~30s, no licensed tools. README.md has the launch command and expected outputs. Complements the chr22 db_construct bundle and the pending HepG2 ms_search fixture per project_test_suite_plan.md (Path A).
fix(db_compare): stageAs for NO_FILE collision + synthetic post_ms test fixture
VennDiagram::venn.diagram with cat.prompts=TRUE prompts the user during
layout calculation. In a non-interactive R session (any nf-core run),
adjust.venn returns NA for max/min, then the script crashes with:
Error in if (max.x - min.x >= max.y - min.y) {
missing value where TRUE/FALSE needed
Calls: venn.diagram -> <Anonymous> -> adjust.venn
cat.prompts=TRUE is debugging mode — must be FALSE for headless runs.
Bug surfaced by the synthetic post_ms test fixture; same call would
fail any post_ms run on real data once the column-case + sentinel
fixes land.
…O_FILE check Two related fixes for db_compare module: 1. Newer PEAKS exports use '-10LgP' (capital L), R's check.names converts to 'X.10LgP', dplyr::select(X.10lgP) misses → script crashes selecting columns. Pre-process headers with sed before R reads them. Operates on copies so staged inputs aren't mutated. 2. After PR #8 added stageAs: 'discard/*' to disambiguate the two NO_FILE inputs, the .name property started returning 'discard/NO_FILE' not 'NO_FILE' — breaking the phase-1/phase-2 gate. Switch to discard_list.size() > 0 (NO_FILE is zero bytes by design) so the check works regardless of stageAs path tricks. Both surfaced by the synthetic post_ms test fixture exercising the real D122_Liver path — every reported error now resolved end-to-end through DB_COMPARE_PHASE1.
Header has 20 fields, original data rows had 21 — extra empty field between Accession and Area columns. R read.csv aligned columns wrong (Mass values into X.10lgP slot, Length set to 0 for all rows), which emptied the post-filter peptide lists fed to venn.diagram and tripped the cat.prompts=TRUE bug. Drop one comma per row so PTM/AScore/Area align correctly.
fix(db_compare): three runtime bugs surfaced by post_ms test fixture
… env The Dockerfile at containers/ipg-tools/Dockerfile builds six C tools (curate_vcf, revert_headers, alt_liftover, triple_translate, squish, origins) for the singularity/docker container image. The pixi engine profile bypasses containers entirely, so these binaries need to exist in bin/ for any --profile pixi run to work. Other tools (squish, triple_translate, etc.) were already gitignored under bin/ but their build path wasn't documented anywhere — leaving new clones unable to run the pipeline in pixi mode without manual gcc invocations. This script: - Compiles all six tools into bin/ with the same flags as the Dockerfile - Gracefully overwrites pre-existing binaries - Is the single source of truth for the pixi build path Surfaced by the D122_Liver post_ms run hitting 'origins: command not found' after the singularity profile failed to pull the non-existent ghcr.io/sanjaysgk/ipg-tools:0.2.0 container. bin/origins added to .gitignore alongside the existing tool entries.
build: add bin/build_ipg_tools.sh — compiles origins + 5 kescull tools for pixi env
Same shape as params_D122_liver.yaml; only input samplesheet differs. Paired with launcher at /fs04/scratch2/xy86/sanjay/ipg-runs/D122_lung_full/run_db_construct.sh (not in repo — launcher lives alongside the run output tree).
assets: add D122_lung samplesheet and params
sanjaysgk
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First production release of
sanjaysgk/ipg. Covers the base IPG functionality — end-to-end RNA-seq → cryptic peptide FASTA (--step db_construct) plus the 2-phasedb_compare+originsanalysis after a MS search (--step post_ms).Validation status
db_constructpost_msms_searchimmunoinformaticsKnown issues (pre-existing, not regressions)
After merge
Test plan