Docs on databases

[jen-reeve]

Current /opt/nesi/db:

-rw-rw-r--  1 nesi-apps-admin nesi-apps-admin  457 Jun 16  2025 2_dram_preparedatabases.slurm
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Dec  1  2024 alphafold_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Oct 29 19:14 blast
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep 10  2021 cartopy
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 13  2020 centrifuge
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 30  2023 CheckM2_DB
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jul 21  2019 CheckM_DB
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 May  7  2020 checkv-db-v0.6
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Feb 20  2021 dammit_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Oct 17  2022 DAS_DB
drwxr-xr-x+ 1 nesi-apps-admin nesi-apps-admin    0 Nov 12 07:36 dfam_3.9
-rw-rw-r--  1 nesi-apps-admin nesi-apps-admin 2400 Apr 18  2022 down_pdb_mmcif.sh
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep 27  2022 DRAM_1.3.5
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Oct 17  2023 eggnog_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jun 16  2024 FCS-GX
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jul 19  2021 gtdbtk_202
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 May  9  2022 gtdbtk_207_v2
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Apr 27  2023 gtdbtk_214
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Apr 10  2024 gtdbtk_220
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar 13  2022 Humann
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar  5  2022 Kaiju
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar 31  2025 Kraken2
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Feb 26  2021 megaX
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Feb 21  2025 nesi-apps-admin
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 27  2021 nullarbor_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov  7  2023 Pfam
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar  4  2022 PhyloPhlAn
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jul 15  2019 prokka
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 18  2023 ProteinDataBank
dr-xr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 24  2020 RQCFilterData
drwxrwxr-x+ 1 nesi-apps-admin nesi-apps-admin    0 Feb  4  2021 sortmerna_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep  8 12:17 SqueezeMeta
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar 16  2020 StrVCTVRE
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Mar 29  2023 TEST
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Feb 26  2021 Trinotate
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep 22 08:43 Uniprot
-rw-------  1 nesi-apps-admin nesi-apps-admin  379 Jul 19  2021 UniRef30_2021_06.md5sums
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jul  5  2021 VariantEffectPredictor
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov  2  2020 VIBRANT_v1.2.1_databases
drwxrwxr-x  1 nesi-apps-admin apps-team          0 Feb 10  2021 VirSorter
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jul 27  2022 waafle

[CallumWalley]

Sorted by atime

-rw-------  1 nesi-apps-admin nesi-apps-admin  379 Feb 24  2025 UniRef30_2021_06.md5sums
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:02 nullarbor_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:02 CheckM2_DB
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:02 ProteinDataBank
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:02 gtdbtk_207_v2
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:03 checkv-db-v0.6
drwxrwxr-x+ 1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:03 sortmerna_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:03 DAS_DB
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:03 CheckM_DB
drwxrwxr-x  1 nesi-apps-admin apps-team          0 Aug 29 15:03 VirSorter
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:03 gtdbtk_214
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:07 gtdbtk_202
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:08 waafle
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:08 Pfam
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:08 nesi-apps-admin
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 StrVCTVRE
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 PhyloPhlAn
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 TEST
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 VIBRANT_v1.2.1_databases
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 FCS-GX
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Aug 29 15:09 megaX
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep  9 14:32 SqueezeMeta
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep 18 10:15 gtdbtk_220
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Sep 19 15:21 prokka
-rw-rw-r--  1 nesi-apps-admin nesi-apps-admin 2400 Oct 29 07:38 down_pdb_mmcif.sh
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 26 20:05 alphafold_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:46 Humann
dr-xr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:47 RQCFilterData
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:47 dammit_db
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:55 VariantEffectPredictor
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:56 Trinotate
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:56 DRAM_1.3.5
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Nov 28 09:56 blast
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Dec  5 13:40 Kaiju
drwxr-xr-x+ 1 nesi-apps-admin nesi-apps-admin    0 Dec  5 13:45 dfam_3.9
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Dec  5 13:45 cartopy
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Dec  5 13:45 centrifuge
-rw-rw-r--  1 nesi-apps-admin nesi-apps-admin  457 Jan  7 14:09 2_dram_preparedatabases.slurm
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 13 09:20 eggnog_db
drwxr-xr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 14 10:14 Uniprot
drwxrwxr-x  1 nesi-apps-admin nesi-apps-admin    0 Jan 14 10:30 Kraken2

[CallumWalley]

Aug 29 was migration date.

could start with a table of all the directories (excluding those not used since migration), sorted alphabetically, with short description and time of last update (mtime)?

[CallumWalley]

printf "|Name | Note | Updated |\n|--|--|--|\n"  && find /opt/nesi/db/ -maxdepth 1 -type d -newerat "2025-08-30" -printf '| `/opt/nesi/db/%f` | note | %TF |\n' | sort

Gives

Name	Note	Updated
/opt/nesi/db/alphafold_db	note	2024-12-01
/opt/nesi/db/blast	note	2025-10-29
/opt/nesi/db/cartopy	note	2021-09-10
/opt/nesi/db/centrifuge	note	2020-01-13
/opt/nesi/db/dammit_db	note	2021-02-20
/opt/nesi/db/db/	note	2025-11-12
/opt/nesi/db/dfam_3.9	note	2025-11-12
/opt/nesi/db/DRAM_1.3.5	note	2022-09-27
/opt/nesi/db/eggnog_db	note	2023-10-17
/opt/nesi/db/gtdbtk_220	note	2024-04-10
/opt/nesi/db/Humann	note	2022-03-13
/opt/nesi/db/Kaiju	note	2022-03-05
/opt/nesi/db/Kraken2	note	2025-03-31
/opt/nesi/db/prokka	note	2019-07-15
/opt/nesi/db/RQCFilterData	note	2020-01-24
/opt/nesi/db/SqueezeMeta	note	2025-09-08
/opt/nesi/db/Trinotate	note	2021-02-26
/opt/nesi/db/Uniprot	note	2025-09-22
/opt/nesi/db/VariantEffectPredictor	note	2021-07-05

[CallumWalley]

Dataset	Path	Likely Official Source	Licence Status (Approx.)	Notes
alphafold_db	/opt/nesi/db/alphafold_db	https://alphafold.ebi.ac.uk/	Free academic use; EMBL-EBI terms	Predicted protein structures generated by AlphaFold
blast	/opt/nesi/db/blast	https://blast.ncbi.nlm.nih.gov/	Public	NCBI nucleotide and protein databases
cartopy	/opt/nesi/db/cartopy	https://scitools.org.uk/cartopy/	LGPL-3.0	Python geospatial library (not a biological dataset)
centrifuge	/opt/nesi/db/centrifuge	https://ccb.jhu.edu/software/centrifuge/	GPL-3.0	Indexed genome databases for classification
CheckM2_DB	/opt/nesi/db/CheckM2_DB	https://github.com/chklovski/CheckM2	MIT	Marker gene database for genome quality estimation
CheckM_DB	/opt/nesi/db/CheckM_DB	https://ecogenomics.github.io/CheckM/	GPL-3.0	Original CheckM marker gene database
DAS_DB	/opt/nesi/db/DAS_DB	Varies (DAS providers)	Public / mixed	Distributed Annotation System data sources
dammit_db	/opt/nesi/db/dammit_db	https://camillescott.github.io/dammit/	Public / mixed	Aggregated annotation databases used by dammit
dfam_3.9	/opt/nesi/db/dfam_3.9	https://dfam.org/	CC-BY 4.0	Transposable element profile HMM database
DRAM_1.3.5	/opt/nesi/db/DRAM_1.3.5	https://github.com/WrightonLabCSU/DRAM	Public	Metabolic annotation databases
eggnog_db	/opt/nesi/db/eggnog_db	https://eggnogdb.embl.de/	CC-BY 4.0	Orthologous group and functional annotation DB
FCS-GX	/opt/nesi/db/FCS-GX	Likely NCBI-related	Public	Genome contamination screening datasets
gtdbtk_202	/opt/nesi/db/gtdbtk_202	https://gtdb.ecogenomic.org/	CC-BY-SA 4.0 (data); GPL-3.0 (software)	Early GTDB-Tk compatible release
gtdbtk_207_v2	/opt/nesi/db/gtdbtk_207_v2	https://gtdb.ecogenomic.org/	CC-BY-SA 4.0 (data); GPL-3.0 (software)	Older GTDB release
gtdbtk_214	/opt/nesi/db/gtdbtk_214	https://gtdb.ecogenomic.org/	CC-BY-SA 4.0 (data); GPL-3.0 (software)	Intermediate GTDB release
gtdbtk_220	/opt/nesi/db/gtdbtk_220	https://gtdb.ecogenomic.org/	CC-BY-SA 4.0 (data); GPL-3.0 (software)	Genome Taxonomy Database release used by GTDB-Tk
Humann	/opt/nesi/db/Humann	https://huttenhower.sph.harvard.edu/humann/	Apache-2.0	Functional profiling databases for metagenomics
Kaiju	/opt/nesi/db/Kaiju	https://bioinformatics-centre.github.io/kaiju/	GPL-3.0	Protein-level taxonomic classification database
Kraken2	/opt/nesi/db/Kraken2	https://ccb.jhu.edu/software/kraken2/	MIT	K-mer based taxonomic classification databases
megaX	/opt/nesi/db/megaX	https://www.megasoftware.net/	Free for academics	Evolutionary analysis reference data
nullarbor_db	/opt/nesi/db/nullarbor_db	https://github.com/tseemann/nullarbor	Public	Reference databases used by the Nullarbor pipeline
Pfam	/opt/nesi/db/Pfam	https://pfam.xfam.org/	CC-BY 4.0	Protein family HMM database
PhyloPhlAn	/opt/nesi/db/PhyloPhlAn	https://github.com/biobakery/phylophlan	BSD-style	Reference genomes and phylogenetic markers
prokka	/opt/nesi/db/prokka	https://github.com/tseemann/prokka	GPL-3.0	Curated bacterial annotation databases
ProteinDataBank	/opt/nesi/db/ProteinDataBank	https://www.rcsb.org/	Open	3D structures of proteins and nucleic acids
RQCFilterData	/opt/nesi/db/RQCFilterData	Likely bundled with RQCFilter	Open / unspecified	Reference data for read quality control
sortmerna_db	/opt/nesi/db/sortmerna_db	https://github.com/biocore/sortmerna	Public	rRNA reference databases
SqueezeMeta	/opt/nesi/db/SqueezeMeta	https://github.com/jtamames/SqueezeMeta	Public	Integrated metagenomic annotation databases
StrVCTVRE	/opt/nesi/db/StrVCTVRE	https://github.com/krassowski/StrVCTVRE	Public	Training and reference data for SV pathogenicity
Trinotate	/opt/nesi/db/Trinotate	https://github.com/Trinotate/Trinotate	Public	Annotation databases for transcriptomes
Uniprot	/opt/nesi/db/Uniprot	https://www.uniprot.org/	CC-BY 4.0	Protein sequence and functional annotation database
VariantEffectPredictor	/opt/nesi/db/VariantEffectPredictor	https://www.ensembl.org/info/docs/tools/vep/	Apache-2.0	Ensembl annotation data for variant effect prediction
VIBRANT_v1.2.1_databases	/opt/nesi/db/VIBRANT_v1.2.1_databases	https://github.com/AnantharamanLab/VIBRANT	Public	Viral genome and HMM databases
VirSorter	/opt/nesi/db/VirSorter	https://github.com/simroux/VirSorter	Public	Viral hallmark gene and profile databases
waafle	/opt/nesi/db/waafle	https://github.com/biobakery/waafle	Public	Reference sets for gene neighborhood analysis
checkv-db-v0.6	/opt/nesi/db/checkv-db-v0.6	https://bitbucket.org/berkeleylab/checkv	Public	Viral genome completeness and contamination DB