withUninomialFuzzyMatch crushes gnverifier

[dimus]

reported in #142

[smith558]

Thanks for looking into this. Very eager to see this fixed as uninomial fuzzy matching is crucial for some of our workflows.

[dimus]

gnames/gnmatcher#64

[dimus]

@smith558 this works for me now where api.json is your query example.

curl -X POST https://verifier.globalnames.org/api/v1/verify \
  -H "Content-Type: application/json" \
  -d @api.json

{
    "nameStrings": [
        "Mus musculus",
"Campylobacter jejuni subsp. jejuni NCTC 11168 = ATCC 700819",
"Faecalibacterium prausnitzii",
"Xylanibacter ruminicola",
"Butyrivibrio fibrisolvens DSM 3071",
"Anaerobutyricum hallii DSM 3353",
"Roseburia intestinalis L1-82",
"Megasphaera elsdenii DSM 20460",
"Marvinbryantia formatexigens DSM 14469",
"Drosophila melanogaster",
"Drosophila subobscura",
"Haloferax volcanii",
"Entamoeba histolytica HM-1:IMSS",
"Sus scrofa domesticus",
"Opisthorchiidae",
"Deinagkistrodon acutus",
"Rattus norvegicus",
"Candida maltosa Xu316",
"Bos taurus",
"Severe acute respiratory syndrome coronavirus 2",
"Influenza A virus",
"Respiratory syncytial virus",
"H5N1 subtype",
"Homo sapiens",
"Influenza B virus",
"Gallus gallus",
"Mycobacterium tuberculosis H37Rv",
"Dictyostelium",
"Guyparkeria hydrothermalis",
"Escherichia coli",
"Toxoplasma gondii GT1",
"Bombyx mori",
"Caenorhabditis elegans",
"Plasmodium falciparum Tanzania",
"Aeromonas hydrophila subsp. hydrophila ATCC 7966",
"Olea europaea",
"Pseudomonas fluorescens",
"Pseudomonas putida KT2440",
"Penaeus vannamei nodavirus",
"Pelagomonas calceolata",
"Mycobacterium smegmatis MKD8",
"biofilm metagenome",
"aquatic metagenome",
"Saccharomyces cerevisiae",
"Secale cereale",
"Hordeum vulgare",
"Avena sativa",
"Triticum aestivum",
"Pseudomonas aeruginosa AES-1R",
"Bacteria",
"Bacteroides thetaiotaomicron",
"Bacteroides acidifaciens",
"Eucalyptus grandis",
"Bacillus subtilis subsp. subtilis str. 168",
"Cupriavidus necator (strain ATCC 17699 / H16 / DSM 428 / Stanier 337)",
"Cryphonectria parasitica",
"Eriocheir sinensis",
"Spongospora subterranea",
"Arabidopsis thaliana",
"Bothrops mattogrossensis",
"environmental samples <bacteria",
"Anabaena sp. BTA17",
"Cricetulus griseus (Chinese hamster)",
"Porphyromonas gingivalis ATCC 33277",
"Magnetospirillum",
"Dechloromonas sp. JD15",
"Azonexus",
"Oryza sativa",
"Pseudoramibacter alactolyticus",
"Anaerostipes caccae",
"Thermus brevis",
"Saccharomycetales",
"Schizosaccharomyces pombe",
"Staphylococcus aureus",
"Pythium oligandrum",
"Equus caballus",
"Bacillus cereus AH187",
"Plasmodium",
"Plasmodium falciparum",
"Cancer borealis",
"Chlamydomonas reinhardtii",
"Acyrthosiphon pisum",
"Candida glabrata (Yeast)",
"Asparagopsis taxiformis",
"Nicotiana tabacum",
"Meloidogyne incognita",
"Solanum lycopersicum",
"Coffea arabica",
"Trionyx triunguis",
"human",
"Human coronavirus 229E",
"environmental samples <Bacteria>",
"Trypanosoma brucei",
"Acanthamoeba genotype T4",
"Phytophthora infestans T30-4",
"Anisakis simplex",
"Influenza A virus (A/Philippines/TMC10-144/2010(H1N1))",
"Anemia",
"Dehalococcoides mccartyi",
"Hathewaya histolytica"
    ],
    "withRelaxedFuzzyMatch": true,
    "withUninomialFuzzyMatch": true,
    "withCapitalization": true,
    "dataSources": [11]
}

[smith558]

@smith558 this works for me now where api.json is your query example.

curl -X POST https://verifier.globalnames.org/api/v1/verify
-H "Content-Type: application/json"
-d @api.json
{
"nameStrings": [
"Mus musculus",
"Campylobacter jejuni subsp. jejuni NCTC 11168 = ATCC 700819",
"Faecalibacterium prausnitzii",
"Xylanibacter ruminicola",
"Butyrivibrio fibrisolvens DSM 3071",
"Anaerobutyricum hallii DSM 3353",
"Roseburia intestinalis L1-82",
"Megasphaera elsdenii DSM 20460",
"Marvinbryantia formatexigens DSM 14469",
"Drosophila melanogaster",
"Drosophila subobscura",
"Haloferax volcanii",
"Entamoeba histolytica HM-1:IMSS",
"Sus scrofa domesticus",
"Opisthorchiidae",
"Deinagkistrodon acutus",
"Rattus norvegicus",
"Candida maltosa Xu316",
"Bos taurus",
"Severe acute respiratory syndrome coronavirus 2",
"Influenza A virus",
"Respiratory syncytial virus",
"H5N1 subtype",
"Homo sapiens",
"Influenza B virus",
"Gallus gallus",
"Mycobacterium tuberculosis H37Rv",
"Dictyostelium",
"Guyparkeria hydrothermalis",
"Escherichia coli",
"Toxoplasma gondii GT1",
"Bombyx mori",
"Caenorhabditis elegans",
"Plasmodium falciparum Tanzania",
"Aeromonas hydrophila subsp. hydrophila ATCC 7966",
"Olea europaea",
"Pseudomonas fluorescens",
"Pseudomonas putida KT2440",
"Penaeus vannamei nodavirus",
"Pelagomonas calceolata",
"Mycobacterium smegmatis MKD8",
"biofilm metagenome",
"aquatic metagenome",
"Saccharomyces cerevisiae",
"Secale cereale",
"Hordeum vulgare",
"Avena sativa",
"Triticum aestivum",
"Pseudomonas aeruginosa AES-1R",
"Bacteria",
"Bacteroides thetaiotaomicron",
"Bacteroides acidifaciens",
"Eucalyptus grandis",
"Bacillus subtilis subsp. subtilis str. 168",
"Cupriavidus necator (strain ATCC 17699 / H16 / DSM 428 / Stanier 337)",
"Cryphonectria parasitica",
"Eriocheir sinensis",
"Spongospora subterranea",
"Arabidopsis thaliana",
"Bothrops mattogrossensis",
"environmental samples <bacteria",
"Anabaena sp. BTA17",
"Cricetulus griseus (Chinese hamster)",
"Porphyromonas gingivalis ATCC 33277",
"Magnetospirillum",
"Dechloromonas sp. JD15",
"Azonexus",
"Oryza sativa",
"Pseudoramibacter alactolyticus",
"Anaerostipes caccae",
"Thermus brevis",
"Saccharomycetales",
"Schizosaccharomyces pombe",
"Staphylococcus aureus",
"Pythium oligandrum",
"Equus caballus",
"Bacillus cereus AH187",
"Plasmodium",
"Plasmodium falciparum",
"Cancer borealis",
"Chlamydomonas reinhardtii",
"Acyrthosiphon pisum",
"Candida glabrata (Yeast)",
"Asparagopsis taxiformis",
"Nicotiana tabacum",
"Meloidogyne incognita",
"Solanum lycopersicum",
"Coffea arabica",
"Trionyx triunguis",
"human",
"Human coronavirus 229E",
"environmental samples ",
"Trypanosoma brucei",
"Acanthamoeba genotype T4",
"Phytophthora infestans T30-4",
"Anisakis simplex",
"Influenza A virus (A/Philippines/TMC10-144/2010(H1N1))",
"Anemia",
"Dehalococcoides mccartyi",
"Hathewaya histolytica"
],
"withRelaxedFuzzyMatch": true,
"withUninomialFuzzyMatch": true,
"withCapitalization": true,
"dataSources": [11]
}

Testing now. At the time of the reporting of the issue, I had tested it extensively, with many payloads, and I was able to replicate it very cleanly. Each payload with dataSources crashed; each payload with preferredSources did not.

I am now seeing the crashes again. But it took about 20 requests. Did not start right away.

@dimus, the 2 requests in the screenshots were sent only 5s apart.

[dimus]

hm, can you try to remove withRelaxedFuzzyMatch and see if there is a difference?

[dimus]

@smith558 I am not able to reproduce the failing you saw.
Running this code with the same input did succed through 100 and 1000 runs with 1s delay between runs or without a delay.

#!/bin/bash

MAX_TRIES=100
DELAY=1
SUCCESS_COUNT=0

for i in $(seq 1 $MAX_TRIES); do
	RESPONSE=$(
		curl -s -w "%{http_code}" -X POST https://verifier.globalnames.org/api/v1/verify \
			-H "Content-Type: application/json" \
			-d @api.json \
			-o response.json
	)

	if [ "$RESPONSE" -eq 200 ]; then
		SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
		echo "$SUCCESS_COUNT"
	else
		echo "Request failed after $SUCCESS_COUNT successful attempts (HTTP $RESPONSE)"
		exit 1
	fi

	if [ $i -lt $MAX_TRIES ]; then
		sleep $DELAY
	fi
done

exit 0

[dimus]

@smith558 I suspect I found a bug that caused API to crash, can you try the sequence of events that did work for you to create the problem you described?

[smith558]

@smith558 I suspect I found a bug that caused API to crash, can you try the sequence of events that did work for you to create the problem you described?

Hello. Yes, will give it a try.