Fix: IT may mirror huge files (#7767)#7773
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7773 +/- ##
========================================
Coverage 85.07% 85.07%
========================================
Files 157 157
Lines 23293 23293
========================================
Hits 19816 19816
Misses 3477 3477 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
achave11-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
from
3a5650d to
135223b
Compare
achave11-ucsc
left a comment
achave11-ucsc
left a comment
There was a problem hiding this comment.
Slight indentation issue (see diff below) and consider consolidating the first commit into the second one, if removing the cache is part of the fix. Or alternatively add a slight description to the body of the first commit as to why the removal may be necessary.
Index: test/integration_test.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/test/integration_test.py b/test/integration_test.py
--- a/test/integration_test.py (revision 135223b9e03da58228c9d3ee5bcfac038b225ed4)
+++ b/test/integration_test.py (date 1770875616184)
@@ -799,9 +799,9 @@
for filters in [self._fastq_filter(catalog), {}]:
if max_size is not None:
filters.update({
- file_size_facet: {
- 'within': [[1, max_size]],
- }
+ file_size_facet: {
+ 'within': [[1, max_size]],
+ }
})
response = self._check_endpoint(method=GET,
path='/index/files',
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
2 times, most recently
from
dbb8ed9 to
1a04108
Compare
hannes-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
from
1a04108 to
71dd9d4
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
Please move my refactoring commits so that they precede the main commit. Use my commit titles (minus the REVIEW prefix) but don't preserve authorship.
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
3 times, most recently
from
997a174 to
6718a81
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
As requested in my previous review, I shouldn't be listed as the author of any commits. Reordering the commits, resolving the conflicts, and fixing any bugs constitutes a significant enough contribution to warrant a change in authorship.
hannes-ucsc
added
1 review
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
from
6718a81 to
dfbb85a
Compare
|
I observed an IT failure with these changes due there being no files in the source within the requested size range. Apparently, there are sources in |
|
Assignee to investigate the number of files and amount of data that was mirrored during integration tests in all seven shared deployments. |
|
There are three public sources in I tested this on a personal deployment with a common prefix set, which should reflect the real status of the index during IT in |
|
Summary of IT across the past 7 weeks:
This doesn't detect duplicate files, so the "average total file size" figure is not reliable for Example query for the first three columns: Example query for counting the number of IT runs. We mirror each source twice to test idempotence, so seeing a source only once probably means a run failed. If IT was run twice with the same source and failed both times, the count will be off. |
|
@hannes-ucsc: "Assignee to move forward with the fix without increasing the mirror limit for the IT catalogs. This will increase the probability of IT failures in case the filter used to find a file for the individual MirrorFileAction does not yield any results. We will reevaluate the need for fixing that later when we know the impact that has." |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
from
dfbb85a to
174ba6f
Compare
Security design review
|
dsotirho-ucsc
force-pushed
the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
from
174ba6f to
5d6d1bf
Compare
dsotirho-ucsc
added
the
sandbox
dsotirho-ucsc
deleted the
issues/nadove-ucsc/7767-it-may-mirror-huge-files
branch
5 participants
Linked issues: #7767
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all linked issuespartiallabelAuthor (reindex)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAuthor (API changes)
APIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues linked to this PRAuthor (before every review)
develop, squashed fixups from prior reviewsmake requirements_updateor this PR does not modifyDockerfile,environment,requirements*.txt,common.mk,Makefileorenvironment.bootRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
Note that after requesting changes, the PR must be assigned to only the author.
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator
reindex:…labels andrcommit title tagno demodevelopOperator (deploy
.sharedand.gitlabcomponents)_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator (post-deploy of
.gitlabcomponent)dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (deploy runner image)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerOperator (sandbox build)
sandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices insandboxanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvilboxsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevOperator (merge the branch)
pif the PR is also labeledpartialOperator (main build)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator (mirroring)
devor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem