You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Initial analysis to improve the memory consumption for federal restrictions processing.
TL;DR;
Changed the federal-restrictions-integration to stream the data from STFP, while unzipping it and saving directly into sims.federal_restrictions.
Main method for this POC: SFTPIntegrationBase.streamResponseFileLines.
Local Tests
Changed queue-consumers locally to execute only federal-restrictions-integration queue.
Executed the current process and measured until the moment the file was downloaded, unzipped, and parsed (no records inserted).
Adapted a new process as follows.
Changed the method reading the file from the SFTP to create a readable stream.
Changed the zip lib to another with stream capabilities. Did not invest much time looking for alternatives, just used the first one that I found for now.
Once a line is read, allow a callback to be invoked, allowing "a code" to process the read line.
Created a callback to insert the parsed record directly into the table, delaying some processes, for instance, the creation of missing restriction codes.
Current PROD Process (downloaded, unzipped, and parsed with no records inserted)
New proposed process (downloaded, unzipped, parsed, and inserted)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial analysis to improve the memory consumption for federal restrictions processing.
TL;DR;
federal-restrictions-integrationto stream the data from STFP, while unzipping it and saving directly intosims.federal_restrictions.SFTPIntegrationBase.streamResponseFileLines.Local Tests
federal-restrictions-integrationqueue.Current PROD Process (downloaded, unzipped, and parsed with no records inserted)
New proposed process (downloaded, unzipped, parsed, and inserted)