This repository was archived by the owner on Aug 21, 2025. It is now read-only.
Develop - test run for deployment#249
Open
KutnerBroadie wants to merge 58 commits into
Open
Conversation
…system CD-644 fix unique terms duplicate results
* update pr_push with sherlock * change name * update current action to notify sherlock and include push to develop, test and main
ah action to deploy dev
* update sql and tests to use alias and integer_id_alias
* fix tests after modifying the rest of mapping tables to use integer aliases * add schema with data source tables
* separate column defintions by entity vs mapping table * change conditional for mapping table and filter out alias ids * add lots of checks for associated project to look like it is on entity tables
* fix system param in unique terms call * fix test
* Implements parsing based includeCount query optimizer. Currently there is a bug that doesn't allow filters on the same table as the endpoint. * Initial Commit Implements parsing based includeCount query optimizer. Currently there is a bug that doesn't allow filters on the same table as the endpoint. * Fixed Entity Filter Bug Added the ability to handle filters involving the endpoint entity table. * Update QueryService Broke out parenthesis substring code into a function. Cleaned up a few things and added distinction between includeCountQuery and countEndpointQuery. * Moved Filter class to own file. * Added new test file. * Updated queryapi controller test. * Wrapped Filter class in try-catch blocks. * Fixed misspelling. * Adding Count Endpoint Code Broke out Filter into it's own file class. Adding code to enable count endpoint optimization. * Successful Merge of Branches Merged two branches to include new count endpoint optimization code with exception handling + test code * Consolidated initialization constructor Worked out method to add a single initialization function to Filter class. * Added new fn and accompanying unit test to trim extraneous parens from filter queries that are compound. * Changed hard-coded use of and, or and where to use swagger generated values. * Disabled half-written test so whole file can be run. * Provided optimized version of parenthesisSubstring to build string all at once. * Finished Optimized Count Endpoint Queries Added full functionality for producing optimized count endpoint query code * Column Name fix Fixed misspelling of integer_id_alias * Resolved Query Issue Resolved construction of preselect query and added json casting * Updated mutation count column names see description * Fixup for parenthesisSubString in cases where WHERE clause doesn't have parens. * Doing runtime check of entity getprimarykeys to ensure we aren't doing an out-of-bounds access on an empty list. * Ensuring that our entity table has a PK for filter usage. * Allowing coalesce statement to start without a paren. * Added sample coalesce statement. * isRoot Argument Removed and Added Count Tests Created ChildFilter class solely to eliminated need for isRoot argument. Added several tests for each entity/count endpoint. * Added Optimized Query to Results Added the code required to pass back the optimized query for count endpoint queries * Refined Filter Class Rewrote filter initialization code to eliminate need for isRoot and id arguments while maintaining the inability to produce a root filter with isRoot variable set to false. * Filtering Streamed Nulls Filtering out nulls in streams for totalFields to count and groupedCountFields * remove logic to handle subquery * Moved paren string processing to a FilterUtils class, updated tests to work with it. * Fixed Count Optimization for Simple Filters Fixed template to properly handle constructing optimized query when only a simple filter is applied * Refactored Count Optimization to Utilize Wildcard Refactored code to use count(*) where appropriate. * Relationship field as backup to Primary Key Adding ability to join on relationship fields if table has no primary key * use count(*) in count queries * split somatic_mutation columns into internal and external fields * Refactored Count Endpoint Query Creation Added method to add distinct counts when not querying somatic_mutations endpoint otherwise use count(*) * Cleanup Cleaning up commented code * Somatic Mutation Work Arounds And Common Alias For Mapping Added several work arounds to handle somatic_mutation table count queries. Also added utilizing the commonAlias in place of mappingEntityKeys which were generated from join paths. * Adding TODOs Added TODOs for MVP * Updated Schema and Added File Count Coverage Updated the schema to reflect database inclusion of the "subject_alias" column in the somatic mutations table. Also added coverage for getting a file count on count endpoint queries when the path between the entity table to the files table is equal to 1. * Assumption for Common Alias and Cleaned Up cda_subject_alias References Added code to assume the commonAlias variable. Also cleaned up cda_subject_alias references to now use the "subject_alias" column. * Added Checks to Mapping and Join Paths for Common Alias Presence Added checks when building Join paths or just a simple mapping table to ensure the commonAlias exists in those tables * Schema Modification and More somatic_mutation Handling Removed foreign keys for cda_subject_* columns as they were sometimes getting chosen over "subject_alias". Also added handling on join statements involving somatic mutation. * Replaced mappingEntityKey with commonAlias See subject * Updated groupedFieldsToCount for Mutations Updated groupedFieldsToCount to utilize better columns. * Fixed SQL Syntax Error Added parenthesis around UNIONINTESECT for count endpoint creation which lacked coverage for simple filters that had a mapping table --------- Co-authored-by: tanner-coon-bh <tanner.coon@bluehalo.com> Co-authored-by: Andrea Haessly <ahaessly@broadinstitute.org>
fixing two minor slightly fragile syntax issues in build.gradle
* Fixed integer_id_alias Assumption Fixed issues that where affecting filters on *_data_source tables * Added Fixes for *_data_source Tables Added edge case handling for filters from *_data_source tables
Added paged query preselect optimization for files table. Note: currently hard coded for subjects table due to temporary issues with files table
Updated paged query optimizer to utilize file table instead of subject. Also added the ability to pass back the optimized query with the results.
* Added Optimization For File Paged Query Preselects Added paged query preselect optimization for files table. Note: currently hard coded for subjects table due to temporary issues with files table * Updated To Utilize Files table Updated paged query optimizer to utilize file table instead of subject. Also added the ability to pass back the optimized query with the results. * Fixed *_associated_project Table Filters & Simplified file Table Joins Treated *_associated_project tables like *_data_source tables as there is no mapping table between them and their respective entity tables. Also, added code to update the join statement and file preselect filter to only use the mapping table for paged queries * Updated File Join Optimization for All Entity Tables see description
see description
Fixed column counted to be the common alias, not the filter table key.
Fixed bug with query generation for simple queries that have a mapping table with the common alias
* Removed bulk-data endpoint * Disabled boolean-query endpoint * Update to set no caching in header response from all of our endpoints. Adjusted to accommodate pen testing results. * Missed the status check
* Removed bulk-data endpoint * Disabled boolean-query endpoint
Fixed bug not properly utilizing AND operations with a lone file filter in the rightFilter
* Fixed Incorrect Paged Queries Fixed incorrect paged queries by utilizing preselect building from the Filter class. * Fixed Join Regex Added proper whitespace check to regex split for getting joins * Fixed Regex Made sure to add '+' for whitespace to regex on join split
Rebuilt logic for dropping unnecessary joins. I now build a join path to get all required tables to join on any tables found in the select clause. Then I only remove any join statements that don't include any of these tables.
* Don't deploy to dev in PRs * release branch is develop
* Updated For New Mutation Table Updated Schemas to reflect changes to data model. Updated MutationSqlGenerator* files to reflect changes to mutation table. Removed work-around code for quirks with somatic_mutation from Filter code. * Updated Mutation Count Endpoint Added "one_consequence" column to the summarized mutation count endpoint. * Updated Mutation Default Order By Changed mutation default order by from case_barcode to integer_id_alias * Updated expected query for CountSqlGeneratorTest Updated expected test result to match the new optimized count query * Bypassing CountSqlGeneratorTest Auto passing the test because it currently isn't written to test against the optimized query
Simple fix to resolve performance issues caused by ordering by non-primarykey column
* BigInt hotfix Fixed issue with bigint column filters processed as text which lead to PostgreSQL errors. * Update tests.yml Updating jacaco due to deprecated issues
* Fixed bug with readable query Found bug in readable query code that doesn't replace the parameters correctly when there are more than 10 present. * Updating SnakeYAML Updating SnakeYAML to non-vulnerable version 2.0. * Upgrading github action upload-artifact
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Testing a deployment