Merge develop.jpeg2000 with the latest develop#89
Open
nalinigans wants to merge 35 commits into
Open
Conversation
…ing a cached instance as there are multithreading issues that need to be figured out (#52)
* Add hdfs multithreading unit tests * Change create_workspace to adhere to the contract in initialize_workspace for consistency
…ssion (#54) Add array_schema tests before changing code to support compression for offsets. Also, fix broken GCS builds - Move to HDFS 2.9.2 for compatibility with latest GCS Connector Replace GCS deprecated properties.
…the compression of the attributes themselves. (#55) * Support compression separately from the attribute for the offsets to attributes with variable number of cells * Factor in the search tile used when reading * Clean up handling of offsets in array schema * Cleanup test_sparse_array_api as the compatibility test is at the test_array_schema_api level * Address review comments * Address review comments
…LE_HANDLES_OPEN is set. Also, set up benchmark harness. (#56) * Allow for file handles to also stay open until close() when TILEDB_DISABLE_FILE_LOCKING=1 * Add benchmarking harness and support TILEDB_KEEP_FILE_HANDLES_OPEN env instead of piggy backing this option through TILEDB_DISABLE_FILE_LOCKS * Cleanup compiler warnings
…ing. (#57) * Support Delta Encoding as a precompression filter
* Adding test utility to print out TileDB array schema * gcc requires string.h include for memset
…ry (#59) * Add lz4 and bitshuffle from the kiyo-masui/bitshuffle GitHub repository
…s. (#61) * Support BitShuffle as a pre compression filter so it can be combined with other codecs rather than just lz4 * Allow for an external LZ4 library. This will allow for GenomicsDB to use the Intel IPP Optimized LZ4 library * Fix travis builds as lz4 codec is included in the repository * Move lz4 sources to the latest 1.9.2 released version * Add error checking with bshuf_bitshuffle specific error codes * Compute offset compression levels separate from compression levels while creating Codec
…ontext. (#63) * Add support for enable_shared_posixfs_optimizations through tiledb context. This sets disable_file_locking and keep_file_handles_open for posix filesystems * Allow TileDBUtils methods to set enable_shared_posixfs_optimizations options via tiledb context * Just delete the workspace dir When replace is true for intialize_workspace as they may have non-tiledb artifacts (e.g. GenomicsDB workspaces)
…l uses the 1.6.3 version (#65)
* Allow read-only locking when arrays are opened in read mode * Change to match changes in consolidation locking behavior Co-authored-by: nalinigans <nalini@omicsautomation.com>
…offsets_compression_level (#69)
* Move build to xenial * Fix reference to JAVA_HOME in the travis resources
* Initial direct Azure Blob Storage support using C++ Lite client
* Fix doc and rename variable to rationalize azure::download_blob_to_stream usage * First pass at support for buffered IO * Implement chunked buffering for reading/writing tiles * Use size_t/off_t for all file read/write buffer methods * Refine the buffered read algorithm
* Implement github action workflows for hdfs/gcs/azure builds
* Use trailing underscore with class variables * Cleanup run_dfs_tests script
…79) * Rename url references to uri * Remove references to master catalog and tiledb_home_ as this is not used anymore * Move common cloud helper routines to base class storage_fs * Update copyrights
…) to return ssize_t (#82) * Fix deletion of new array objects in benchmark test * Adding checks for all malloc/realloc calls and refactoring file_size() to return ssize_t to allow for return values of -1 * Copyright updates and adding to tests
* Include support for azure blob endpoints * Install azurite to run azure tests in GitHub workflow
…ing with github actions. (#85) * cmake support for building with aws sdk * add_dependencies to tiledb objects for resolving include dependencies to awssdk * awssdk core modules seem to require the macos framework while creating lbtiledb.dylib, does not require the framework for the static library though. * Fix what looks like a transitive dependency with the framework on Mac with awssdk static libraries - not sure why they introduce this * Support S3 using aws-sdk-cpp libraries. Also setup minio for testing this with github actions. * Allow github actions to test on aws and minio * Support multiple builds running concurrently with run_dfs_tests
Codecov Report
@@ Coverage Diff @@
## develop.jpeg2000 #89 +/- ##
====================================================
+ Coverage 61.67% 61.69% +0.01%
====================================================
Files 44 56 +12
Lines 16205 17390 +1185
====================================================
+ Hits 9994 10728 +734
- Misses 6211 6662 +451
Continue to review full report at Codecov.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This gets all the changes merged in, but have pushed jpeg2k validation changes out of tiledb.cc and into codec.[h|cc] for now. Once, we have a
codec registering schemein place all these changes can migrate with codec_jpeg.[h|cc] where the codec is implemented.@kgururaj, do you mind if the
bloscsupport is pulled out? Also, maybe it will be a good time to move array_schema to a json format when we implement apluginsort of a mechanism for codecs.