Feat: Databento integration improvements #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…More robust code. Refactored and properly passing tests.
feat: Record Type of different DataBento scheme feat: handle Auth-Request/Response msgs test:feat: deserialize Heartbeat json msg, parse tcp response/request test:feat: LiveApiClient quick connection and debug process
Martin-Molinero
requested changes
Jan 28, 2026
Member
Martin-Molinero
left a comment
Martin-Molinero
left a comment
There was a problem hiding this comment.
Thank you! Shared detailed review in slack, few requests and bug fixing required
Unify market data deserialization using a new DataConverter and MarketDataBase abstract class. Remove custom JsonSettings and legacy deserialization methods. Update all model classes to inherit from MarketDataBase, make LevelOneBookLevel prices nullable, and improve error handling for unknown/invalid JSON. Move event args to .Models.Events. Expand and update tests for new d eserialization logic and edge cases. These changes improve robustness, extensibility, and maintainability of DataBento data ingestion.
- Add ErrorResponse model for structured API error parsing - Refactor HistoricalAPIClient to auto-correct invalid date ranges - Expand unit tests for error scenarios and data bounds - Add tests for error response deserialization - Support additional OHLCV record types in DataConverter - Use snake-case JSON settings for consistent deserialization - Improve logging and handling of empty/malformed responses
Introduce DataSetSpecifications and PredefinedDataSets for richer dataset metadata, including delay info and documentation links. Refactor DataBentoSymbolMapper to map markets to DataSetSpecifications. Update provider logic to log delay warnings and extract DataSetID as needed. Add support for new error case DataStartAfterAvailableEnd and corresponding test. Remove obsolete methods and simplify subscription/history logic.
Replaced hardcoded schema strings with documented constants in HistoricalAPIClient for maintainability. Refactored GetRange to set record limits by schema, preventing timeouts on large requests. Added tests for 11-year minute and second data ranges. Updated all schema usages to use new constants.
Refactored DataBentoProvider to inherit from MappedSynchronizingHistoryProvider, removing custom map file handling and multi-request GetHistory logic. Cleaned up unused usings and simplified history request processing by leveraging the new base class for map file resolution and request splitting.
Refactored error handling in HistoricalAPIClient to prevent repeated log entries for the same error type by introducing boolean flags. Added a helper method to standardize unprocessable content error logs, making logging more concise and maintainable. Removed redundant log statements.
- Added support for DataBento BBO-1s (1s) and BBO-1m (1m) Level 1 quote schemas alongside MBP-1 (tick) - Refactored API and provider to fetch Level 1 data at tick, second, and minute resolutions, mapping to correct schema - Replaced GetTickBars with GetLevelOneData throughout codebase and tests - Enhanced DataBentoProvider to support quote ticks and intraday/interday quote bars with new helper methods - Improved test coverage for Level 1 data at various resolutions and edge cases (null price, ulong.MaxValue ts_event) - Updated LevelOneData and Header models: TsRecv/TsEvent now ulong, Price is nullable, UtcTime handles max value - Added HasBidOrAskPrice extension for LevelOneBookLevel - Improved error handling/logging and updated copyright headers
Refactored GetIntraDayQuoteBars and GetQuoteTicks to process only the top level (best bid/ask) from LevelOneData, instead of iterating through all levels. This simplifies QuoteBar and Tick construction, yields only top-of-book data, and improves performance for Level 1 data scenarios.
Centralize timestamp conversion logic in MarketDataBase with TryGetDateTimeUtc, prioritizing Header.TsEvent and falling back to TsRecv. Remove Header.UtcTime and update all consumers to use the new method, skipping or logging on invalid timestamps. Add exchange time zone conversion and caching in the history provider. Refactor quote bar aggregation and enhance error logging. Update tests for new timestamp logic.
Replaces TryGetDateTimeUtc with explicit UtcDateTime properties on Header and LevelOneData. Updates all consumers to use these properties directly, with null checks for undefined timestamps. Removes MarketHoursDatabase/time zone cache and inlines exchange time conversion.
- Replace _isConnected with ManualResetEventSlim for connection state signaling - Start data receiver task in constructor; separate connection monitoring and data receiving - Improve authentication error handling and throw AuthenticationException with descriptive message - Add CleanupConnection() for centralized resource cleanup - Enhance Dispose() to properly cancel, reset, and dispose resources - Raise ConnectionLost events when connection is lost - Improve exception handling and resource management throughout - Add new test case for "open connection limit" authentication error - Minor code style and documentation improvements
Closed
This was referenced Feb 3, 2026
Remove async/await from data receiving and authentication logic. Replace async methods with blocking StreamReader.ReadLine(). Set NetworkStream.ReadTimeout to detect stalled connections. Improve exception handling and connection reset logic. Simplifies code and makes network operations timeout-aware.
Refactored AuthenticationMessageRequest to require a non-null heartbeat interval, removing default and minimum value logic. Updated related test to pass explicit interval. Added comprehensive XML documentation to the struct, its fields, and methods for improved clarity and maintainability.
Introduce LevelOneDataBase to unify Level 1 data structure. Split tick (MBP1) and interval (BBO1s/BBO1m) data into LevelOneData and BestBidOfferInterval, respectively. Add ActionType enum for type-safe action handling. Update HistoricalAPIClient, DataConverter, provider, and tests to use new types and methods. Improves type safety, clarity, and test coverage for DataBento Level 1 data.
- Replace char/int with strongly-typed Side and Flags enums - Change LevelOneDataBase to use LevelOne (single) instead of Levels (list) - Update deserialization to enforce single level and use new property - Refactor providers, history, and tests to use LevelOne and enums - Add shared assertion helpers and improve documentation and licensing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PRs