You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 3, 2024. It is now read-only.
This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the dialect-map-core models, or the dialect-map-data-diff DiffMessage model) mismatch the adapters and models defined within this repository.
For those cases, it would be great to adapt the current pubsub-job execution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️
Then, create a recovery-job execution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.
This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the
dialect-map-coremodels, or thedialect-map-data-diffDiffMessage model) mismatch the adapters and models defined within this repository.For those cases, it would be great to adapt the current
pubsub-jobexecution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️Then, create a
recovery-jobexecution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.