Skip to content

DocumentDB sync service and ChangeStreams feature gate#234

Draft
WentingWu666666 wants to merge 4 commits intodocumentdb:mainfrom
WentingWu666666:users/wentingwu/documentdb-sync
Draft

DocumentDB sync service and ChangeStreams feature gate#234
WentingWu666666 wants to merge 4 commits intodocumentdb:mainfrom
WentingWu666666:users/wentingwu/documentdb-sync

Conversation

@WentingWu666666
Copy link
Collaborator

@WentingWu666666 WentingWu666666 commented Feb 9, 2026

Summary

Add a Python-based change stream sync service that replicates data from a DocumentDB instance (created by the operator) to Azure DocumentDB (with MongoDB compatibility), along with operator changes to support change streams via a new feature gate.

Changes

Sync Service (new: documentdb-playground/sync_service/)

  • sync.py: Collection-level change stream watcher with idempotent upsert/delete sync, exponential backoff retry, and graceful shutdown (SIGINT/SIGTERM)
  • state.py: Crash-safe resume token persistence using atomic file writes (write-to-temp-then-rename), per-collection token tracking, and sync statistics
  • config.yaml: Template configuration for source/target URIs and collection watch list
  • requirements.txt: Python dependencies (pymongo, pyyaml)
  • README.md: Full documentation including Azure DocumentDB creation via az cosmosdb mongocluster create, configuration, usage, troubleshooting, and architecture diagram

Operator - ChangeStreams Feature Gate

  • documentdb_types.go: Added FeatureGates field with ChangeStreams gate to the DocumentDB CRD spec
  • feature_gates_test.go: Unit tests for feature gate validation
  • cnpg_cluster.go: Set PostgreSQL wal_level=logical when ChangeStreams gate is enabled
  • cnpg_cluster_test.go: Tests for WAL level configuration based on feature gate
  • CRD manifests: Updated documentdb.io_dbs.yaml in both config/crd/bases/ and documentdb-helm-chart/crds/
  • zz_generated.deepcopy.go: Auto-generated deep copy for new types

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants