Skip to content

Overhaul ETL framework with advanced pipeline architecture#9

Open
bjaus wants to merge 1 commit intomainfrom
overhaul-etl-framework
Open

Overhaul ETL framework with advanced pipeline architecture#9
bjaus wants to merge 1 commit intomainfrom
overhaul-etl-framework

Conversation

@bjaus
Copy link
Copy Markdown
Owner

@bjaus bjaus commented Mar 29, 2026

Complete overhaul of the ETL package from a minimal 3-file Pipeline[S,T] implementation into a production-grade framework.

Core changes:

  • 4 job types (ScanJob, FetchJob, SeekJob, PageJob) with specialized runners replacing the single generic pipeline
  • 40+ optional interfaces discovered via type assertion at construction — error handling, dead-lettering, panic recovery, lifecycle/stage hooks, filtering, progress reporting
  • Checkpointing support for resumable jobs with epoch-based cursor tracking
  • Resilience layer (rate limit → circuit breaker → retry) configurable per-stage
  • Graceful shutdown with drain timeout and failureSignal coordination

Subpackages:

  • batch/ — 9 pull-based collector implementations (Simple, Sized, Grouped, Striped, Windowed, Dedupe, Coalescing, Priority, Sorted)
  • compose/ — Embeddable composables for common patterns (ErrorSkip, ErrorSkipN, ErrorStage, PanicSkip, Lifecycle, Progress) using *slog.Logger
  • resilience/ — Retrier, TokenBucket/LeakyBucket rate limiters, RampingRateLimiter (6 curves), CircuitBreaker

Other:

  • Internal dependencies replaced with stdlib (slog, sentinel errors)
  • Documentation and examples use generic domain types throughout

Replace the simple 3-file Pipeline[S,T] implementation with a
production-grade framework supporting 4 job types (Scan, Fetch, Seek,
Page), 40+ optional interfaces, pluggable batch collectors, resilience
primitives, checkpointing, and comprehensive lifecycle/stage hooks.

Adds batch/, compose/, and resilience/ subpackages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant