Skip to content

Sumitbansal96/data-sync-pipeline

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 

Repository files navigation

Cross-Platform Data Synchronization Pipeline

Case study: Automated contact and campaign data sync between GuruDB and EnumHQ using Python and Google Apps Script.


Business Problem

Contact records and campaign data existed in two separate systems — GuruDB and EnumHQ — with no synchronization between them. This led to mismatched audience data, duplicate outreach, and inconsistent campaign results. Manual reconciliation was periodic, error-prone, and time-consuming.

Objective

Implement a reliable, automated sync layer that keeps contact and campaign data consistent between the two platforms in near real-time, with conflict resolution logic when records diverge.


Tools & Platforms

Layer Tools
Scripting Python, Google Apps Script
Source system GuruDB
Target system EnumHQ
Scheduling Trigger-based (time-driven, Apps Script)
Data format JSON via REST APIs

Architecture Overview

[GuruDB — Source of truth]
         ↓
[Python extract script]
         ↓
[Data validation + conflict detection]
         ↓
[Google Apps Script — orchestration layer]
         ↓
[EnumHQ API — write / update records]
         ↓
[Error log + notification on failures]

Implementation Summary

  1. Extract — Python script pulls updated records from GuruDB via API, filtered by modification timestamp
  2. Transform — Field mapping converts GuruDB schema to EnumHQ-compatible format; normalizes phone numbers and email casing
  3. Conflict detection — Compares last-modified timestamps; newer record wins; ambiguous conflicts flagged for review
  4. Load — Google Apps Script handles the write to EnumHQ API with retry logic on rate limit errors
  5. Error handling — Failed records logged to a Google Sheet for manual review; notification triggered on batch failures
  6. Scheduling — Apps Script time-driven trigger runs sync on a regular cadence

What Was Kept Private

  • Actual API credentials and endpoint URLs
  • Customer data and PII
  • Proprietary field structures and business logic specifics
  • Client-specific campaign configurations

Outcome

  • Consistent contact data across both platforms — no more manual reconciliation
  • Reduced campaign errors caused by stale or mismatched audience data
  • Error logging gave visibility into data quality issues that previously went unnoticed
  • Eliminated a recurring manual task from the operations workflow

What I Learned

  • Timestamp-based conflict resolution works well but requires both systems to use UTC — timezone mismatches cause subtle sync failures
  • Google Apps Script execution quotas (6 min/run) shape how batch sizes are structured; chunking is essential for large record sets
  • REST APIs with rate limits need exponential backoff, not simple retry — naive retries amplify the rate limit problem
  • A lightweight logging layer (even a Google Sheet) dramatically reduces time spent debugging sync failures

Screenshots / Diagrams

[Add sanitized architecture diagram and workflow screenshots here — no PII or credentials visible]


Sumit Bansal — sumit.mft@gmail.com | LinkedIn

About

Case study: Cross-platform contact and campaign data sync between GuruDB and EnumHQ using Python and Google Apps Script.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors