S3 Syncer Sidecar

A lightweight Docker container that continuously syncs data from AWS S3 to a local directory. Designed to run as a sidecar container alongside the HL7 Log Extractor.

Purpose

This sidecar enables the HL7 Log Extractor to read files from a local directory (/data) while the actual source data resides in S3. The syncer continuously pulls new files from S3, making them available to the extractor.

Features

• Continuous or one-time sync modes
• Configurable sync interval
• Optional deletion of local files not present in S3
• IAM role-based authentication (no credentials required)
• Minimal resource footprint

Environment Variables

Variable	Required	Default	Description
S3_SOURCE_BUCKET	Yes	-	S3 bucket name to sync from
S3_SOURCE_PREFIX	No	-	S3 key prefix (folder path)
SYNC_DESTINATION	No	/data	Local directory to sync to
SYNC_INTERVAL	No	60	Seconds between syncs (continuous mode)
SYNC_DELETE	No	true	Delete local files not in S3
SYNC_MODE	No	continuous	continuous or once

Usage

Standalone

docker run \
  -e S3_SOURCE_BUCKET=my-bucket \
  -e S3_SOURCE_PREFIX=hl7-logs/ \
  -e SYNC_INTERVAL=30 \
  -v /local/path:/data \
  ghcr.io/washu-tag/s3-syncer:latest

As Kubernetes Sidecar

See helm/extractor/hl7log-extractor/templates/deployment.yaml for integration example.

Building

cd docker/s3-syncer
docker build -t s3-syncer:latest .

IAM Permissions Required

The container requires the following S3 permissions on the source bucket:

{
  "Effect": "Allow",
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:GetBucketLocation"
  ],
  "Resource": [
    "arn:aws:s3:::bucket-name",
    "arn:aws:s3:::bucket-name/*"
  ]
}

Architecture

┌─────────────────────────────────┐
│       Kubernetes Pod            │
├─────────────────────────────────┤
│  ┌──────────┐   ┌────────────┐ │
│  │ S3 Syncer│   │ HL7 Extract│ │
│  │ Sidecar  │   │    -or     │ │
│  │          │   │            │ │
│  │ Syncs S3 │   │ Reads from │ │
│  │ → /data  │   │ /data      │ │
│  └────┬─────┘   └─────┬──────┘ │
│       │               │        │
│       └───────┬───────┘        │
│          Shared emptyDir       │
│          /data volume          │
└─────────────────────────────────┘
            │
            ▼
      ┌──────────┐
      │ AWS S3   │
      │ Bucket   │
      └──────────┘