Customer ETL pipeline failing with OutOfMemoryError during nightly batch job

[archdata-ai]

Description

The customer ETL pipeline is consistently failing with OutOfMemoryError during the nightly batch processing job.

Steps to Reproduce

1. Wait for the scheduled nightly batch job at 2am
2. Monitor the ETL pipeline execution
3. Observe the failure with OOM error

Expected Behavior

The ETL pipeline should complete successfully without memory-related failures during the nightly batch job.

Actual Behavior

The pipeline fails with java.lang.OutOfMemoryError during execution.

Error Details

java.lang.OutOfMemoryError

Additional Context

• Frequency: Occurs daily at 2am
• Job Type: Nightly batch job
• Environment: Customer production environment
• Impact: Daily ETL processing is disrupted

Suggested Actions

• Analyze memory usage patterns during batch processing
• Review heap size configuration
• Investigate data volume changes
• Consider memory optimization or scaling solutions