Support delta and iceberg via builder pattern

[george-zubrienko]

Key changes:

1. SparkSessionProvider Refactor and Catalog Support

• Refactored SparkSessionProvider to enable modular configuration for Delta Lake, Iceberg REST, and Hive Metastore backends using dedicated with_* methods and config models.
• Added support for using Apache Iceberg through session configurator

2. Docker Compose Stack for Local Development

• Added a new docker-compose.yaml for testing Iceberg integration.

3. CI/CD Workflow and Dependency Updates

• Updated GitHub Actions workflows for building and releasing: switched to Docker Compose for environment setup, upgraded action versions, improved dependency installation via a reusable Poetry action, and set up code coverage configuration to exclude tests.
• Modernized dependencies in pyproject.toml, including making delta-spark and kubernetes optional and updating dev dependencies to latest major versions. Added pytest config to specify test paths.

4. Linting and Minor Configuration Improvements

• Adjusted .pylintrc to enable missing module docstring warnings and increased maximum arguments.

AI generated summary - edited

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
bootstrap-lk.py	13	10	23%	6–46, 50
spark_utils/common
functions.py	32	4	88%	56, 62, 80, 112
spark_job_args.py	68	9	87%	106, 113, 135, 151, 170, 179–183
spark_session_provider.py	83	24	71%	60–61, 110, 128, 155, 171, 190–201, 211–281, 294, 310
spark_sql_utils.py	20	10	50%	54–63, 79–82
spark_udf.py	4	2	50%	36–37
spark_utils/dataframes
functions.py	98	48	51%	48, 55–65, 76–78, 93–95, 108–122, 141, 151, 191, 199–202, 211, 230, 246–250, 259–272
spark_utils/dataframes/sets
functions.py	6	4	33%	39–42
spark_utils/delta_lake
delta_log.py	17	1	94%	52
functions.py	60	38	37%	57–67, 97–127, 140–160, 196–197, 200, 220–223
spark_utils/models
iceberg_rest_config.py	21	3	86%	51–53
k8s_config.py	18	2	89%	18–19
test
test_iceberg.py	15	4	73%	9–10, 20–21
test_spark_session_provider.py	24	6	75%	11–12, 24–25, 35–36
TOTAL	693	165	76%

Tests	Skipped	Failures	Errors	Time
59	0 💤	0 ❌	0 🔥	1m 30s ⏱️