DuckDB Core Pipeline Enhancments: UDF, Attach and SQL Sinks

Enhancements to the core handler pipeline to expose DuckDB. This allows for:

• User Defined Functions - Register and execute UDFs on streaming data (refs #70 )
• User Defined Sinks - Execute SQL on sink data, enabling all DuckDB supported outputs (refs #76 )
• Prometheus Metrics (refs #12 )

Iceberg Sink Support

Adds iceberg as a sink - #63

Implements iceberg sink using pyiceberg https://py.iceberg.apache.org/.

Support for appending to an already defined iceberg table.

Adds pandas deserialization

Adds back in double serialization for the auto-type serialization

refs #48

Config from Envvars starting with SQLFLOW_

Adds support for configuration using envars starting with the SQLFLOW_ prefix.

Updates Jinja2 for critical security vuln.

Updates bluesky config to allow configuration kafka brokers from env vars.

Blueskey Firehose Support!

This release includes primitives to interact with the Bluesky Firehose including:

• Websocket Source
• HTTP Debug Endpoint to run arbitrary SQL against SQLFlow database for debugging purposes
• Local Parquet Sink to write parquet data out to the local filesystem
• Performance improvements by removing redundant serialization

In Memory Tumbling Window Support

• Support for in memory tumbling windows.
• Benchmark Harness

CSV Table Support

• Support for CSV-based join tables
• CSVs can be read dynamically using read_csv()
• CSVs can be loaded statically into memory
• CSV benchmarks
• Enhanced testing of configuration files
• Configuration file naming conventions and standardization

MVP

Created an MVP proving out the DuckDB SQL transformation approach, including:

• Documentation
• Local dev harness
• Test Suite
• Synthetic Data generation
• Config validator
• Benchmarks
• Configuration Examples
• Docker image