PoC: Iceberg incremental read support

[cbb330]

Summary

PoC validating that Iceberg's incremental read APIs work correctly on the OpenHouse stack. Users have asked about incrementally consuming data between snapshots for OH tables — this PR confirms the feature is supported and documents version-specific behavior and UX caveats.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

New Features / Tests:

IncrementalReadTest.java (Spark 3.1 & 3.5) — tests IncrementalAppendScan via the DataFrame API:

spark.read.format("iceberg")
  .option("start-snapshot-id", startId)
  .option("end-snapshot-id", endId)
  .load("catalog.db.table")

• Basic incremental read between two snapshots
• Single-snapshot-range precision
• Multi-snapshot spanning reads
• Overwrite in range: Iceberg 1.2 throws UnsupportedOperationException; Iceberg 1.5 silently skips non-append snapshots

ChangelogViewTest.java (Spark 3.5 only) — tests create_changelog_view stored procedure (Iceberg 1.4+):

CALL catalog.system.create_changelog_view(
  table => 'db.tbl',
  options => map('start-snapshot-id', '1', 'end-snapshot-id', '2'))

• Appends, overwrites, deletes, net changes, multi-snapshot span

Key UX caveats for users:

1. DataFrame API only — no Spark SQL syntax for incremental reads
2. IncrementalAppendScan is append-only; use create_changelog_view for overwrites/deletes
3. create_changelog_view is only available on Spark 3.5 (Iceberg 1.5+)
4. Does not work on views — only base Iceberg tables

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

All tests pass:

• IncrementalReadTest: 4/4 on Spark 3.1, 4/4 on Spark 3.5
• ChangelogViewTest: 5/5 on Spark 3.5
• Spotless formatting applied

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.