Add CREATE TABLE ... WITH (...) support for ad-hoc K8s-stored tables

[ryannedolan]

Summary

• Added Table CRD.
• Added create table ... with DDL.
• Added planner rule to match such tables.

Details

Currently, all tables must exist within a Database, backed by a JDBC driver. Often, we want to point to a table-like object that doesn't exist in the catalog, or that differs appreciably from a table in the catalog. For this reason, we add create table ... with, which lets users add connector configuration to the catalog directly. Such tables exist in the catalog but do not necessarily have a physical Database associated with them. The pipeline planner simply injects whatever configuration is provided in the with clause, plus any configuration from matching TableTemplates.

Testing Done

0: Hoptimator> create table foo (name varchar) with (connector 'likafka', topic 'my-topic');
...
0: Hoptimator> !pipeline create materialized view venice.bar as select * from foo
CREATE DATABASE IF NOT EXISTS `DEFAULT` WITH ();
CREATE TABLE IF NOT EXISTS `DEFAULT`.`FOO` (`NAME` VARCHAR) WITH ('CONNECTOR'='likafka', 'TOPIC'='my-topic', 'offline.table.name'='ads_offline', 'job.properties.account'='foo', 'pipeline'='VENICE.BAR');
CREATE DATABASE IF NOT EXISTS `VENICE` WITH ();
CREATE TABLE IF NOT EXISTS `VENICE`.`BAR` (`NAME` VARCHAR) WITH ('connector'='venice', 'key.fields'='KEY', 'key.fields-prefix'='', 'key.type'='PRIMITIVE', 'partial-update-mode'='true', 'storeName'='BAR', 'value.fields-include'='EXCEPT_KEY', 'offline.table.name'='ads_offline', 'job.properties.account'='foo', 'pipeline'='VENICE.BAR');
INSERT INTO `VENICE`.`BAR` (`NAME`) SELECT `NAME` FROM `DEFAULT`.`FOO`;

[ryannedolan]

I don't think database should be required. It looks like Claude is currently filling it in as DEFAULT, but it should be null in most cases. I think the only reason to include a database is if you explicitly want to opt-in to connector configuration from a database-specific TableTemplate.

[ryannedolan]

I think database should be null in this case?