DE-8265: Upgrade DBT version to 1.9.x

[jbreeden]

Overview

This is another stab at uptaking the latest DBT changes (basically #41, but rebased on main and with some additional changes).

This upgrade has been a long time coming, but progress has been slow due to a number of stops & starts, with long pauses in between. I'm putting this up to explain current status and seek some help from another pair of eyes.

Current Status & Challenges

• I picked up the branch that Eric started months back
• At some point I had the unit tests passing (I don't think they do on the current HEAD)
• When I tried to run the examples, I hit some unexpected errors and eventually identified that it's not using the materializations for our adapter, and is instead falling back to defaults in the "global_project" provided by the dbt-adapters package.
• I went through the release & migration notes for the DBT version we were based on, up to 1.9.x, and found the changes were pretty substantial (including this dbt-core/dbt-adapters decoupling, which is a significant change in how the adapters works)
• Since then I've mostly been trying to connect the dots as to why it's not picking up our materializations
  • Many chats with AI have not been helpful 😭
  • I tried scaffolding a new adapter and comparing some of the setup with ours, and didn't spot anything
  • I've been through the docs on the macro & materialization routing, and tried all of the troubleshooting steps I've found, and had no luck

In the end I feel it may be something very silly that I'm missing, and I just can't spot it.

Here's an example run of one of the examples, amended to highlight some of the logs that have helped me trace the issue so far (and omit some noise):

❯ DECODABLE_PROFILE=jbreeden dbt run --select http_events --debug
...
##### NOTE: We're on dbt 1.9.2
16:37:14  Running with dbt=1.9.2
16:37:14  running dbt with arguments {'printer_width': '80', 'debug': 'True', 'profiles_dir': '/Users/jbreeden/.dbt', 'log_format': 'default', 'warn_error': 'None', 'use_colors': 'True', 'empty': 'False', 'log_cache_events': 'False', 'partial_parse': 'True', 'introspect': 'True', 'cache_selected_only': 'False', 'invocation_command': 'dbt run --select http_events --debug', 'static_parser': 'True', 'target_path': 'None', 'quiet': 'False', 'no_print': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'fail_fast': 'False', 'use_experimental_parser': 'False', 'log_path': '/Users/jbreeden/decodable/repos/dbt-decodable/example_project/example/logs', 'indirect_selection': 'eager', 'write_json': 'True', 'version_check': 'True', 'send_anonymous_usage_stats': 'True'}
...
##### NOTE: It's picked up our adapter
16:37:14  Registered adapter: decodable=2.0.0-rc3
...
##### NOTE: It's performing a rename that's not part of our materialization logic.
##### NOTE: This threw me for a while until I realized it's coming from the default materializations
##### NOTE: and reflects some more recent changes to how they materialize tables into a temp location first
  File "/Users/jbreeden/decodable/repos/dbt-decodable/dbt-venv/lib/python3.10/site-packages/dbt/adapters/decodable/impl.py", line 314, in rename_relation
    traceback.print_stack()
16:37:16  Compilation Error in model http_events (models/example/http_events.sql)
  Relation "".""."http_events__dbt_tmp" not found!

##### NOTE: This materialization comes from the global_project module in dbt-adapters.
  > in macro materialization_table_default (macros/materializations/models/table.sql)
  > called by model http_events (models/example/http_events.sql)
16:37:16  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '5cc7a623-4213-4295-b555-242945ede993', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1083075b0>]}
16:37:16  1 of 1 ERROR creating sql table model ..http_events ............................ [ERROR in 0.69s]
16:37:16  Finished running node model.example.http_events
16:37:16  Marking all children of 'model.example.http_events' to be skipped because of status 'error'.  Reason: Compilation Error in model http_events (models/example/http_events.sql)
  Relation "".""."http_events__dbt_tmp" not found!

  > in macro materialization_table_default (macros/materializations/models/table.sql)
  > called by model http_events (models/example/http_events.sql).
...
16:37:16
16:37:16  Completed with 1 error, 0 partial successes, and 0 warnings:
16:37:16
16:37:16    Compilation Error in model http_events (models/example/http_events.sql)
  Relation "".""."http_events__dbt_tmp" not found!

  > in macro materialization_table_default (macros/materializations/models/table.sql)
  > called by model http_events (models/example/http_events.sql)
16:37:16
16:37:16  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1=
...