Skip to content

Comment on pre-existing table crashes target-postgres #197

@laurentS

Description

@laurentS

I tried running this target (via meltano) with postgres_schema set to an existing schema which already contained tables (with data). Some of the tables had comments on them, as created by:

COMMENT ON TABLE public.mytable IS ''my custom table with important data';

When running the target, it crashed with the following stacktrace:

target-postgres | ERROR Exception writing records
target-postgres | Traceback (most recent call last):
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 237, in write_batch
target-postgres |     self.setup_table_mapping_cache(cur)
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 224, in setup_table_mapping_cache
target-postgres |     table_path = json.loads(raw_json).get('path', None)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/__init__.py", line 357, in loads
target-postgres |     return _default_decoder.decode(s)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 337, in decode
target-postgres |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 355, in raw_decode
target-postgres |     raise JSONDecodeError("Expecting value", s, err.value) from None
target-postgres | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
target-postgres | CRITICAL ('Exception writing records', JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))

With some extra logging, raw_json actually contains the value of the comment mentioned above, which clearly isn't JSON.
This is the line of code that breaks

table_path = json.loads(raw_json).get('path', None)

Is this a bug, or does target-postgres expect the schema to be empty when it starts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions