Skip to content

Improve robustness of metadata results merging #34

@michaelwood

Description

@michaelwood

In https://github.com/ThreeSixtyGiving/datagetter/blob/master/getter/get.py#L297 we currently

  1. Take the list of publishers/files from the registry
  2. In each multiprocess take one of the entries from the list and add the metadata information
  3. recreate the list of publishers/files from the registry from the returned results of (2)

If something goes wrong we should return the original entry from the registry to rebuild the list, this is done by catching all the exceptions and returning the original data. This is fine.. but we don't want to rely on that mechanism, especially in the case where one of those multiprocesses gets terminated without an exception but doesn't terminate the parent process (if that is even possible).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions