Skip to content

Bug: Argentina preprocess doesn´t skip based on variable #205

@outerlook

Description

@outerlook

Problem

preprocess_flow.py processes ALL available dates instead of respecting LAST_PREPROCESS_SUCCESS_DATE like other Argentina flows do.

Current behavior (line 188-194):

for date in self.raw_provider.list_available_keys():
    if self.processed_provider.exists(date):
        continue
    await self.process_date(date)

Expected behavior: Filter dates by LAST_PREPROCESS_SUCCESS_DATE first, then check if already processed.

Solution

Add date filtering before the loop:

last_preprocess_date = await variables.Variable.aget(
    ArgentinaFlowVariableNames.LAST_PREPROCESS_SUCCESS_DATE,
    default=ArgentinaFlowVariableNames.DEFAULT_DATE
)
dates_to_process = [d for d in self.raw_provider.list_available_keys() if d > last_preprocess_date]

This matches the pattern already used in aggregate_products_flow.py and insert_products_flow.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions