Skip to content

Forward lineage must ignore failed jobs #1522

@wajda

Description

@wajda

The impact graph is constructed such that if all execution events were successful. This is not correct.
The design assumption is that failed events do not modify any data and therefore should be ignored in the impact view.

Image Image

Test scenario

export BASE=http://localhost:8080
export CT="application/vnd.absa.spline.producer.v1.2+json"
export CONSUMER="$BASE/consumer"
export PRODUCER="$BASE/producer"
curl -s -X POST "$PRODUCER/execution-plans" -H "Content-Type: $CT" -d '{
  "id": "11111111-1111-1111-1111-111111111134",
  "name": "Lineage vs error UI",
  "operations": {
    "write": { "id": "w1", "outputSource": "jdbc:demo:warehouse/table_z", "append": false,
      "childIds": ["r1"], "extra": { "destinationType": "jdbc" } },
    "reads": [{ "id": "r1", "inputSources": ["jdbc:demo:warehouse/table_a"] }],
    "other": []
  },
  "systemInfo": { "name": "manual-stub", "version": "1.0" }
}'
curl -s -X POST "$PRODUCER/execution-events" -H "Content-Type: $CT" -d '[
  { "planId": "11111111-1111-1111-1111-111111111134", "timestamp": 1735812000000 },
  { "planId": "11111111-1111-1111-1111-111111111134", "timestamp": 1735815600000,
    "error": { "message": "Second run failed", "code": "EX34" } }
]'
curl -s -X POST "$PRODUCER/execution-plans" -H "Content-Type: $CT" -d '{
  "id": "11111111-1111-1111-1111-111111111135",
  "name": "Consumer of Z",
  "operations": {
    "write": { "id": "w1", "outputSource": "jdbc:demo:warehouse/table_p", "append": false,
      "childIds": ["r1"], "extra": { "destinationType": "jdbc" } },
    "reads": [{ "id": "r1", "inputSources": ["jdbc:demo:warehouse/table_z"] }],
    "other": []
  },
  "systemInfo": { "name": "manual-stub", "version": "1.0" }
}'
curl -s -X POST "$PRODUCER/execution-events" -H "Content-Type: $CT" -d '[{
  "planId": "11111111-1111-1111-1111-111111111135", "timestamp": 1795820000000
}]'

Metadata

Metadata

Assignees

Type

No fields configured for Bug.

Projects

Status
Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions