Skip to content

Fix #767: drop duplicates when merging '..._append' lists#778

Open
phverg wants to merge 1 commit intoolofk:mainfrom
phverg:fix-fileset-append-duplicates
Open

Fix #767: drop duplicates when merging '..._append' lists#778
phverg wants to merge 1 commit intoolofk:mainfrom
phverg:fix-fileset-append-duplicates

Conversation

@phverg
Copy link
Copy Markdown
Contributor

@phverg phverg commented May 7, 2026

Problem

Fixes #767.

A natural inheritance pattern using YAML merge plus ..._append:

sim_base: &sim_base
  filesets_append: [files_dv]
  ...
unit_test:
  <<: *sim_base
  filesets_append: [files_dv]

…leaves unit_test with filesets_append: [files_dv, files_dv] after
YAML's list merge. Every file in files_dv then ends up listed twice in
the emitted EDAM, and the EDA backend rejects the build with a
file already defined error. The same happens for files_append
re-listing a file already present in files. The natural inheritance
pattern is effectively unusable.

Solution

Deduplicate when extending the base list inside
CoreData._append_lists. Items that are already present (in the base
list or earlier in the same ..._append) are dropped; new items are
still appended in order. This keeps the existing append-only and
base-only test cases unchanged.

Test plan

  • Extended tests/capi2_cores/misc/append.core with three new
    targets covering the duplicate cases (in-base, within-append,
    append-only).
  • Extended test_capi2_append to assert duplicates are dropped and
    order preserved; existing assertions still pass.
  • Manually verified the issue scenario (YAML merge + double
    filesets_append): EDAM file list now contains each file once.
  • Full pytest suite green locally.
  • pre-commit run -a clean.

When YAML-merge target inheritance left a target with both a base
list and an ``..._append`` list pulling in items already in the base
list (or with duplicates inside ``..._append`` itself), the merged
list kept the duplicates. Worked example::

    sim_base: &sim_base
      filesets_append: [files_dv]
      ...
    unit_test:
      <<: *sim_base
      filesets_append: [files_dv]

YAML merge concatenates the two ``filesets_append`` lists, so
``files_dv`` ended up listed twice on ``unit_test``. Every file in
``files_dv`` then appeared twice in the emitted EDAM, and the EDA
backend rejected the build with a ``file already defined`` error.
The natural inheritance pattern was effectively unusable.

Deduplicate when extending the base list inside
``CoreData._append_lists``. Items unique to ``..._append`` are still
appended in order; items already present (either via the base list
or earlier in the same ``..._append``) are dropped. This keeps the
existing append-only and base-only test cases unchanged.

Fixes olofk#767.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@olofk
Copy link
Copy Markdown
Owner

olofk commented May 10, 2026

I understand what this fixes, but thinking about it I can't see how on earth we would get into this situation. The provided example kind of undermines the whole idea of inheritance as the point is that you would not have to add the fileset in the target that inherits. I realize your example is intentionally contrived, but could you (or someone else) come up with a more realistic situation where this could happen?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CAPI2 Syntax Gotcha: fileset_append existing fileset

2 participants