Incomplete import from archivist

[shibbard-shibumi]

Describe the Bug

On both world setup and sync I noticed that the list of imported characters/items/locations is incomplete. In Sync diffs I see a list of items that are flagged as deleted. In Sync Import New, I see a list of items that weren't there before. Is there a limit to the number of items that the API returns per call?

Steps to Reproduce

1. Go to "Journal"
2. Click on 'Sync with Archivist'
3. Wait for list to load
4. Import all available items
5. Wait for sync to complete.
6. Click on 'Sync with Archivist' again
7. Wait for list to load
8. Notice "Review Diffs" has items that have not been deleted in Archivist
9. Notice "Import New" has items that weren't shown on the last sync and have not been added to Archivist since the last sync.

Expected Behavior

9. Notice "Review Diffs" is empty
10. Notice "Import New" is empty

Screenshots

1st sync

2nd sync

Environment

• Foundry VTT Version: v13.351
• Module Version: 1.3.12
• Game System: dnd5e v5.2.5
• Browser: Chrome Version 139.0.7258.154
• Operating System: Ubuntu 24.04.3 LTS

Console Errors

Additional Context

I have a decently large campaign.
160 characters
52 items
80 locations
32 factions

[scotthibbard]

I found the root cause of this. The Archivist API was not returning characters in a consistent order which meant that pagination did not work as intended. Fetching the first page twice in a row did not yield the same results. So that means in my use case where I had 153 characters, the first page would contain 100 characters. The second page would have 53 characters consisting of a mix between characters already received in the first page and new characters. This also means, that some of the 153 characters would be missing.

Then, in this bit of code, the duplicate characters would be trimmed out, leaving a subset of the total.
const byId = { Character: new Map(A.characters.map((c) => [String(c.id), c])), Item: new Map(A.items.map((i) => [String(i.id), i])), Location: new Map(A.locations.map((l) => [String(l.id), l])), Faction: new Map(A.factions.map((f) => [String(f.id), f])), Session: new Map(A.sessions.map((s) => [String(s.id), s])), };

This is why there would be a mix of "deleted" and "new" characters on every sync despite doing a full sync previously.

I've already reached out to the Gregs on their discord server and they fixed the API. I tested it and the issue is gone! This issue can be closed.