Sometimes the scrapper skips adding some files to total_items

[jetrotal]

I solved it by adding a second total_items = total_items.concat(items); right before the function resolves itself.

Idk if that solves every case of this issue, since I didn'tunderstand it for real... I guess the counter couldn't deal with every time it captures an asset.

if (items.length <= 0) {
                    this.page.off('response');
                    total_items = total_items.concat(items); //👈
                    resolve(total_items);
                    break;
                } else {
                    total_items = total_items.concat(items);
                    console.log(`${total_items.length} collected so far`)
                    offset += count;
                }

---

EDIT - didn't solve at all, it skips 4 or 5 files at random.

---

[jetrotal]

A workaround I did to get all the items in a collection is to change the getRequestHeader to:

await this.page.goto(`https://opensea.io/collection/${this.config.collectionName}?search[sortAscending]=${mode}`)

the mode loads the assets on ascending order or descending.

With that param, I repeated all the opensea scrape functions 2 times:
First one setting mode to true, then to false.

It works well for small collections. The bigger ones idk

[sametcodes]

Thank you for reporting this issue! But I need some detalis.

I solved it by adding a second total_items = total_items.concat(items); right before the function resolves itself.

Idk if that solves every case of this issue, since I didn'tunderstand it for real... I guess the counter couldn't deal with every time it captures an asset.

@jetrotal Could you share the collection name? I need to reproduce the bug and then I'm going to check what could cause the issue.

With that param, I repeated all the opensea scrape functions 2 times: First one setting mode to true, then to false.

@jetrotal I would appreciate it if you could share the changes you made with a PR and see is it possible to add up the changes to the codebase.