Remove filtering of duplicate quads

[karelklima]

There was a bug in Comunica causing duplicate quads returned with some queries, which does not conform to the SPARQL standards. As a result, LDkit had to filter the quads to prevent duplicates.

The bug in Comunica has been fixed, so the filtering on LDkit side can be removed.

comunica/comunica#52

[ddeboer]

Unfortunately, I’m getting duplicate quads again, also reported in comunica/comunica#52.

[karelklima]

@ddeboer What version of LDkit and Comunica are you using please?

[ddeboer]

LDkit 2.5.0 with Comunica 4.3.0.

[karelklima]

Thanks for the clarification, I will take a look at it today.

[ddeboer]

As mentioned in comunica/comunica#52 (comment), adding distinctConstruct works directly in Comunica (but not yet through LDkit).

[karelklima]

@ddeboer Do you have some code online that includes the error so that I can take a look at it / debug it?

I need to make sure that the flag is set correctly.

The tests in LDkit work well, although for Comunica version 4.3.0 they are not dependent on the distinctConstruct apparently. Setting it to false still produces correct results. That didn't happen in prior versions.

[karelklima]

I will use the example you shared in comunica/comunica#1590 for my tests, let's see if that works or not.

[ddeboer]

@karelklima Had a quick look and could it be the distinctConstruct parameter simply isn’t being passed to the query engine?

Also: do I need to use the Comunica query engine explicitly or would this work with LDkit out of the box?

[karelklima]

@ddeboer I am preparing a test repo to verify that it is working with Comunica as expected (or not).

In terms of query engine - the default query engine is essentially just a SPARQL endpoint proxy. Based on my testing, SPARQL endpoints do not return duplicate quads for CONSTRUCT queries, at least I haven't encountered that issue yet. I have tested Virtuoso and GraphDB, both were without issues. But if you do not use Comunica with LDkit and are running to duplication issues, then it seems like there might be another SPARQL engine that returns duplicate quads. The distinctConstruct parameter is Comunica specific. If there is another engine that returns duplicates, then I will have to revert my change and incorporate the deduplication algorithm directly in LDkit (I wanted to avoid that for performance reasons, although to be honest the impact is not too big).