Asset 623: all ARAs Pass, but ARS marked as Fail

[isbluis]

For test asset 623, all ARAs report passing this test, but ARS reports a failure for at least the last few test runs:

[maximusunc]

This one is interesting! @ShervinAbd92 can you take a look please? Here's the parent PK: d624af8b-a524-4cac-9a54-6e5b15e8645c It looks like the ARS score for this result is 0.998 and the rank is 45, so it seems like it's in the top 10%.

[ShervinAbd92]

@maximusunc correct me if i am wrong but the topAnswer is currently passing if the expected output is within first 30 results, so the rank 45 will fall out of this range and fail.

[maximusunc]

Then maybe something else is going on. Looking at Aragorn's result output, the answer is ranked at 210 but Aragorn passes the Top Answer test.

[ShervinAbd92]

i see whats happening, in Aragorn results, there are 4 instances of 'Rabeprazole', the first 2 being at result 3 & 15. Most likely is taking those and pass it

[maximusunc]

Ok, I see. I'm going to open an issue in the test runner to stop after it finds the first instance of the answer so we don't account for duplicate answers.

[isbluis]

Are we in general concerned with duplicate results from a single ARA, though? Perhaps that alone should merit a fail of some sort?

[maximusunc]

But then my next question is, why is this answer coming out at rank 45 when it's in the top 10 of most ARAs? And yes, I'm also concerned about this Luis, but I don't think it's necessarily a prevue of the tests at this point.

[isbluis]

Hi all,
Even though Asset 623 seems to have been fixed, I see this same situation happening with Asset 621, and they seem to be related somehow.
Of note: Asset 621 used to pass this test (and 623 failed) up to Test Run 161. Since Test Run 162, Asset 621 has failed, while 623 has passed (in CI).

[ShervinAbd92]

Yes @isbluis after looking into the query. i see that for Aragorn has multiple instance of Pantoprazole and thats probably causing the ARS merge to be ranked at 54 and therefore failing the topAnswer threshold.