Conformance Tests: Sample Payloads/Queries

[ignorantbliss]

Describe the bug
When running conformance tests, it's difficult to replicate errors without knowing the details of the request that is being sent

Expected behavior
When the request takes parameters/JSON, some additional detail appears showing what the request was.

Proposed resolution
Along with a failure message, a copy of the payload that was sent (ie. for the objects/list call, the JSON sent in the request body).

[jwise-mfg]

This might be a nice feature, or it might clutter the results. Wouldn't it be easier just to open the Developer Tools tab in your browser and just watch the request/response there?

[ignorantbliss]

I didn't actually look - wasn't sure if the tests were run server-side or client-side.

If they're all client-side, that works - but it'd still be nice to have a little expandable/accordion control under the failed test that I could open to see the JSON that was sent.

Hidden-by-default is certainly a sensible way to do it - it'd be unnecessary clutter in most cases, but it's very helpful in some circumstances.

[ignorantbliss]

I've just tried what you suggested, and the results seem to be running server-side not client-side.

So using debug tools doesn't work - our web-browsers aren't actually sending the individual requests, so there's no way to see what is sent.

[ignorantbliss]

Note that this also makes it impossible to test i3X servers in closed networks - it has to be internet-accessible for your server to reach it and perform the test. Luckily this isn't a major issue for us, but I'd imagine it's awkward for some.

If a client-powered version (one where the HTTP requests originated from the browser) was available, it would resolve some of these issues.

[jwise-mfg]

You're right, the test suite runs server-side. i3X Explorer does expose developer tools that can be used to trace responses.

However, this does not limit testing inside closed networks. Dozens of such tests have already been performed.

I haven't seen any other reports that people don't know what their own responses look like and I'm not convinced this tool is the right way to do that, but I'll leave this open for anyone else to chime in if the original issue is a constraint for them.

[ignorantbliss]

I didn't notice the downloadable version - that resolves the isolated network issue.

In my case, I had a 500 error when testing using the conformance test that I couldn't replicate in my own. I had to download the tester and add debugging outputs to find out which specific types/objects were being filtered for and subscribed to so I could replicate the issue - they only happened with very specific objects.