Testing improvements

[pcattori]

💬 RFC

Goals:

• Easier to write unit tests (no need to understand responses API) (Utilities for fake resources #424)
• End-to-end (E2E) tests for workflows
  • Continuous mastering
  • Continuous categorization
  • Setup mastering project
• Track Tamr versions: "which was the most recent version of Tamr this ran successfully on?"
• (Ideal) property-based tests

🔦 Context

💻 Examples

import pytest

import tamr_client as tc
from tests.utils import fake_tamr

@fake_tamr
@given(s=session, i=instance)
def test_from_resource_id_mastering(s: Session, i: Instance):
    p = tc.project.from_resource_id(s, instance, "1")

    assert isinstance(p, tc.MasteringProject)
    assert p.name == "proj"
    assert p.description == "Mastering Project"

[skalish]

The naming of these testing resources as fake has now come up a few times in conversation. "Fake" is a generally accessible word, but as far as I can gather it has a pretty specific definition in the context of "test doubles", where fakes are test objects with working (but limited) implementations. In the utilities introduced in #424, fake seems to be an accurate descriptor for the objects returned by fake.session, fake.dataset, etc. however, the use of the responses library is better described as setting up a stub server "that holds predefined data and uses it to answer calls during tests".

I don't think there's an obvious answer to how these utilities should be named, but it's probably worth hashing it out and recording a deliberate decision in an ADR. My main motivation here is being generous towards TC contributors encountering this topic for the first time (I include myself here), since they could benefit if we are thoughtful about the terminology.

[pcattori]

My interpretation:

Mocking a server means faking its responses. I don't think the issue is "mock" vs "fake", but rather "server" vs "response".

If you are faking a server, call it fake.server; if you are faking responses, call it fake.response; if you are faking JSON data call it fake.json.

So I still think the current terminology is appropriate.

[pcattori]

For rationale for faking the server in integration tests, see https://kentcdodds.com/blog/stop-mocking-fetch .