Support for multiple firestore clients

[mfisher29]

--- THIS COMMENT IS OUTDATED AND THEREFORE ONLY KEPT FOR REFERENCE --

This pull request intends to resolve issue #86.

I have followed the discussion there and created what I believe to be a nice solution based on Option 3.

Primary things to note about this solution:

• The current way of configuring a client/db via the configure() method remains in tact.
• Support for multiple clients builds on top of that and relies on storing of a ConfigItem in the CONFIGURATIONS dict.
• Upon creation of a config, both a sync and async client will be set for that config
• Code examples introduced below are included in a multiple_clients_examples directory, and can be removed before merge.
• I have added ~20 more unit tests, but there may be some gaps. Happy to add more with your suggestions.

Other small things:

• readme was updated with some more small details, and I recommend using pytest instead for development. It may be that I don't know poetry well enough to fix this, but I found that upon running poetry run invoke test my current changes were rolled back to the current firedantic version. I suppose this is because poetry is locked to version = "0.11.0" in the pyproject.toml and so I used pytest instead.
• Aside from testing support of multiple clients, some test cases were added to check more types of operands, i.e. op.EQ and "==". All work as expected.
• Apologies for the added spacing everywhere. I believe one of the formatters from the poetry setup did that. If it bothers anyone i can go back and remove it.

Now for the examples...

Example of client creation:

## OLD WAY:
def configure_client():
    # Firestore emulator must be running if using locally.
    if environ.get("FIRESTORE_EMULATOR_HOST"):
        client = Client(
            project="firedantic-test",
            credentials=Mock(spec=google.auth.credentials.Credentials),
        )
    else:
        client = Client()

    configure(client, prefix="firedantic-test-")
    print(client)


def configure_async_client():
    # Firestore emulator must be running if using locally.
    if environ.get("FIRESTORE_EMULATOR_HOST"):
        client = AsyncClient(
            project="firedantic-test",
            credentials=Mock(spec=google.auth.credentials.Credentials),
        )
    else:
        client = AsyncClient()

    configure(client, prefix="firedantic-test-")
    print(client)


## NEW WAY:
def configure_multiple_clients():
    config = Configuration()

    # name = (default)
    config.create(
        prefix="firedantic-test-",
        project="firedantic-test",
        credentials=Mock(spec=google.auth.credentials.Credentials),
    )

    # name = billing
    config.create(
        name="billing",
        prefix="test-billing-",
        project="test-billing",
        credentials=Mock(spec=google.auth.credentials.Credentials),
    )

    config.get_client()  ## will pull the default client
    config.get_client("billing")  ## will pull the billing client
    config.get_async_client("billing")  ## will pull the billing async client

Under the hood of config.create():

• (seeconfiguration.py for more details) Created suggested Configuration class for added support of multiple configurations/clients.
• The general idea is to save each configuration to the CONFGURATIONS Dict. When using the singular configure method, it will be saved there as (default).
• New single clients can also be configured with the new multi client method. Multi is not required to use multi.

class Configuration:
    def __init__(self):
        self.configurations: Dict[str, ConfigItem] = {}

    def create(
        self,
        name: str = "(default)",
        prefix: str = "",
        project: str = "",
        credentials: Credentials = None,
    ) -> None:
        self.configurations[name] = ConfigItem(
            prefix=prefix,
            project=project,
            credentials=credentials,
            client=Client(
                project=project,
                credentials=credentials,
            ),
            async_client=AsyncClient(
                project=project,
                credentials=credentials,
            ),
        )
        # add to global CONFIGURATIONS
        global CONFIGURATIONS
        CONFIGURATIONS[name] = self.configurations[name]

For saving/finding entries:

## With single client
def old_way():
    # Firestore emulator must be running if using locally.
    configure_client()

    # Now you can use the model to save it to Firestore
    owner = Owner(first_name="John", last_name="Doe")
    company = Company(company_id="1234567-8", owner=owner)
    company.save()

    # Prints out the firestore ID of the Company model
    print(f"\nFirestore ID: {company.id}")

    # Reloads model data from the database
    company.reload()


## Now with multiple clients/dbs:
def new_way():
    config = Configuration()

    # 1. Create first config with config_name = "(default)"
    config.create(
        prefix="firedantic-test-",
        project="firedantic-test",
        credentials=Mock(spec=google.auth.credentials.Credentials),
    )

    # Now you can use the model to save it to Firestore
    owner = Owner(first_name="Alice", last_name="Begone")
    company = Company(company_id="1234567-9", owner=owner)
    company.save()  # will use 'default' as config name

    # Reloads model data from the database
    company.reload()  # with no name supplied, config refers to "(default)"

    # 2. Create the second config with config_name = "billing"
    config_name = "billing"
    config.create(
        name=config_name,
        prefix="test-billing-",
        project="test-billing",
        credentials=Mock(spec=google.auth.credentials.Credentials),
    )

    # Now you can use the model to save it to Firestore
    account = BillingAccount(name="ABC Billing", billing_id="801048", owner="MFisher")
    bc = BillingCompany(company_id="1234567-8", billing_account=account)

    bc.save(config_name)

    # Reloads model data from the database
    bc.reload(config_name)  # with config name supplied, config refers to "billing"

    # 3. Finding data
    # Can retrieve info from either database/client
    # When config is not specified, it will default to '(default)' config
    # The models do not know which config you intended to use them for, and they
    # could be used for a multitude of configurations at once.
    print(Company.find({"owner.first_name": "Alice"}))

    print(BillingCompany.find({"company_id": "1234567-8"}, config=config_name))

    print(
        BillingCompany.find(
            {"billing_account.billing_id": "801048"}, config=config_name
        )
    )

A simplified example for the find() method:
Note here, the multi client method is using the (default) config, as no config was specified.

print("\n---- Running OLD way ----")
configure_client()

companies1 = Company.find({"owner.first_name": "John"})
companies2 = Company.find({"owner.first_name": {op.EQ: "John"}})
companies3 = Company.find({"owner.first_name": {"==": "John"}})
assert companies1 != []
assert companies1 == companies2 == companies3


print("\n---- Running NEW way ----")
configure_multiple_clients()

companies1 = Company.find({"owner.first_name": "Alice"})
companies2 = Company.find({"owner.first_name": {op.EQ: "Alice"}})
companies3 = Company.find({"owner.first_name": {"==": "Alice"}})
assert companies1 != []
assert companies1 == companies2 == companies3

[joakimnordling]

A big thanks for the PR!

I'm really sorry to say we're pretty busy for the next month, so going to frankly say we're most likely not able to have a deeper look into the PR within the next month.

[joakimnordling]

Okay, I finally manage to get some time to start looking through this PR.

First of all a big thanks for taking the time to get into this! And sorry for the delay!

I think there's some things missing and some things where I've likely not managed to express my idea very clearly in the suggestion I made originally (in #86) or then we might have had just a bit of different visions. I'll try to list those things below.

• My idea was that we'd replace the CONFIGURATIONS dictionary in configurations.py with simply one instance of the Configuration class, by doing something similar to configuration = Configuration() and then use the get_*** methods instead of directly using the dictionary. I think this would hide the actual way we store the data and especially the add() method would be helping out a fair bit with making the configs easily either by providing the values for constructing the clients or then allowing you to pass in the pre-configured clients.
• My idea was also that each model class would keep the info about which configuration it should be using, by referring to it by it's name, for example by using __db_config__ = "billing" you'd tell the class to use the named configuration called billing and it could then be fetched from the config with something like configuration.get_client("billing") where that billing would be looked up from the __db_config__ value of the class. This means you wouldn't need to pass in the name of the config to each of the methods that interact with the DB (save, reload, delete, find etc), which I think would sooner or later lead to bugs when you sometime accidentally forget to pass in the name (imagine reading a User from one DB/collection and when updating it saving it to another one by accident). In the use cases I have in mind I think you'd most of the time want all your models of the same kind to be in the same database + collection, not scattered around in different databases or collections. If you still do need to have let's say User's in two different databases/collections, I think using subclasses with different __db_config__ values would be a nicer solution and ensure you always load and save the same object from/to the same collection and database. Of course in some highly dymanic case where you'd have a lot of databases and collections, I can see that subclassing might not be an option; for example if you'd have dozens of customers and a software that would need to store User entries for each of those into different databases and you dynamically onboard new customers and thus create new databases on the fly. If you have a such need, I think firedantic has likely been more or less impossible to use so far. I was about to write that if there's a need for something like that, then I think adding the config parameter to the save/find/delete etc would be OK and that then the default value should not be set to (default), rather to None and if it's None, the value from __db_config__ should be used. But after a second thought I'm not sure that would either be a good idea, simply because if you have such dynamic use, then giving each connection a name might not be that easy, or it could lead to a lot of connections being created etc. If you have some kind of such need, I'd value if you could share some level of insight into the actual need.
• You mentioned you had issues with your changes being wiped out by running the tests the way recommended in the repo. I believe you've been developing your changes in the _sync folder primarily, as there's a lot of changes in the model.py there that are not present in the _async version. The way the repo is set up and intended to be used is so that all changes are done in the _async folder hierarchy (or then outside the _async/_sync) folders. There's some tooling in place that actually generates the _sync content from the _async version, with a fair bit of find and replace operations. This is described in the About sync and async versions of library section of the README.
• This PR also seems to be missing all the things to make the multiple connections work together with the index/TTL policy creation, which will require admin clients.
• There seems to also be a fair bit of odd spacing (as you mentioned) that should likely be taken care of by the pre-commit hooks (which I guess you didn't want to run to not wipe out all the changes in the _sync folder, in which you should not do any manual changes).
• Going to be honest and say I didn't really yet look into the changes in the README or tests due to all the issues mentioned above; I think it makes sense to first clear up the issues and only after that check those.

All in all, I'm sorry to say that it looks like there's a fair bit of work left still to get this finished. And this is a quite complex and big change to this library. I actually also had to spend quite a long time to read my (maybe too long) original suggestion to recap it etc. How do we proceed from this point forward? Do you think you have still time and energy to continue on this? I'll also need to check a bit with my colleagues how much time I could put on following up this or if I could offer to help out with some part of it or something such. But let's get back to that when I know a bit more about your status.

[mfisher29]

Hi @joakimnordling ,
Thanks so much for the detailed review and suggestions. I've been MIA on this since your message but I should have time next week to give it another go. Will keep you posted.
M

[mfisher29]

Hi @joakimnordling,
I've got things to a place I am feeling pretty good about now! I was hoping you could take a look, specifically at the /async folder changes and the examples I've added in the /integration_tests folder.
I see now the automation in place for converting async code to sync code, however I wanted to make sure everything works, and so I went with brute force for now! So I still have the changes in both places. Once I get your OK on how things are looking, I will revert the manual sync code changes and ensure that the automation does it's thing in that sense.
Curious if you like the new changes, and mainly if they align more to what you were thinking.
Have a nice weekend!
Marissa

[lietu]

Hey @mfisher29

Thanks for the PR and your update.

A few things I saw though that need to be solved:

• In the README.md you decided to add something about using a separate venv instead of just using poetry. This is unacceptable. It sounds like you just don't know how to use Poetry? If I run poetry install in the folder it does not even download firedantic from PyPI, thus it's impossible for it to run the published code because it does not know what it is. If I then run poetry run invoke test, it runs the tests fine. If I edit a file and add raise NotImplementedError() and run the tests, the tests run the code I edited. It does not modify any files I didn't ask it to. If you have issues with files being modified, it's being done by something else than poetry or by some other commands than those commands.
• The change to start_emulator.sh seems unnecessary, and no matching change was made to start_emulator.bat. Why was it done?
• You added pipx as a "prerequisite", it is not required for using or developing firedantic, it's just one of the many ways to install poetry, thus not relevant to this project. nvm is also not necessary in any way even if you chose to use it.
• You added some instructions on how to install openjdk but only specifically for mac, this seems rather pointless considering not every developer is using a mac - how would you like it if I only included instructions on how to install the tools on Arch Linux? If including any instruction it should point to the instructions from the project itself.

A reasonable "prerequisites" section would look something like

## Development

Prerequisites:

- [Node 22](https://nodejs.org/en/download)
- [Pre-commit](https://pre-commit.com/#install)
- [Python 3.10+](https://wiki.python.org/moin/BeginnersGuide/Download)
- [Poetry](https://python-poetry.org/docs/#installation)
- [Google Firebase Emulator Suite](https://firebase.google.com/docs/emulator-suite/install_and_configure#install_the_local_emulator_suite)

[lietu]

Oh and of course if you only want to run pytest poetry run pytest is fine. Personally I would even recommend adding https://pypi.org/project/pytest-watch/ to the project so you can poetry run ptw and automatically run them every time you make a change.

[mfisher29]

Hi @lietu, thanks very much for your points and suggestions.
I've incorporated your suggestions and apologies for leaving those things hanging, I intended to go back and get the tests working as it was with poetry. However my first goal was to ensure the general purpose of the changes were aligned with @lukwam and @joakimnordling's proposals for multiple clients before tidying up. But understand the PR must be clean and fully functional to be considered for future inclusion.

My problem was not with poetry. What I didn't realize is that 'invoke' calls a tasks.py file, which uses unasync.py. I was not familiar with invoke. Since any async/sync replacement text needed to be declared in the unasync.py file, any time I ran the tests via poetry, my sync folder was overwritten with async variables, etc. So to make progress when I was starting out this PR, I went only with pytest for the time being and decided to figure that out later. My bad for leaving those steps in the readme and not asking or figuring how to do this properly, until I poked around enough to notice the replacement functionality in unasync.py. I am adding that detail to the readme to clarify so others don't make the same mistake as me.

As for the prerequisites, I can't explain much there other than installing those libraries unblocked me early on, it was too long ago now to remember what exactly they resolved back in August but I see they are not needed. They are removed and overwritten with your suggestion.

I am happy to remove the integration tests as well if you all don't see a need for them. They were mainly useful for me to test how the legacy and new versions could both be used.

Otherwise, there are 2 failing unit tests for the delete method to resolve before I think this can be fully reviewed.

Thanks again.

[mfisher29]

Update:

This pull request intends to resolve issue #86 and I have created a solution based on Option 3.

Primary things to note about this solution:

• The current way of configuring a client/db via the configure() method remains in tact.
• Support for multiple configurations relies on addition of a ConfigItem with support for all of the following settings, i.e.:

        item = ConfigItem(
            name=name,
            project=normalized_project,
            database=database,
            prefix=prefix,
            client=client,
            async_client=async_client,
            admin_client=admin_client,
            async_admin_client=async_admin_client,
            credentials=credentials,
            client_info=client_info,
            client_options=client_options,
            admin_transport=admin_transport,
        )

• Upon creation of a config, both a sync and async client can be set for that config, and are created lazily when called.
• Code examples introduced in the /integration_tests folder with a corresponding README for running them and their purpose.
• Additional unit test coverage, and addition of codecov checks.
• Detailed examples and instruction updates in the README.

All tests are now passing. I am happy to remove the integration tests/examples if you don't want those in the repo.
Looking forward to your feedback

[lietu]

As just a quick general comment - having integration tests is fine (I didn't read them or test them though). I think they should be documented in the /README.md not a README.md in a sub-folder, but this can be debated. They should also not use venv, from the root you can do poetry run python integration-tests/foo.py or poetry run shell and then cd integration-tests etc.

[mfisher29]

Hi all,

Thanks a lot for your feedback and suggestions. I'm looking forward to hearing back and getting it packaged up hopefully soon :)

Happy new year!