Currently we create one Hikari connection pool per sqlite file. Each connection pool opens three threads (housekeeper, connection adder, connection closer) for the rest of the process. While this is a non-issue with few gpkg files this can quickly consume large amounts of memory just for the background threads with hundreds of geopackage files. I'm not sure if we even benefit from connection pooling with geopackages, we should investigate the performance effects of 1) simply opening and closing the DB connection (file) per query 2) keeping single shared connection open per gpkg (this might be problematic with our stream-it-all ideology) 3) lazily opened threadlocal connection per gpkg.
Currently we create one Hikari connection pool per sqlite file. Each connection pool opens three threads (housekeeper, connection adder, connection closer) for the rest of the process. While this is a non-issue with few gpkg files this can quickly consume large amounts of memory just for the background threads with hundreds of geopackage files. I'm not sure if we even benefit from connection pooling with geopackages, we should investigate the performance effects of 1) simply opening and closing the DB connection (file) per query 2) keeping single shared connection open per gpkg (this might be problematic with our stream-it-all ideology) 3) lazily opened threadlocal connection per gpkg.