Partitioned events by engobi · Pull Request #1 · engobi/eventstore

engobi · 2026-01-29T10:21:28Z

Allow EventStore to work with an events table partitioned and use pg_partman (a PostgreSQL extension) to automaticaly manage partitions.

…vent_id unicity

…events table.

…sure there is only one SQL statement per init function.

Update getting started documentation

config/config.exs

grtcdr · 2026-01-29T10:52:09Z

config/dev.exs

+  schema: "event_store",
+  column_data_type: "jsonb", 
+  partitioned_events: true,
+  use_pg_partman: true


Event store supports, according to the docs, Postgres 9.5 an up.

I spun up Postgres 9.5 in a container with:

podman run --name some-postgres -e POSTGRES_PASSWORD=pass -e POSTGRES_USER=admin -d postgres:9.5.21 podman exec -it some-postgres bash psql -h localhost -U admin

And ran select * from pg_available_extensions; and pgpartman was not part of the list.

I think use_pg_partman and partitioned_events should therefore be set to false in every configuration as it may not be available for every version that event_store supports.

This obviously impacts the test suite's actual scope (if you disable the option, you wouldn't be testing how partitioning affects the validity of the code). I think we should discuss a testing strategy that aligns with the official library's own support goals.

Proposition 1: Rerun the whole test suite with the option on and off

Proposition 2: Create new tests for the enabled option

I'm personally more inclined to go with the first (because it means you can verifiably say the configuration option does not impact the original implementation in any way).

If the idea is to use this fork in Ublo directly before contributing the changes upstream, I think we can pick this conversation at a later time.

I agree with you partitioned_events and use_pg_partman should be set to false. I have made the change.

This obviously impacts the test suite's actual scope (if you disable the option, you wouldn't be testing how partitioning affects the validity of the code). I think we should discuss a testing strategy that aligns with the official library's own support goals.

Proposition 1: Rerun the whole test suite with the option on and off

Proposition 2: Create new tests for the enabled option

I'm personally more inclined to go with the first (because it means you can verifiably say the configuration option does not impact the original implementation in any way).

In the current implementation, we have 3 possible combinations:

partitioned_events and pg_partman are both false, which means it should work as the original implementation

partitioned_events is true and pg_partman is false, which means the events table is partitioned but we don't use pg_partman to manage partitions

partitioned_events and pg_partman are both true, which means pg_partman must be installed, the events table is partitioned and partitions are managed by pg_partman

For testing, I chose an approach where the same test suite works successfully accros all the configuration. Instead of having one set of tests per configuration combination, we use a single test suite that is compatible with all configuration combinations.

For testing, I chose an approach where the same test suite works successfully accros all the configuration.

Does that mean you've configured the test suite to run with each combination?

It means that if you change the combination and you run test suite, the tests will create events table according what you configured and test with it. You don't have do change any thing in the test suit, all the tests work with all the possible combinations. If you never need have a partitioned events table, you never need to test it. I applied my change at the lowest level possible so it's transparent for the test. If conf parameter partitioned_events is set, the test will automaticaly pass the parameter to the eventstore functions which need it.

Okay, I understand that a little bit better. One more question: Wouldn't you have to constantly turn the feature on and off to ensure the codebase is working correctly for all configuration options following a specific change? I was thinking we'd automate that (we could run it asynchronously if the partitioned database can run concurrently with the non-partitioned database, which means we wouldn't affect the suite's performance by all that much).

config/test.exs

lib/event_store/sql/reset.ex

lib/event_store/streams/stream.ex

grtcdr · 2026-01-29T15:14:27Z

I'm going to have a second look tomorrow morning.

grtcdr · 2026-01-30T09:30:54Z

config/bench.exs

  pool_size: 10,
-  serializer: EventStore.TermSerializer
+  serializer: EventStore.TermSerializer,
+  partitioned_events: true,  # Default false, set to true if you want a partioned events table


Comment longer reflects reality:

Suggested change

partitioned_events: true, # Default false, set to true if you want a partioned events table

partitioned_events: true,

grtcdr · 2026-01-30T09:32:52Z

lib/event_store/sql/statements.ex


    @external_resource file

+    #EEx.function_from_file(:def, fun, file, args ++ [:partitioned], engine: EventStore.EExIOListEngine)


Suggested change

#EEx.function_from_file(:def, fun, file, args ++ [:partitioned], engine: EventStore.EExIOListEngine)

grtcdr · 2026-01-30T09:33:59Z

lib/event_store/sql/statements.ex

        {:soft_delete_stream, [:schema]},
-        {:hard_delete_stream, [:schema]},
+        {:hard_delete_stream, [:schema, :partitioned]},


soft_delete_stream doesn't take :partitioned as an argument, is that intentional?

Yes! soft_delete_stream just update the line and set deleted_at field at NOW() when hard_delete_stream delete rows form several tables including events_root table if we have a partitioned events table.

grtcdr · 2026-01-30T09:36:21Z

test/storage/read_events_test.exs

    %{conn: conn, schema: schema} = context

-    Appender.append(conn, stream_id, recorded_events, schema: schema)
+    partitioned = Application.get_env(:eventstore, EventStore)[:partitioned_events] || false


Shouldn't this option come from context?

grtcdr · 2026-01-30T09:36:30Z

test/streams/all_stream_test.exs

    events = EventFactory.create_events(3)

-    :ok = Stream.append_to_stream(conn, stream_uuid, 0, events, opts)
+    #IO.inspect(opts)


Suggested change

#IO.inspect(opts)

grtcdr · 2026-02-02T09:35:59Z

lib/event_store/storage/appender.ex

    {expected_version, opts} = Keyword.pop(opts, :expected_version)
    {created_at, opts} = Keyword.pop(opts, :created_at_override)
+    partitioned = Keyword.get(opts, :partitioned_events, false)
+    {debug, opts} = Keyword.pop(opts, :debug)


You should use Logger.level/0 to retrieve the level and print your messages instead of passing debug to the keyword list.

grtcdr · 2026-02-02T09:37:43Z

lib/event_store/streams/stream.ex

      when length(events) < 1000 do
    {serializer, new_opts} = Keyword.pop(opts, :serializer)
-
+        


Suggested change

grtcdr · 2026-02-02T09:47:42Z

lib/event_store/sql/init.ex

+      CREATE TRIGGER no_delete_events
+      BEFORE DELETE ON #{schema}.events
+      FOR EACH STATEMENT
+      EXECUTE PROCEDURE #{schema}.event_store_delete('Cannot delete events');


Suggested change

CREATE TRIGGER no_delete_events

BEFORE DELETE ON #{schema}.events

FOR EACH STATEMENT

EXECUTE PROCEDURE #{schema}.event_store_delete('Cannot delete events');

CREATE TRIGGER no_delete_events

BEFORE DELETE ON #{schema}.events

FOR EACH STATEMENT

EXECUTE PROCEDURE #{schema}.event_store_delete('Cannot delete events')

engobi and others added 8 commits December 29, 2025 22:52

Init events_root table and events partitioned table

6ebd676

Adding tables creation functions in init

3a22c8f

Inserting in events_root table before inserting in events to ensure e…

a1ce23c

…vent_id unicity

Add new config parameter to EventStore to enable using a partitioned …

0822cfa

…events table.

Add :partitioned parameter to all functions that insert events and en…

e2577fe

…sure there is only one SQL statement per init function.

Fix all tests

bd1fae3

Fix bench

f1c221b

Update Getting Started.md

5a900df

Update getting started documentation

engobi assigned engobi and grtcdr Jan 29, 2026