For some use cases of a write ahead log, I'd like to keep an in-memory representation (which gets recovered) of the contents of the log. When a checkpoint is made, I can drop the check-pointed information from memory since it is now part of the checkpoint.
One difficulty, as far as I can tell, is that the checkpoint method is called in a separate thread -- this means the following could happen:
- Commit some data
A to the log
- Initiate a checkpoint
- Commit some data
B to the log
- Actually perform the checkpoint
The problem I have is that at that point, the contents of A and B are in the in-memory representaiton, while I only want to checkpoint (and potentially drop) A. For my use case, I'm fine keeping the in-memory representation behind a Mutex (there should be only one writer to the log), which also simplifies a lot of the logic.
There are a few things that could help me:
- If the result of committing data indicated that a checkpoint would be taken. I think this would only help if there was a lock, but then I could detect that a checkpoint would be taken after
A, and arrange to have a separate in-memory representation started for data added after that.
- If there was a "pre-checkpoint" hook that was called before other commits were allowed. Again, this would allow me to split the in-memory representation for B.
For some use cases of a write ahead log, I'd like to keep an in-memory representation (which gets recovered) of the contents of the log. When a checkpoint is made, I can drop the check-pointed information from memory since it is now part of the checkpoint.
One difficulty, as far as I can tell, is that the checkpoint method is called in a separate thread -- this means the following could happen:
Ato the logBto the logThe problem I have is that at that point, the contents of
AandBare in the in-memory representaiton, while I only want to checkpoint (and potentially drop)A. For my use case, I'm fine keeping the in-memory representation behind a Mutex (there should be only one writer to the log), which also simplifies a lot of the logic.There are a few things that could help me:
A, and arrange to have a separate in-memory representation started for data added after that.