[MSD benchmark] Annotations: Split cleanup into separate queries and deletes to avoid deadlocks on MySQL

[khaliqgant]

MSD benchmark PR recreated from Martian Code Review Bench.

Source PR: grafana#80329
Dataset: grafana
Original title: Annotations: Split cleanup into separate queries and deletes to avoid deadlocks on MySQL

What is this feature?

Writes to annotations are sometimes rejected on MySQL due to a recurring deadlock with the annotation cleanup job.
Details here: grafana#64979

Why do we need this feature?

The deadlock seems to be a result of the subquery, under normal circumstances a plain insert and delete should not lock, and there are no extended transactions at play.

This PR splits the subquery into a separate SQL statement. These statements do not share a transaction, and therefore allow locks to flush in between.

Unfortunately we could not reduce this to a single DELETE statement, due to the batching - DELETE LIMIT is not supported on all databases without needing to reintroduce the subquery. The IDs loaded into memory are of bounded size due to the batch size configuration.

Which issue(s) does this PR fix?:

grafana#64979

Special notes for your reviewer:

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.