feat(keeper): implement database migration and archival strategy (#438)

[Danielodingz]

Closes #438

feat(keeper): Database Migration and Archival Strategy (#438)

Summary

Implements the core architecture for issue #438 — a robust system for managing massive execution logs, with automated archiving to cold storage and seamless queryability across both hot and cold storage layers.

As agreed with the maintainer, the implementation is delivered as a self-contained module (keeper/src/archival-strategy/) ready to be wired into the existing infrastructure.

---

Changes

New module: keeper/src/archival-strategy/

File	Description
errorTracker.js	Circuit breaker + error aggregation. Trips after 5 consecutive failures, halting all downstream operations to prevent cascading failures.
migrator.js	Fault-tolerant schema migrations using BEGIN TRANSACTION / COMMIT / ROLLBACK. Safe to run with zero downtime.
archiver.js	Batched archival pipeline — fetches old logs from the hot DB, writes to cold storage, then deletes from hot DB. Configurable batchSize and daysOld.
queryEngine.js	Unified query interface that transparently routes requests to hot DB, cold storage, or both depending on the requested time window.
index.js	Barrel export for clean integration.

New tests: keeper/__tests__/archival-strategy/

• errorTracker.test.js
• migrator.test.js
• archiver.test.js
• queryEngine.test.js
• index.test.js

New documentation: keeper/ARCHIVAL_STRATEGY_DOCS.md

---

Test Coverage

Test Suites: 5 passed, 5 total
Tests:       23 passed, 23 total

File             | % Stmts | % Branch | % Funcs | % Lines
-----------------|---------|----------|---------|--------
All files        |     100 |      100 |     100 |     100

Exceeds the >90% coverage requirement across all metrics.

---

Acceptance Criteria

• Feature implemented according to requirements
• Unit and integration tests passing (23/23)
• Security review completed (documented in ARCHIVAL_STRATEGY_DOCS.md)
• Comprehensive documentation written

---

Integration Notes for Maintainer

To wire this into the live system:

1. Replace the cold storage mock in archiver.js with the actual cloud storage client (e.g. AWS S3, GCS).
2. Schedule archiver.archiveOldLogs() via the existing cron/background worker.
3. Route existing log query endpoints through queryEngine.fetchLogs().

[drips-wave]

@Danielodingz Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Danielodingz]

Hello @ayomideadeniran please review and merge. Thank you

[ACodehunter]

Pr under review