Skip to content

feat(dag): Implement DAG compaction to bound storage growth #2026

@xilosada

Description

@xilosada

Problem

DAG history grows unbounded:

// All deltas persisted forever in ContextDagDelta
// parent_hashes pruned at 10k but deltas remain in DB

Impact:

  • Storage grows linearly with transaction count
  • Old deltas never cleaned up
  • Sync sends full history even when snapshot would be smaller

Current State

Checkpoints are partially supported:

// crates/node/src/delta_store.rs:1139
let checkpoint = CausalDelta::checkpoint(head_id, boundary_root_hash);

But no compaction logic exists to:

  1. Determine when to checkpoint
  2. Prune deltas older than checkpoint
  3. Handle sync requests for pruned deltas

Proposed Solution

  1. Add compaction configuration:
pub struct DagCompactionConfig {
    /// Minimum deltas before compaction eligible
    pub min_deltas_before_compact: usize,  // Default: 10000
    /// Keep recent history for delta sync
    pub retain_recent_count: usize,        // Default: 1000
    /// Compaction interval
    pub check_interval: Duration,          // Default: 1 hour
}
  1. Implement compaction logic that:
    • Creates checkpoint at boundary
    • Prunes deltas older than checkpoint
    • Handles sync for pruned history (offer snapshot)

Acceptance Criteria

  • Compaction runs periodically based on config
  • Creates checkpoint delta at compaction boundary
  • Prunes deltas older than checkpoint from DB
  • Retains configurable recent history
  • Sync falls back to snapshot for pruned deltas
  • Compaction metrics exported

Risks & Mitigations

Risk Mitigation
Data loss if checkpoint fails Transactional checkpoint + prune
Slow nodes need old deltas Keep generous retain_recent_count
Checkpoint verification Verify state hash matches before prune

Priority: P3 (Medium Impact, High Effort)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions