[Bug] manifest.delete-file-drop-stats causes empty stats after compaction

[mao-liu]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

1.3.1

Compute Engine

Flink, Java API

Minimal reproduce step

• Create a PK table with metadata.stats-mode = "full" and metadata.delete-file-drop-stats = "true"
• Write a few records
• Inspect manifest: manifest entries contain stats
• Run compaction
• Inspect manifest: manifest entries no longer contain stats

Compaction create commits that are DELETE and ADD on the same data file, changing the "level" of the data file only.
With manifest.delete-file-drop-stats enabled, this means stats are empty after compaction

What doesn't meet your expectations?

Intuitively, metadata.delete-file-drop-stats should drop stats when data is deleted, and should keep stats on new writes.

Since compaction is very important for PK tables, there should be a way to persist stats through this kind of compaction commits for PK tables.

Alternatively, documentation should be made clearer to explain this is the behaviour for PK tables, and that metadata.delete-file-drop-stats should not be enabled if stats are needed

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!