Skip to content

[Bug] We should change the default manifest format to parquet. #7029

@LinMingQiang

Description

@LinMingQiang

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

Manifest compression defaults to zstd, but the avro format still attempts to load snappy.

in the class `org.apache.avro.file.CodecFactory` we find :
    static {
        addCodec("null", nullCodec());
        addCodec("deflate", deflateCodec(-1));
        addCodec("bzip2", bzip2Codec());
        addCodec("xz", xzCodec(6));
        addCodec("zstandard", zstandardCodec(3, false));
        addCodec("snappy", snappyCodec());
    }

Could not load 'org-xerial-snappy.properties' from classpath: java.lang.NullPointerException
java.lang.NullPointerException
at org.xerial.snappy.SnappyLoader.getRootClassLoader(SnappyLoader.java:147)
at org.xerial.snappy.SnappyLoader.injectSnappyNativeLoader(SnappyLoader.java:256)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:227)
at org.xerial.snappy.Snappy.(Snappy.java:48)
at org.apache.paimon.shade.org.apache.avro.file.SnappyCodec$Option.(SnappyCodec.java:35)
at org.apache.paimon.shade.org.apache.avro.file.CodecFactory.snappyCodec(CodecFactory.java:69)
at org.apache.paimon.shade.org.apache.avro.file.CodecFactory.(CodecFactory.java:140)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.resolveCodec(DataFileStream.java:158)
at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:144)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:143)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.(DataFileReader.java:134)
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:74)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:84)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:70)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.(AvroBulkFormat.java:57)
at org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:54)
at org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:130)
at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:180)
at org.apache.paimon.utils.ObjectsCache.readSegments(ObjectsCache.java:129)
at org.apache.paimon.utils.ObjectsCache.read(ObjectsCache.java:93)
at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:149)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:134)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:105)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:101)
at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:86)
at org.apache.paimon.operation.ManifestsReader.readManifests(ManifestsReader.java:99)
at org.apache.paimon.operation.ManifestsReader.read(ManifestsReader.java:85)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:393)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:233)
at org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:296)

Compute Engine

master

Minimal reproduce step

none

What doesn't meet your expectations?

none

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions