Support `parquet` columnar format in the `aws_s3` sink

[binarylogic]

Similar to #1373 we should support the Parquet format. The parquet format is a columnar format that enables faster and more efficient data access schemes such as column selection and indexing.

Implementation

Unfortunately, I do not have deep experience with this format as I do with ORC, but like everything else, we should start very simple. Fortunately, there appears to be a Rust library that supports basic writing of this data.

[LucioFranco]

One issue with that parquet library is that it requires nightly rust https://github.com/sunchao/parquet-rs/blob/master/src/lib.rs#L123 and https://github.com/apache/arrow/blob/master/rust/parquet/src/lib.rs#L18, we can't use these features because we are on the stable compiler.

[binarylogic]

Any idea what the timeline is for those to be on stable?

[LucioFranco]

Well really specialization is the last one on the updated apache/arrow but that feature has been in progress since 2016, so we will likely not see it any time soon. Our best bet would be to attempt to remove that feature from that library but I am not sure how hard that would be or if that is even possible.

[camerondavison]

I think that it still requires nightly, but just throwing into the mix https://github.com/apache/arrow/tree/master/rust/parquet

[davemt]

Progress in Apache Arrow project:

• PR: Remove Rust specialization - ARROW-10636: [Rust][Parquet] Switch to Rust Stable by removing specialization in parquet apache/arrow#8698
• JIRA: Support stable Rust - https://issues.apache.org/jira/browse/ARROW-6717
• JIRA (comment): re. remaining dependency other than specialization - https://issues.apache.org/jira/browse/ARROW-6718?focusedCommentId=17002842&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17002842

[alexgavrisco]

IIRC Apache Arrow now works on stable Rust 🎉 (unless SIMD feature is used). Would it be possible to re-evaluate this one?
In case the team doesn't have bandwidth, but somebody can help with review and some pointers, I can look into it.

[jszwedko]

Hi @Alexx-G . This hasn't been scheduled yet, but you wanted to pick it up, I'd be happy to advise/review!

[alexgavrisco]

I've checked the parquet crate it seems to have some basic support for writing data.
@binarylogic @jszwedko I see that the ORC issue has some additional context, do you have any insights/pointers for this one? Especially about schema and the changes around the current encoding logic.
I'll try to run some tests with Athena+S3 and parquet to see whether I can get an MVP.

[CorbyAsCode]

Is there an update on this? I'm very interested in JSON input to parquet output to S3.