Help regarding GetTableLayoutResponse partition serialization

[sanatdeshpande1]

Hi there, I came across your repository as I was trying to find how exactly the serialization for the GetTableLayoutResponse partitions works. I am currently working on a similar Athena connector using Javascript and Apache Arrow JS and I would really appreciate if you could help me out.

This is the code that you have in your models.py for GetTableLayoutResponse.

batch = pa.RecordBatch.from_arrays(data, list(partition_keys))
return {
    "aId": str(uuid4()),
    "schema": AthenaSDKUtils.encode_pyarrow_object(batch.schema),
    "records": AthenaSDKUtils.encode_pyarrow_object(batch)
}

Now, pyarrow package in Python has this convenient method serialize() which is called by your encode_pyarrow_object method and which serializes any pyarrow_object. However, that is not the case in Apache Arrow JS. So, I was wondering if you have any idea about how the batch.schema and batch get serialized and converted into Base64 behind the scenes.

I would really appreciate if you can provide any kind of help here. Thanks!