[core] Add row filter & column masking support for table read #7034
Conversation
| TableQueryAuth queryAuth = | ||
| ((FileStoreTable) table).catalogEnvironment().tableQueryAuth(options); | ||
| TableQueryAuthResult authResult = | ||
| queryAuth.auth(readType == null ? null : readType.getFieldNames()); |
There was a problem hiding this comment.
New Read will invoked in every works. We should not let so many works to do RPC to metastore.
I think maybe we can introduce QueryAuthSplit with filter and masking. And here wrapping new TableRead to do filter and masking.
| public void testColumnMasking() { | ||
| spark.sql( | ||
| "CREATE TABLE t_column_masking (id INT, secret STRING, email STRING, phone STRING) TBLPROPERTIES" | ||
| + " ('bucket'='1', 'bucket-key'='id', 'file.format'='avro', 'query-auth.enabled'='true')"); |
There was a problem hiding this comment.
Don't need to test bucketed table. Just no bucket.
There was a problem hiding this comment.
Don't need to test bucketed table. Just no bucket.
done
|
|
||
| Map<String, Transform> columnMasking = new HashMap<>(); | ||
| columnMasking.put("secret", maskTransform); | ||
| restCatalogServer.setColumnMaskingAuth( |
There was a problem hiding this comment.
Also add case for row filtering.
There was a problem hiding this comment.
Also add case for row filtering.
done
plusplusjiajia
force-pushed
the
column-masking
branch
2 times, most recently
from
7bc811f to
5cc62fd
Compare
plusplusjiajia
changed the title
|
|
||
| /** Input splits. Needed by most batch computation engines. */ | ||
| public class DataSplit implements Split { | ||
| public class DataSplit implements QueryAuthSplit { |
There was a problem hiding this comment.
Don't modify DataSplit, just create a new class to wrap it.
DataSplit has a heavy historical burden, and we need to ensure its compatibility. Its serialization protocol is related to multiple languages.
There was a problem hiding this comment.
Don't modify DataSplit, just create a new class to wrap it.
DataSplit has a heavy historical burden, and we need to ensure its compatibility. Its serialization protocol is related to multiple languages.
Got it!
plusplusjiajia
force-pushed
the
column-masking
branch
from
dc73579 to
942da63
Compare
|
+1 |
2 participants
Purpose
Add row filter & column masking support for table read