Problem using ACLs for authentication

[quartox]

Original issue:
dask/dask#2852

I am tagging @bdrosen96 at the recommendation of @martindurant

Summary:

token = hdfs3.HDFileSystem().delegate_token(user='jlord')
hdfs = hdfs3.HDFileSystem(token=token)
hdfs.ls('/user/jlord') # everything works
hdfs.ls('/user/hive/warehouse/database_name.db/table_name') # File Not Found Error Below

The error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-35-1412502a9b50> in <module>()
----> 1 hdfs.ls('/user/hive/warehouse/database_name.db/table_name')

/nas/isg_prodops_work/jlord/conda/envs/dask/lib/python3.6/site-packages/hdfs3/core.py in ls(self, path, detail)
    333         """
    334         if not self.exists(path):
--> 335             raise FileNotFoundError(path)
    336         num = ctypes.c_int(0)
    337         fi = _lib.hdfsListDirectory(self._handle, ensure_bytes(path),

FileNotFoundError: /user/hive/warehouse/database_name.db/table_name

I can use hdfs.ls on any directory that I would expect to have access to except for the hive warehouse which is protected using Sentry. I have been told that Sentry uses ACLs to manage permissions to files/directories in the hive warehouse.

Looking at the ACLs yields:

$ hadoop fs -getfacl /user/hive/warehouse/database_name.db/table_name
# file: /user/hive/warehouse/database_name.db/table_name
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group::---
group:hive:rwx
group:hdpruff:rwx
group:hdpqra:r-x
mask::rwx
other::--x

I do not have group hive (which I assume is managed by sentry), but I do have group hdpruff and hdpqra.

hadoop fs -ls can find that directory and I can use spark to read the underlying parquet file for that hive table.

[quartox]

Calling hdfs3._lib.hdfsGetLastError() yields

InvalidParameter: Invalid parameter: cannot convert 4601 to "Permission"

[quartox]

The problem is the if statement here

libhdfs3-downstream/libhdfs3/src/client/Permission.cpp

Line 33 in 18ae956

if (!isFileEncryption && mode >> 10) {

The number 4601 is binary 0b1000111111001. The rightmost 9 bits still appear to correspond to the permissions. user: rwx, group: rwx, and other: --x. The problem is the extra leftmost bit which is not expected by Permission.cpp. I cannot find any documentation that explains why that bit is 1.

I was able to comment out the if statement, rebuild the library and re-run hdfs.ls('/user/hive/warehouse/database_name.db/table_name') which now returns a result. I don't know why this if statement exists, however, so I do not know if removing that statement will have unintended consequences (and it probably will).

[martindurant]

Good detective work, @quartox !
Since the further code doesn't use any of the bits other than the right 9 as you describe, and 13 for encryption, I'd be tempted to ignore the if statement too. @bdrosen96 any thoughts on this?

[bdrosen96]

I believe that we have this addressed in my fork as well as the uncommitted branch (not sure what is happenning with that?):

https://github.com/martindurant/libhdfs3-downstream/blob/brett/kms/libhdfs3/src/client/Permission.cpp#L32-L47

You could ignore the encryption bit for doing directory listing, but will have problems reading file contents if it is encrypted without using a version that handles the encrypt/decrypt stuff using KMS.

[martindurant]

Re: what's the status of the unmerged branch: that's where we were seeing problems with the implementation of the proxy/impersonation, i.e., secure things would work only is the user were configured as a proxy entity, whereas only the hdfs user should normally be.
Conversely, using the current master with your libgsasl branch seemed enough to get "private" security working (but not KMS, of course). I am not currently assigned to work on any of this, hence the pause in progress.

[martindurant]

@quartox , you are most welcome to try the PR branch too, and see if it works for you, especially in conjunction with various secure configurations.

[quartox]

Is there a way that I can build the conda package locally or can you try emailing me the zipped conda build of the PR branch? It is annoying to build in each of my environments but if I could just do a local conda install after building and trick hdfs3 into thinking that it is a reasonable dependency, then I can work with that for now.

[martindurant]

You can download the recipe from conda-forge/hdfs3-feedstock, and change the source to point to your local git clone/branch. If you run conda-build on this, it will make a real conda package in some temporary location (which is echoed to the console) which you can directly conda-install from. Using a local file will not update dependencies, which is probably what you want.