Treat pandas as an optional dependency (#489)

[gs11]

What type of PR is this?

• Refactor
• Feature
• Bug Fix
• Other

Description

This naive PR changes pandas to be an optional dependency - just like pyarrow.
Tests fail but seemingly due to unrelated things.

How is this tested?

• Unit tests
• E2E Tests
• Manually
• N/A

Related Tickets & Documents

#489

[gs11]

Any chance to get this reviewed?
@deeksha-db @samikshya-db @jprakash-db @yunbodeng-db @jackyhu-db @benc-db

[FBruzzesi]

Hey @gs11 thanks for addressing this, I just realized about your PR.
To close #489 you should make sure to avoid importing pandas as top level import in src/databricks/sql/client.py. That would have been my strategy.

I really hope this gets through eventually

[gs11]

To close #489 you should make sure to avoid importing pandas as top level import in src/databricks/sql/client.py. That would have been my strategy.

Thanks! @FBruzzesi I realize this change wasn't pushed properly to my branch. Updated it as well as rebased to resolve the poetry.lock conflict issues due to upstream being updated.

[dtroberts]

Hello, I'm just wondering if it's possible to get any movement on this? The large pandas dependency is a blocker for my project.

[gs11]

Any chance of getting this reviewed?
@deeksha-db
@samikshya-db
@jprakash-db
@yunbodeng-db
@jackyhu-db
@benc-db

[ottovis]

Is there any way to accelerate this PR? The underlaying issue was opened almost a year ago, and this PR nearing half a year and the issue is still quite blocking for being able to use databricks-sql-python at all.

If there is any way for me to contribute, lmk.

[snowman2]

@gs11 it appears that there are conflicts. If I had a guess, it would be the lock file needs to be regenerated.

[snowman2]

Suggested change

	self.disable_pandas = False if pandas is None else kwargs.get("_disable_pandas", False)
	self.disable_pandas = True if pandas is None else kwargs.get("_disable_pandas", False)

I had that backwards. If panda is not installed, it should be disabled.

[snowman2]

Pandas is also imported here:

databricks-sql-python/src/databricks/sql/result_set.py

Line 7 in 4b7df5b

import pandas

[snowman2]

I imagine they would want a warning message similar to the one for pyarrow:

databricks-sql-python/src/databricks/sql/client.py

Lines 81 to 86 in 4b7df5b

	if pyarrow is None:
	logger.warning(
	"[WARN] pyarrow is not installed by default since databricks-sql-connector 4.0.0,"
	"any arrow specific api (e.g. fetchmany_arrow) and cloud fetch will be disabled."
	"If you need these features, please run pip install pyarrow or pip install databricks-sql-connector[pyarrow] to install"
	)

[gs11]

Pandas is also imported here:

databricks-sql-python/src/databricks/sql/result_set.py

Line 7 in 4b7df5b

import pandas

I imagine they would want a warning message similar to the one for pyarrow:

databricks-sql-python/src/databricks/sql/client.py

Lines 81 to 86 in 4b7df5b

	if pyarrow is None:
	logger.warning(
	"[WARN] pyarrow is not installed by default since databricks-sql-connector 4.0.0,"
	"any arrow specific api (e.g. fetchmany_arrow) and cloud fetch will be disabled."
	"If you need these features, please run pip install pyarrow or pip install databricks-sql-connector[pyarrow] to install"
	)

Thanks, updated.

[snowman2]

Are there any tests in the CI run without pandas installed? If not, it would be important to add for future stability.

[gs11]

Are there any tests in the CI run without pandas installed? If not, it would be important to add for future stability.

I can't comment on the effectiveness but there's

databricks-sql-python/tests/unit/test_client.py

Line 482 in 4b7df5b

def test_disable_pandas_respected(self, mock_thrift_backend_class):

[snowman2]

Are there any tests in the CI run without pandas installed? If not, it would be important to add for future stability.

I can't comment on the effectiveness but there's

databricks-sql-python/tests/unit/test_client.py

Line 482 in 4b7df5b

def test_disable_pandas_respected(self, mock_thrift_backend_class):

That's helpful. Did you check the CI files to see if there is a test run without optional dependencies installed?

[jprakash-db]

@gs11 Have started the tests to run, will review after the tests have passed

[gs11]

That's helpful. Did you check the CI files to see if there is a test run without optional dependencies installed?

No, there seems to be a single set of dependencies used for tests. https://github.com/databricks/databricks-sql-python/blob/main/.github/workflows/code-coverage.yml#L52-L62

[gs11]

Fixed the linting error but there are plenty of tests failing on unrelated issues 😕

[jprakash-db]

@msrathore-db Can you look at this? Looks like some core Os libraries for kerberos are missing in the actions ubuntu images

[deepyaman]

@jprakash-db @msrathore-db Can this be prioritized? It seems like a no-brainer change, given you already support disable_pandas on the connection (and the use is so minimal). This is especially important given the library doesn't yet allow pandas 3 (also should be easy to support, given you only use pandas in one place?), so it's unnecessarily blocking pandas upgrades.