[SPARK-55242][PYSPARK] Handle np.ndarray elements in list-valued columns when converting from pandas

[azmatsiddique]

What changes were proposed in this pull request?
In DataTypeOps.prepare() (python/pyspark/pandas/data_type_ops/base.py),
added a pre-processing step that detects object-dtype pandas Series whose
elements are np.ndarray objects and converts them to plain Python lists
via .tolist() before the existing col.replace({np.nan: None}) call.

This is a targeted, minimal fix: the ndarray-to-list conversion only fires
when all three conditions hold:

1. The Series dtype is object
2. The Series is non-empty
3. The first non-null element is a np.ndarray

Why are the changes needed?
In pandas 3, when a DataFrame column is created from a list-of-lists
(e.g. [[e] for e in ...]), each element is stored internally as a
np.ndarray object rather than a plain Python list.

DataTypeOps.prepare() calls col.replace({np.nan: None}), which
internally compares every element with np.nan using ==. Comparing a
np.ndarray with a scalar via == returns an array, not a bool, so
pandas raises:

ValueError: The truth value of an array is ambiguous.
Use a.any() or a.all()

This makes ps.from_pandas() (and ps.DataFrame(), ps.from_pandas(series),
etc.) crash whenever the input contains list-valued columns in a pandas 3
environment.

Reproducer:
import numpy as np
import pandas as pd
import pyspark.pandas as ps

pdf = pd.DataFrame(
    {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9],
     "b": [[e] for e in [4, 5, 6, 3, 2, 1, 0, 0, 0]]},
    index=np.random.rand(9),
)
psdf = ps.from_pandas(pdf)  # raises ValueError on pandas 3

Does this PR introduce any user-facing change?
Yes. This is a bug fix.

Before: ps.from_pandas(pdf) with a list-valued column raised
ValueError: The truth value of an array is ambiguous on pandas 3.

After: the call succeeds and the DataFrame is created correctly, with
the list column properly inferred as ArrayType in the Spark schema.

This affects pandas 3 users only; the fix is backward-compatible with
earlier pandas versions.
How was this patch tested?
Added test_from_pandas_with_np_array_elements in
python/pyspark/pandas/tests/data_type_ops/test_complex_ops.py.

The test reproduces the exact scenario from SPARK-55242:

• Creates a pandas DataFrame with integer column "a" and a
  list-valued column "b" (one list per row) with a float index.
• Calls ps.from_pandas(pdf) — this previously raised ValueError.
• Asserts that column "a" round-trips correctly.
• Asserts that column "b" has the expected number of rows.
  Was this patch authored or co-authored using generative AI tooling?
  No