[Data] Add glob pattern support to read_xxx APIs

[jiangxt2]

Description

Add glob pattern support (*, **, ?, [...]) to all read_xxx APIs in Ray Data, enabling ray.data.read_parquet("s3://bucket/*.parquet") to work natively.

The implementation injects glob expansion into _resolve_paths_and_filesystem() — the single resolution path shared by V1 and V2 datasources — so all read_xxx callers benefit automatically.

Key design decisions:

• Uses pathlib.Path.glob() for local paths and FileSelector listing + segment-aware matching for cloud paths (S3, GCS, etc.)
• * matches within a single path segment; ** spans multiple segments (dynamic programming matcher)
• Bucket-level globs (s3://bucket-*/*.parquet) are explicitly rejected with a clear error
• URI query strings (?X-Amz-Signature=...) are not treated as glob characters

Related issues

Fixes #59304

Additional information

Files changed (8):

• path_util.py: New helpers _has_glob_chars, _split_glob_base, _glob_match_path, _expand_glob; inject expansion into _resolve_paths_and_filesystem
• file_based_datasource.py: Enable expand_globs=True, fix ignore_missing_paths for empty glob results
• parquet_datasource.py / parquet_datasource_v2.py / file_indexer.py: Enable glob expansion
• read_api.py: Add ignore_missing_paths to read_parquet, add glob examples to all read_xxx docstrings

Tests (31 total):

• test_glob_patterns.py (16): Integration tests for local + S3 (moto), V1/V2 parameterization
• test_path_util_glob.py (15): Unit tests for pure helper functions

[gemini-code-assist]

Code Review

This pull request introduces robust glob pattern support (including recursive and segment-aware matching) across Ray Data readers, resolving patterns before path resolution. The review feedback highlights a few important issues: a variable shadowing bug in _resolve_paths_and_filesystem that overwrites the shared filesystem parameter across loop iterations; OS-dependent case-sensitivity bugs on Windows due to using fnmatch.fnmatch instead of fnmatch.fnmatchcase for cloud paths; a potential unpacking crash when paths is empty; and a risk of AttributeError when directly accessing datasource.ignore_missing_paths instead of using getattr.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Reassigning the filesystem variable inside the loop (filesystem, _ = _resolve_single_path_with_fallback(...)) shadows and overwrites the input filesystem parameter for all subsequent iterations. If the input list contains multiple paths that require different filesystems (or if the first is a glob and the second is a non-glob), subsequent paths will incorrectly use the filesystem resolved from the first path.

To fix this, track the resolved filesystem in a separate variable (e.g., resolved_filesystem) and use a local resolved_fs inside the loop to avoid mutating the shared filesystem state.

    resolved_filesystem = filesystem
    resolved_paths = []
    for path in paths:
        # Expand glob patterns before resolving individual paths.
        # Only enabled when callers explicitly opt in (expand_globs=True),
        # since this function is also used by write, download, and
        # partition paths where glob expansion is undesirable.
        if expand_globs and _has_glob_chars(path):
            # Resolve relative paths to absolute before glob expansion
            # to ensure consistency across processes (e.g., Ray workers).
            # Only for local paths — cloud URIs (s3://, gs://, etc.) are
            # already absolute; os.path.isabs misidentifies them as relative.
            parsed = urlparse(path)
            is_cloud_fs = filesystem is not None and not isinstance( 
                filesystem, LocalFileSystem
            )
            if not parsed.scheme and not os.path.isabs(path) and not is_cloud_fs:
                path = os.path.abspath(path)

            resolved_fs = filesystem
            if resolved_fs is None:
                try:
                    resolved_fs, _ = _resolve_single_path_with_fallback(path, resolved_fs)
                except (ValueError, ImportError) as e:
                    if ignore_missing_paths:
                        logger.debug(
                            f"Failed to resolve filesystem for glob "
                            f"'{path}': {e}, skipping"
                        )
                        continue
                    raise ValueError(
                        f"Failed to resolve filesystem for glob pattern '{path}': {e}"
                    ) from e
            expanded = _expand_glob(path, resolved_fs, ignore_missing_paths)
            resolved_paths.extend(expanded)
            resolved_filesystem = resolved_fs
            continue

[gemini-code-assist]

fnmatch.fnmatch is OS-dependent (case-insensitive on Windows, case-sensitive on Unix). However, cloud filesystems (S3, GCS, ADLS) are strictly case-sensitive regardless of the operating system of the driver or worker. Using fnmatch.fnmatch can lead to inconsistent behavior and bugs when running on Windows drivers. Use fnmatch.fnmatchcase instead to ensure consistent, case-sensitive matching for cloud paths.

Suggested change

	for j in range(1, n_rel + 1):
	if fnmatch.fnmatch(rel_segs[j - 1], seg):
	dp[i][j] = dp[i - 1][j - 1]
	for j in range(1, n_rel + 1):
	if fnmatch.fnmatchcase(rel_segs[j - 1], seg):
	dp[i][j] = dp[i - 1][j - 1]

[gemini-code-assist]

If paths is empty and ignore_missing_paths is False, the code will fall through to the else block and attempt to unpack paths, file_sizes = map(list, zip(*...)). Since zip(*[]) is empty, this will raise a cryptic ValueError: not enough values to unpack (expected 2, got 0). Handling the empty paths case explicitly with a clear error message improves robustness and user experience.

Suggested change

	if ignore_missing_paths and len(paths) == 0:
	raise ValueError(
	"None of the provided paths exist. "
	"The 'ignore_missing_paths' field is set to True."
	file_sizes = []
	else:
	paths, file_sizes = map(
	list,
	zip(
	*meta_provider.expand_paths(
	paths,
	self._filesystem,
	partitioning,
	ignore_missing_paths=ignore_missing_paths,
	)
	),
	)
	if len(paths) == 0:
	if not ignore_missing_paths:
	raise ValueError("No paths resolved or provided.")
	file_sizes = []
	else:
	paths, file_sizes = map(
	list,
	zip(
	*meta_provider.expand_paths(
	paths,
	self._filesystem,
	partitioning,
	ignore_missing_paths=ignore_missing_paths,
	)
	),
	)

[gemini-code-assist]

Directly accessing datasource.ignore_missing_paths can raise an AttributeError if the datasource implementation does not define or expose this attribute (or if it is stored as a private attribute like _ignore_missing_paths). Using getattr with a fallback is safer and more defensive.

Suggested change

	if datasource.ignore_missing_paths:
	return from_blocks([])
	if getattr(datasource, "ignore_missing_paths", False) or getattr(datasource, "_ignore_missing_paths", False):
	return from_blocks([])

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 9eecafb. Configure here.}

[cursor]

Local question-mark globs not expanded

Medium Severity

Glob expansion is gated on _has_glob_chars, but that helper only treats ? as a metacharacter when the path has a URI scheme. Local patterns such as part-?.parquet are therefore not expanded and are resolved as literal paths, so single-character ? globs on local filesystems do not match the documented behavior or the new tests.

 

^{Reviewed by Cursor Bugbot for commit 9eecafb. Configure here.}